CHALLENGES FACING COMMUNITY YOUTH SPORT PROGRAMS

Sport can provide participating youth with a range of positive outcomes, including improved self-esteem, quality peer interactions, and character development (^{Côté & Fraser-Thomas, 2016}). However, although sport can provide opportunities for such benefits, their emergence is not automatic (^{Coakley, 2011}). Indeed, sport programs must be structured appropriately, meaning that the activities undertaken are suitable for a particular age category and skill level and that the social agents within the environment engage in healthy interactions and provide adequate support to developing athletes (^{Côté et al., 2014}). As an example, the Personal Assets Framework (PAF; ^{Côté et al., 2020}), which was developed specifically for sport, posits that quality social dynamics (e.g., relationships, cohesion), personal engagement in activities (e.g., practice, play), and appropriate settings (e.g., organizational structures, community size), all contribute to immediate, shortand long-term youth development.

In addition to exploring the mechanisms through which sport can facilitate youth development (Holt, 2016), researchers have also emphasized the need to improve the evaluation of established youth development programs (^{Toivonen et al., 2021}; ^{Weiss, 2016}). For instance, research-based programs such as the First Tee (^{Weiss et al., 2013}) and the Sports United to Promote Education and Recreation (SUPER; ^{Danish et al., 2004}) were designed to foster the transfer of life skills in sport. Although results from preliminary evaluative research-based programs are promising (^{Brunelle et al., 2007}; ^{Weiss et al., 2013}), these programs only evaluated program effectiveness which is insufficient to evaluate the programs impact fully. Recently, an evaluation of the Girls on the Run program also measured the delivery of the program and its curriculum (^{Weiss et al., 2020}). However, thorough investigations into different components of a program are still warranted (e.g., the long-term sustainability). Further, program evaluation in the context of sport remains relatively new and requires further understanding of the difficulties faced by community-based programs -outside of the research paradigmfor youth development programs to be most effective.

Despite the vast number of community-based programs, there remains minimal intersection between researchers and practitioners (^{Graham et al., 2006}). This is important as intervention programs may only be sustainable while they have the necessary resources available to them (e.g., training, personnel; ^{Turnnidge et al., 2014}). Specifically, ^{Whitley and colleagues (2014}, ²⁰¹⁵⁾ outlined the challenges faced when evaluating and sustaining programs for underserved youth, with a main issue being the discrepancy between researcher and community program objectives. These authors noted a lack of communication between researchers and community stakeholders, resulting in poor trust and knowledge translation, especially in marginalized populations. Creating effective partnerships with established community programs requires consistent and transparent communication, which will ensure the pragmatic nature of the research and the subsequent relevance of empirical findings to community stakeholders. Further, researchers have noted the lack of intervention studies that test positive youth development (PYD) program impact beyond internal validity (i.e., effectiveness) and advocate for better integration of theory and application through intervention and evaluation studies (i.e., knowledge translation, ^{Turnnidge et al., 2014}; ^{Weiss, 2016}).

Effective knowledge translation involves interaction between decision-makers and researchers and results in mutual learning through the process of planning, producing, disseminating, and applying existing or new research in decision-making (^{Graham et al., 2006}). Although a gap remains between evidence-informed interventions and community implementation, knowledge translation efforts focus on creating community partnerships in the hopes that a reciprocal approach will better engage both researchers and practitioners (^{Benoit et al., 2011}; ^{Graham et al., 2006}). One approach advocated to facilitate such reciprocal efforts has been to emphasize the community program’s needs as the research focal point (e.g., ^{Verhagen et al., 2013}). To best understand these needs, it is important to first evaluate programs to understand their current impact and identify barriers faced by practitioners (e.g., lack of resources, limited time, poor communication, ^{Hoekstra et al., 2018}). However, although evaluation frameworks provide insight into a program’s impact on the target community (e.g., underserved youth), traditional efficacy-based evaluation can be insufficient for understanding the many dimensions of program effectiveness and sustainability (^{Glasgow et al., 1999}). Indeed, a recent systematic review by ^{Williams and colleagues (2022)} suggests there remains a lack of evaluations which focus on the quality of the program design and their evaluation. The RE-AIM framework was created with this issue in mind and was designed as a multidimensional tool used to evaluate both the internal (effectiveness) and external (reach, adoption, implementation, maintenance) validity of a particular program (^{Glasgow et al., 1999}).

THE RE-AIM FRAMEWORK

The RE-AIM model enables the evaluation of five dimensions: (a) Reach, (b) Effectiveness, (c) Adoption, (d) Implementation, and (e) Maintenance. Reach is defined as the absolute participation rates of the program and the representativeness of the individuals the program was aimed for. Effectiveness encompasses the influence of the program for all individuals involved (e.g., athletes, coaches, parents, etc.), pertaining to both positive and negative outcomes. Adoption is the assessment of the delivery of the program, including the proportion of settings, participation, and program resources. This can include adherence to the program principles or the accessibility of the program itself. Implementation is defined as the extent to which the program was delivered as intended. Maintenance is the degree to which the program is sustained, and participant outcomes are maintained over time.

By using these broad and flexible dimensions, researchers and practitioners can collaboratively evaluate the impact of sport programs. Such efforts have been undertaken with multisport service organizations (MSO; ^{Lawrason et al., 2021}); however, we were not aware of attempts to collaborate with grass-roots level municipal sport programs. Accordingly, the purpose of the current project was to demonstrate the feasibility and benefit of an intersectoral exchange with a not-for-profit community sport program. In partnership with the NUTMEG program, we collaboratively conducted an evaluation using the RE-AIM framework (^{Glasgow et al., 1999}). The name NUTMEG refers to a skill in football/soccer whereby one player dribbles the ball through the legs of their opponent. We established evidence-informed strategies for evaluating their program’s future impact.

NUTMEG SOCCER PROGRAM

NUTMEG was established in 2016 and is a not-for-profit soccer organization targeting physical and life skills (i.e., skills that can be applied in contexts beyond sport; ^{Pierce et al., 2016}) for underserved youth in Toronto, Canada (^{NUTMEG, 2018}). Importantly, underserved populations, such as low-income families, typically represent the lowest participation rates in youth sport, with financial and time restrictions posing as the most significant barriers (^{Holt & Neely, 2011}). It is a free ‘drop-in’ program that enables families from diverse demographic and socio-economic backgrounds to participate. By providing a more casual ‘drop-in’ setting, youth can attend sessions frequently, or infrequently, as they are without fear of financial consequence or ‘falling behind’ in skills. Informal settings such as these are important as they promote sustained participation in youth sport and thus beneficial developmental outcomes to those who may not otherwise have access (^{Holt & Neely, 2011}).

At the time of this study, NUTMEG provided two separate programs (i.e., co-ed and female only) that ran during the summer (i.e., outdoor season) and fall/winter (i.e., indoor season). Programs are held once a week for two hours, with each session including 90 minutes of activity and 30 minutes of personal development education. The education sessions change weekly and are designed to develop confidence, emphasize equality, and discuss topics such as prejudice and peer pressure using in-house (i.e., coach-athlete and athlete-athlete interactions) and guest speaker programming (e.g., Toronto Police Service, ^{NUTMEG, 2018}). Such an approach aligns with life skills research that advocates for the intentional and systematic teaching physical, psychological, and social skills (^{Camiré et al., 2011}).

NUTMEG has six core values that inform all weekly programs, coach training, and education sessions: friendship, fun, respect, equity, education, and teamwork. Programs such as NUTMEG, which focus on value-based programming, have been shown to be socially rewarding while specifically providing an opportunity for atrisk youth to stay on track personally (^{Bean et al., 2014}). Beyond the benefits to youth, NUTMEG also provides volunteer opportunities for adults and older adolescents to engage with their communities. At the time of the evaluation, 19 staff and volunteers worked within the program. No certifications are required to volunteer with NUTMEG, but optional workshops and training are provided. For example, the ‘High Five’ certification is offered to help interested coaches provide positive youth experiences and promote life-long participation (^{High Five, 2020}).

NUTMEG provides a positive and caring environment to play sports for those who otherwise would not have the means to do so. Programs such as NUTMEG that focus on motivating athletes to learn sport and life skills in accessible (i.e., free of charge) and developmentally appropriate settings have been shown to influence the social and personal development of youth (^{Bean et al., 2018}; ^{Gould et al., 2011}). Interestingly, whereas the potential benefits of such programs are clear, ^{Martinek and colleagues (2001)} caution that simply providing developmental opportunities in sport does not guarantee their transfer to other contexts. Consequently, it is important to evaluate such programs to ensure their success in engaging and maintaining PYD for participating children/youth.

THE NUTMEG EVALUATION

This evaluation involved a pragmatic approach to ensure the needs of the current program were met (^{Gibson, 2017}). Such an approach allowed data collection to remain feasible for NUTMEG while prioritizing their needs and guaranteeing their input was heard. The NUTMEG staff approached the research group and expressed their desire to have a complete program evaluation conducted. Once the two sides agreed to work together, an initial list of indicators was created and sent to NUTMEG representatives. A meeting was scheduled to ensure the organization’s objectives were met and to answer any questions and concerns from either party.

From the initial meeting, it was determined that the project would involve two phases: (1) an evaluation based on the data that were currently available (e.g., registration, attendance lists, season-end evaluation forms, exit interviews) and (2) the provision of recommendations for data to be collected in the future (e.g., coach training, PYD transfer, physical skill development). Data available included information such as athlete demographics (e.g., age, gender, postal code), athlete perceived engagement and satisfaction of the program (e.g., “How much do you enjoy the [activities] of the NUTMEG program?”), and athlete/coach reasoning for leaving the program (e.g., they moved). The NUTMEG organization received a Royal Bank of Canada (RBC) Learn to Play grant, which required a summary report to be submitted to the agency. The documents provided to RBC assisted the researchers in understanding what data NUTMEG had previously collected and were also used in culmination with RE-AIM requirements to create an overarching list of potential indicators (see Table 1). The following sections summarize the results from the initial evaluation (Table 1) and advance recommendations for researchers and practitioners involved in community-based program evaluation efforts.

DISCUSSION

The long-term sustainability of any sport program is dependent on continual evaluation, ongoing planning, and improvement of partnerships, resources, effective recruitment, enhanced internal capacity, and clear objectives (^{Whitley et al., 2015}). As there is a limited body of research on pre-existing community-based sport programs and even less on programs aimed at underserved youth, it is important to critically discuss the evaluation process in such contexts (^{Cullen et al., 2006}; ^{Jung et al., 2018}).

Researchers rarely discuss the challenges they face in research and tend to present an ‘ironed-out’ version of their work that has been subject to peer review (^{Whitley et al., 2014}). However, real-world sport programs do not have the luxury of funded experimental trials or intervention efforts. They must navigate the balance between delivering quality programming with limited resources and volunteer support. Such reality positions evaluation research as central to understanding the strengths and opportunities for growth within programs and aiding the research community to understand more accurately what is happening in youth sport programs. In this regard, it is important to highlight challenges and limitations identified throughout the current evaluation to inform future work. Specifically, three main limitations faced when using the RE-AIM framework were: (1) the lack of indicators being currently measured, (2) the lack of clarity between and within the indicators measured, and (3) the need to navigate current program constraints, such as funding requirements.

First, few indicators from the RE-AIM framework were collected. Although steps had been put in place by NUTMEG to evaluate their program, very few of those measures could be used within the RE-AIM indicators. Such a finding speaks to the narrative that a gap remains between research and practice. Researchers should be providing practical evidence-informed resources that practitioners can use outside of the research paradigm. For instance, ^{Whitley and colleagues (2015)} suggested that programs are most effective when they address specific PYD outcomes, have appropriate resources, trained staff, and include meaningful collaborations. The NUTMEG program’s overarching objective was to foster PYD in participating youth. Unfortunately, the omission of measures that enabled the effectiveness assessment made it difficult to determine the extent to which this objective was being met. In sport, researchers have highlighted the need to clearly articulate a program’s objectives and how the attainment of those objectives will be evaluated (^{Lawrason et al., 2021}). Although it is an important endeavor for youth sport programs, it is worth noting the difficulty based on the current research landscape for practitioners to obtain specific tools to facilitate the evaluation of their programs.

A second closely related limitation was that data collected by NUTMEG were not easily translated into RE-AIM measures. Much of the available data did not align with the represented indicators deemed important by researchers. Traditional evaluations put unequal importance on evidence-informed effectiveness measures in comparison to other RE-AIM indicators. The emphasis on evidence-informed over pragmatic indicators can serve as an additional barrier for pre-existing programs as rigorous methods are not likely to be adopted by grassroots programs (^{Lawrason et al., 2021}; ^{Shaw et al., 2019}). Additionally, certain aspects of the NUTMEG philosophy were unclear, making it difficult to measure. For instance, education is one of the six principles within NUTMEG, but a consistent description of their education sessions was not provided. This made it difficult to provide suggestions on potential measurement options in the future. This could also be because it was difficult for NUTMEG to define what effectiveness meant to them, which is in line with other non-profit programs (^{Herman & Renz, 1997}). As such, this exemplifies the need for bottom-up research, whereby the needs of the practitioners should be given the highest priority, and resources should be tailored to accommodate the context of interest (e.g., ^{Verhagen et al., 2013}).

Finally, challenges were encountered because NUTMEG was a pre-existing program with policies, funding, and programming already in place. As previously mentioned, other PYD sport programs were more easily evaluated as they were implemented with their foundations rooted in theory (e.g., ^{Weiss et al., 2013}). In this instance, the evaluation had to align with predetermined requirements put in place by the RBC Learn to Play grant. An extensive summary report is requisite for receiving this grant, so to ease the burden placed on NUTMEG directors, parents, athletes, and volunteers when completing the evaluation, extensive efforts were made to use the Learn to Play measures when possible. For instance, the absolute number of participants was a required measurement for the grant; however, due to the drop-in nature of the program, this was of limited value for NUTMEG to understand their impact in the community.

RECOMMENDATIONS

This section provides recommendations for data collection for youth sport practitioners. These recommendations offer insight pertaining to the facilitators and barriers to completing an evaluation on pre-existing community programs. It is also our intent that this section provides guidance to interested researchers and practitioners seeking to engage in a similar partnership in the future. In addition to the availability of specific indicators, as discussed previously, the feasibility of data collection by community-based programs must be considered. Underserved communities and programs have reduced access to many social and economic resources, which could negatively impact the potential for PYD of its participants. In this regard, we make practical tools more readily available to ensure the sustainability of programs that cultivate PYD (^{Whitley et al., 2015}). This evaluation process informed the following four recommendations for both researchers and practitioners: (a) develop meaningful definitions and measures for individual programs, (b) assess indicators at individual and organizational levels, and (c) create clear and measurable definitions of program success (including effectiveness, adoption, and implementation), and (d) emphasize and be flexible about the needs of the program.

CONCLUSION

The RE-AIM framework provided the structure for the evaluation of the NUTMEG program. This paper conducted an evaluation based on data currently collected by the program. These results were then used to inform the recommendations provided to researchers and practitioners. In summary, the analysis identified the need for collaborative efforts to develop feasible evaluation tools for pre-existing community-based sports programs. The use of a flexible framework, like RE-AIM, can enhance the assessment of the impact that community-based programs can have on individuals by allowing for individualized measures to be developed across appropriate levels within the program (e.g., athletes, parents, coaches, stakeholders).

Programs such as NUTMEG are particularly important as they provide the personal and social benefits underserved children may not otherwise experience (^{Holt & Neely, 2011}). By sharing the experience of evaluating a small, community-based program, this study increased awareness of the complexity of community partnerships, hopefully increasing the long-term sustainability of a program such as NUTMEG.