february 2010ENGAGiNG OLdEr YOuTH Program and City-Level Strategies to Support Sustained Participation in Out-of-School Time Sarah N. DeScheNeS amy arbretoN PriScilla m. little carla herrera JeaN balDwiN GroSSmaN heather b. weiSS with DiaNa lee Commissioned by
Engaging OldEr YOuth Program and City-Level Strategies to Support Sustained Participation in Out-of-School Time Sarah N. DeScheNeS amy arbretoN PriScilla m. little carla herrera JeaN balDwiN GroSSmaN heather b. weiSS with DiaNa lee aPril 2010 Commissioned by
about usharvarD Family reSe arch ProJectHarvard Family Research Project (HFRP), housed in Harvard University’s GraduateSchool of Education, researches, develops, and evaluates strategies to promote thewell-being of children, youth, families, and their communities. We work primarily withinthree areas that support children’s learning and development: early childhood education,out-of-school time programming, and family and community support in education.Underpinning all of our work is a commitment to evaluation for strategic decision making,learning, and accountability. Building on the increasing recognition among all stakeholders thatschools alone cannot meet the educational and developmental needs of our nation’s children andyouth, we also focus national attention on complementary learning. Complementary learning isthe idea that a systemic approach, which integrates school and nonschool supports, can betterensure that all children have the skills they need to succeed. To learn more about how HFRP cansupport your work with children and families, visit our website at www.hfrp.org.Public/Private veNtureSPublic/Private Ventures is a national leader in creating and strengthening programs thatimprove lives in low-income communities. We do this in three ways:• Innovation: We work with leaders in the field to identify promising existing programs or develop new ones.• Research: We rigorously evaluate these programs to determine what is effective and what is not.• Action: We reproduce model programs in new locations, provide technical assistance where needed, and inform policymakers and practitioners about what works.P/PV is a 501(c)(3) nonprofit, nonpartisan organization with offices in Philadelphia, NewYork City, and Oakland. For more information, please visit www.ppv.org.the wall ace FouNDatioNThe Wallace Foundation seeks to support and share effective ideas and practices that expandlearning and enrichment opportunities for all people. Its three current objectives are:• Strengthen education leadership to improve student achievement• Improve afterschool learning opportunities• Build appreciation and demand for the artsFor more information and research on these and other related topics, please visit our KnowledgeCenter at www.wallacefoundation.org.To fulfill its mission, The Wallace Foundation often commissions research and supports thecreation of various publications. In all cases, the findings and recommendations of individualreports are solely those of their authors.
acknowledgementsA study of this size is possible only with the hard work and contributions of alarge team. HFRP and P/PV staff and consultants, as well as all of our study’srespondents, supported this work in many ways. We would like first to thank the people in our research sites who took thetime to speak with us about their work with older youth, particularly ourmain contacts in each city who helped coordinate our efforts for each site: JimChesire in Chicago, Rebecca Kelley in Cincinnati, Christopher Caruso andCathleen Collins in New York City, Elizabeth Devaney in Providence, SandraNaughton and Laura Moye in San Francisco, and Meeta Sharma-Holt andEllen London in Washington, DC. Many additional staff members from both HFRP and P/PV made excellentcontributions to this study. Helen Westmoreland, Laurie Kotloff, and JulieGoldsmith helped with data collection on site visits. Heidi Rosenberg, KarinLiiv, Helen Malone, Katie Franklin, Meredith Mira, and Julie Goldsmithcontributed to the qualitative analysis for the report. Tina Kauh analyzed thecities’ MIS data, and Sarah Pepper conducted the analyses of the programsurveys. Sharon Deich, vice president, Cross & Joftus, provided insightful feedbackon the framing and analysis in the report and was a skillful reviewer. The Community of Practice for this study, comprising teams from 12cities across the country, shared ideas and provided additional thinkingabout participation for older youth in our series of audio conferences and inadditional communications. Community of Practice members are listed inAppendix B of this report. At The Wallace Foundation, we would like to thank Edward Pauly, MaryMattis, and Zakia Redd, our program officers over the course of the study,who supported our efforts and contributed to our thinking. Sheila Murphyand Dara Rose also vetted ideas and were helpful guides for the report, andPam Mendels provided terrific editorial guidance from the Foundation. We are also grateful to Naomi Stephen, Marcella Franck, and Carly Bournewho led the editorial and production process at HFRP for this report. Thereport has benefited greatly from their input and expertise.
Contentsabout uS iiiackNowleDGemeNtS ivExecutive Summary ix overview x research Strategy and methods xi major research Findings xi implications xiiiIntroduction 1 research Purpose and Questions 3 contributions of the research Study 3 Structure of the report 4chaPter 1Research Methods and Overview 5 city Selection 6 Data collection 6 Sample Selection 8 Program Sample Descriptions 10 youth Served 11 analysis 12 limitations of the Study 14chaPter 2Keeping Youth Engaged Over Time 15 empirical evidence of Program characteristics that matter for Sustained Participation 16 Distinguishing Program Practices of high-retention Programs 17 Distinguishing Structural Features of high-retention Programs 21 Promising Practices for Supporting Participation: additional Survey and interview Findings 21
Promising recruitment Practices 26 Summary 28chaPter 3Developmental Differences Between Middle Schooland High School Programs 31 middle School Programs 33 high School Programs 34 Summary 36chaPter 4City-Level Supports to Promote and Sustain Participation 37 overview of city-level Supports 38 Program views on Participation in oSt initiatives 42 Summary 46chaPter 5Key Findings and Implications 47 key Findings 48 implications 49 concluding thoughts 51Notes 53Appendices 55 appendix a cities and initiatives 56 appendix b youth Participation community of Practice 61 appendix c respondent list 64 appendix D Sample Selection and Description 68 appendix e analysis Description 76 appendix F twenty-eight Programs/organizations interviewed 79 appendix G Practices and Features of high-retention Programs 84 appendix Notes 89
list of tablestable 1.1 overview of cities 7table 1.2 city oSt initiatives 7table 1.3 miS Participation rates by Sample 10table 1.4 Program characteristics 11table 1.5 youth Served 12table 1.6 Program retention rates by Sample 13table 2.1 key Program Practices and Features corresponding to higher rates of retention in Programs 17table 2.2 leadership opportunities in high-retention Programs 19table 2.3 Strategies to keep informed about youth 20table 2.4 rewards and incentives 25table 2.5 Parent engagement activities 26table 4.1 Programs’ Perceptions of city-level Supports 43Appendix Tablestable D.1 Participation rate calculations 69table D.2 Participation rates by Sample including ranges 70table D.3 Staff experiences and characteristics 72table D.4 types of activities offered 73table D.5 types of Services offered 73table D.6 leadership opportunities 73table D.7 recruitment Strategies 75table D.8 Strategies to engage Parents 75table D.9 initiative-related activities 75table D.10 top 10 types of initiative help 75table G.1 leadership opportunities 84table G.2 Strategies to keep informed about youth 85table G.3 Structural Features 85table G.4 opportunities to interact with Peers 86table G.5 activity types 86table G.6 Services for youth 87table G.7 rewards and incentives 87table G.8 Parent engagement activities 88table G.9 recruitment Strategies 88
Executive Summaryoverview tives are attempting to build the capacity of programs to deliver better-quality programming by engaging in oneOut-of-school time (OST) programs represent a vital or more of the following efforts: supporting professionalopportunity and resource for learning and development development for providers, providing funding, imple-for children and youth. There is growing recognition that menting quality improvement efforts, establishing dataOST is important not just for elementary school stu- tracking systems, and connecting OST programs to onedents, whose parents need supervision for their children another and to other community institutions. All of thesewhen they are not in school, but also for middle and efforts can directly or indirectly support improved accesshigh school youth, whose participation in OST programs to and sustained participation in OST programs.can help keep them connected to positive role models Given the potential of city-level OST initiatives toand engaged in their education at a time when many are support participation, and against the national backdropbeginning to disengage from schools. of inequitable access to quality OST programs for older Further, evidence suggests that once older youth youth from disadvantaged communities, The Wallacehave enrolled in a program, meaningful and sustained Foundation commissioned this research study. Toparticipation is a key factor in attaining positive out- understand how to promote sustained participationcomes. However, despite the well-documented benefits in OST programs, this study examined the programof OST participation for older youth, their participation characteristics—both program practices and structuralwanes with age. OST programs struggle with how to features—associated with high participation and reten-recruit and retain older youth and continue to look for tion that were employed by OST programs, primarilyguidance on how to do so more effectively. There are serving disadvantaged youth, in six cities that havealso real discrepancies in access to and participation in worked toward building OST initiatives. In particular,OST programs by location and socioeconomic status. this report addresses how OST programs keep middlePredictably, youth from lower-income families and and high school youth engaged over time (i.e., theneighborhoods have fewer OST opportunities than duration of participation) and how the supports that citytheir more privileged peers, and many low-income and initiatives provide can help foster youth participation,minority families report unmet need for high-quality with the assumption that programs can have a potentiallyand accessible programming. The lack of opportunity for greater impact if they are able to work with these youthsome youth is especially problematic given our nation’s over an extended period of time.increasing dropout rates. If, as research suggests, OST We examined three key questions:programs have the potential to support graduation andpostsecondary success, then better access to quality 1. What are the characteristics of high-participationOST programs may have the potential to help address OST programs that support sustained participation aseducational inequalities, particularly in urban areas. measured by retention? In response to the evidence pointing to the benefits 2. How do these characteristics differ for middle schoolof out-of-school time, coupled with the lack of access and high school youth?in many urban neighborhoods, many cities are creating 3. What strategies are city initiatives implementingcitywide infrastructures to support networks of OST to support access to programs and sustainedprograms, with one goal being to support participation. participation, and how do OST programs perceive theWith support from The Wallace Foundation and other usefulness of city-level strategies for achieving theirprivate and public dollars, these nascent OST city initia- participation goals?
executive Summary xiresearch Strategy and methods Altogether, we analyzed data from 198 program surveys, 28 program interviews, and 47 city-levelUsing mixed-methods research strategies, the study respondents. Our quantitative analysis focused on thedesign brought together both survey data from a large program practices and structural features associatedsample of programs and in-depth interview data. This with retention (i.e., duration of participation) of youthdesign allowed for both breadth and depth in our in programs. To identify characteristics that wereunderstanding of critical issues related to access to significantly associated with higher rates of retentionand sustained participation in OST programs for older among older youth participants, we first examinedyouth. We collected and integrated these qualitative and which of the numerous individual program practicesquantitative data and used an iterative analytic process, and structural features from the survey data wereweaving together findings from both sets of data to significantly more common in high-retention programsconfirm, augment, and challenge our understanding of than in lower-retention programs. For this study, weprogram characteristics—both program practices and define high retention as retention of 50 percent or morestructural features—and support from city initiatives. of a program’s youth participants for 12 months or more. The six cities in the study—Chicago, Cincinnati, We then conducted a regression analysis of retentionNew York, Providence, San Francisco, and Washington, to isolate which of the many competing practices andDC—were chosen because they have an intermediary or features were uniquely associated with the variation ingovernment agency coordinating funding and providing retention rates, even when taking into account otherservices for OST programs, a management information practices and features.system (MIS) or database to keep track of attendance and Analysis of our interviews, in addition to documentparticipation, extensive programming aimed at middle review, enabled us both to identify program practicesand high school youth, and a focus on low-income youth that respondents cited as relating to greater retention andand distressed neighborhoods. The initiatives in these to create a picture of what it takes in programs and at thecities all provide a set of supports to OST programs in city level to keep youth engaged in programs over time,the community, and they are making efforts to raise the using a grounded theory approach. We focused on theprofile and increase understanding of out-of-school time major themes present across programs related to the suc-in their cities; they are also all relatively new, having been cesses and challenges of achieving high participation andfounded between 2004 and 2007. retention rates and what program practices or features After we identified the six cities for inclusion in the were linked to these efforts. We also analyzed programstudy, we then identified a large number of programs in data to understand how programs participate in OSTthese cities with high participation rates among middle initiatives. Throughout the analysis, we cross-walkedand high school youth, based primarily on MIS data findings from the interviews and the survey against eachgathered by the city-level OST initiatives, and adminis- other to refine our understanding of participation.tered a survey to program leaders, asking about programactivities and features, staffing, youth participants, familyinvolvement, use of data, recruitment and orientation major research Findingspractices, practices for fostering and supporting engage- Five program characteristics (two program practicesment, and involvement with the OST initiative in the city. and three structural features) were identified that setOut of the sample of programs that returned a survey, apart the programs that were the most successful inwe selected a smaller subset of programs to interview in supporting high retention:depth. The survey sample had an average program-levelparticipation rate of 70 percent, and the interview sample • Providing many leadership opportunities tohad an average program-level participation rate of 79 youth in the programspercent. We also selected a group of city-level respon- • Having staff keep informed in several waysdents to be interviewed for the study. about youth outside programs
xii engaging older youth• Being community-based youth opportunities to interact with peers, create struc-• Enrolling 100 or more youth tures and routines to make youth feel comfortable and• Holding regular staff meetings safe, and take advantage of their participants’ willingness to try new things, particularly through peer interaction. These practices and features explained 38 percent High school programs focus their programming moreof the variance in retention. Our analyses indicate that on providing formal and informal opportunities toamong the group of programs serving older youth, explore and prepare for college and other postgraduationthe ones that achieve relatively high rates of retention plans; giving youth more responsibility through job-likeemphasize youth leadership and outperform other OST programming, apprenticeships, and mentoring; andprograms in their efforts to stay connected with youth; offering the content and the particular skills older teensthey are also more likely to be larger community-based want to learn.organizations that give staff members regular opportuni-ties to meet about their programs. City-level OST initiatives employ a set of common recruitment and retention supports, but it is less clearThere is an additional set of retention and recruitment that these efforts have made a difference in programs’practices that, while not statistically related to retention abilities to recruit or retain older youth.when we account for other factors, were consistently City initiatives provide a set of services aimed atreported as being important in engaging older youth. increasing OST participation broadly rather than solelyHigh-retention programs often employ these practices. for older youth. These supports include:Retention practices: fostering a sense of community • Engaging in citywide recruitment effortsthrough connections to program staff and peers, provid- • Coordinating information about programs across theing developmentally appropriate activities and incentives, city and helping programs networkand engaging families. • Collecting and using data on OST programsRecruitment practices: using peers and staff as recruit- • Supporting quality improvement effortsers, using organizational relationships, and matching • Providing professional development and technicalprogram attributes to youth needs. assistance to programsThese additional strategies may be associated with They were also beginning to foster relationships withengagement and/or participation frequency; more school districts and to work with families on a citywideresearch is needed. basis. Based on city-level respondents’ reports, these efforts may be increasing recruitment and participationThe study found that the same five program features at the city level.and strategies were significant in understanding how The data collected for this study, however, providedprograms retained middle and high school youth, little evidence that accessing these city-level supportsyet program leaders reported that there were also (which were deemed useful by the programs surveyed)important differences geared toward meeting the needs was directly related to the retention rates of individualof each age group. programs. Helping programs to network, providing The factors that were quantitatively linked to retention training in youth engagement, and helping with evalu-were the same across the two age groups—keeping ation were three of the supports used by the greatestinformed about youth participants’ lives, providing many number of programs surveyed. Both high- and lower-leadership opportunities, and the presence of certain retention programs, however, reported similar patternsstructural features. However, our interviews with the of use of these and many other supports that they were28 high-participation programs allowed us to better asked about on the survey. In two cases where there wereunderstand how these and other practices manifested differences, it was the lower-retention programs thatthemselves differently when working with middle or high were more likely to use the supports.school youth. Successful middle school programs give
executive Summary xiii In addition, programs reported that being part of a Choice is an important program component and acity-level initiative created new challenges having to do key feature of youth development, but it seems to matterwith data management, program competition, and tying in different ways for middle school and high schoolparticipation numbers to quality within a high-stakes programs. Our interviews with program staff suggestedfunding environment. that youth become more focused in their interests as they move into high school, which often means that they are in more specialized or single-focus programs. As a result,implications while activity choice within programs is developmentallyOur findings can help programs move toward a more appropriate for middle and high school students, highnuanced approach to recruiting and retaining older school students may also benefit from choice across ayouth and help cities understand their role in supporting variety of more specialized programs. Cities can workparticipation. In addition, these findings have implica- toward this objective either by providing programs withtions for future investment and policy decisions about funding to add specialized activities or by creating aOST programming for older youth. Therefore, we offer variety of specialized OST opportunities for high schoola set of implications aimed at key decision makers—city youth.leaders, funders, and others—whose goal is to continue OST programs’ attention to developmental changesto improve access to and participation in OST programs can support continuing youth engagement in OSTas part of their overall efforts to support learning and programs.development and to create pathways of opportunity for Understanding developmental growth can helpolder youth. programs retain youth longer as well as support programThe program practices distinguishing programs that participants’ transition from middle school to highachieve high rates of retention among older youth from school. High-retention high school program providersthose that do not can help guide the actions of program reported that their participants want programming todirectors and city leaders as they try to improve partici- help them meet concrete goals, such as taking the SAT.pation within a context of limited resources. Middle school programs reported that, particularly Our findings about the two practices that set high- around eighth grade, youth stop attending because theyretention programs apart—providing many leadership want a program that feels “older.” OST programs can useopportunities to youth in the programs and having staff this finding as an opportunity to create programmingmembers keep informed about youth outside programs for eighth and possibly ninth graders that includes morein several ways—can give other programs an idea of responsibility and skills aimed at having a successfulwhere to direct scarce resources. Because we know these ninth-grade year. Cities can support these efforts bypractices support retention, city initiatives can target bringing OST providers and school staff together toprofessional development and technical assistance efforts create curricula for transition programs and establish ato ensure that these practices are implemented effectively. team approach to the transition. By supporting youth in The other practices that high-retention programs use, transition from middle to high school, this collaborativeeven though they did not prove to be significant in the effort could lower the dropout rates for particularregression analysis, warrant further attention. Although schools.we do not know conclusively whether these practices Family engagement matters for older youthpromote retention in other settings, we do know that participation.they were reported by the programs in our study (both Program and city-level respondents alike clearlyon the survey and in interviews) as being part of an understand and value family engagement as a strategy tooverall “participation package.” recruit and retain older youth, but are challenged as toCities should consider offering a variety of specialized how to implement effective family engagement strategies.activities for high school youth. Further, though family engagement practices were not
xiv engaging older youthstatistically related to retention, high-retention programs Improved data-based decisions can improvein this study reported using more strategies to engage participation.families than did lower-retention programs. Our findings Cities use data in multiple ways to support participa-have implications for city-level professional development tion, including data about location of and access toefforts, which could be designed to include training on programs, where underserved youth live, participationworking with families. They also have implications for rates, and quality across the initiatives. Overall, programsrecruitment strategies, which should include reaching reported that the city-level supports that enabled them toout to families in a variety of ways to persuade them of obtain and use information were helpful for improvingthe value of OST participation for older youth. recruitment and retention; they also reported challenges, however, related to data collection and use that citiesSupporting school–program partnerships can help need to address. Initiatives can work, for example, torecruitment efforts. ensure that data collection and databases are supporting Initiatives are in a strong position to influence and programs’ work and that programs are spending theiradvocate for partnerships between school and district time managing data in ways that are helpful for partici-leaders and OST program leaders. They can increase pation and are not sapping organizational resources. Cityyouth access to programs by actively supporting the initiatives can support programs’ understanding and useestablishment and development of these partnerships. of participation data in order to improve recruitment andThe stronger the partnerships between programs and retention. The next step in the coordination of data is toschools, the more energy they can invest in targeted link OST data to other data systems, including those ofrecruitment fairs and strategic marketing efforts during schools, to develop a more comprehensive understandingand outside of the school day. City-level initiatives can of participation and outcomes across all the supports,support partnerships not only by linking and connect- including schools, available to youth in the city.ing schools with OST providers, but also by helpingprograms and schools develop mutually beneficial goals City-level initiatives should work with programsand expectations; streamlined tools for data sharing; and for older youth to learn how to better supportclear, two-way channels of communication regarding retention goals.students. All of the cities in our study employ city-level sup- ports to improve access to and sustained participationResources for organizational capacity are important to in OST programs; few of these strategies, however,support participation. appeared targeted toward the participation of older youth Our findings suggest that high-retention programs in particular. Rather, the strategies were part of cities’have strong organizational capacity and sound program overall initiative-building efforts to support the qualitymanagement. These programs’ staff members have time and sustainability of OST programs. Although citiesto go the extra mile, attend meetings and plan program- reported using strategies that directly addressed recruit-ming, network with other providers and schools, and ment, such as social marketing, most of the strategiesattend professional development opportunities. In fact, they employed addressed retention only indirectly.many of the programs selected for our in-depth study Further, none of these strategies supported high-were supported by large OST intermediaries (like Beacon retention programs’ participation goals in a statisticallyinitiatives and Boys & Girls Clubs) that provide this significant way. Therefore, applying what we have learnedkind of capacity building. These findings suggest that about the high-retention programs in our study—andinvestments in direct service alone are necessary but with the understanding that recruitment and retentionnot sufficient to improve retention and that resources are two sides of the same coin—it is important for citiesshould be allocated to sufficiently support organizational to strengthen their recruitment and retention efforts,development, including resources to support the finding finding out from programs what is needed to promotethat regular staff meetings matter for retention. the sustained participation of older youth.
introductionO ut-of-school time (OST) programs represent a the skills necessary for a productive adulthood; duration vital opportunity and resource for learning and of participation may be a critical factor in attaining development. There is growing recognition positive outcomes.7 that OST is important not just for elementary Despite the findings linking OST participation toschool students, whose parents need supervision for their positive outcomes, programs still struggle with how tochildren when they are not in school, but also for middle attract and engage older youth.8 Historically, adolescentand high school youth,i whose participation in OST participation in OST programs has been relatively lowprograms can help keep them connected to positive role compared with that of elementary school-aged youth.9models and engaged in their education at a time when For example, of the youth who participate in afterschoolmany are beginning to disengage from schools.1 programs, only 18 percent are in middle school and 12 The benefits of OST participation for older youth are percent in high school.10well documented, with research indicating that participa- Participation of older youth in OST programstion in well-implemented OST programs and activities plummets for a number of reasons. Adolescents havehas the potential to support postsecondary success and many options for how they spend their time outsidea healthy adulthood. Participation is associated with of school and do not necessarily have to be involveda range of academic and learning-related outcomes, in programs for afterschool care. Older youth haveincluding improved academic achievement and gradu- needs that are quite different from those of youngeration rates and higher rates of school attendance.2 OST children, and many programs are ill equipped to handleparticipation has also been correlated with positive developmental differences. Older youth might need tofeelings toward school and improvement in school take on family responsibilities like child care, might havebelonging, particularly for the oldest youth.3 Finally, OST jobs to help family finances, or might prefer to hang outparticipation bolsters social, career, and civic skills for with friends.11 Older youth, particularly those at risk ofolder youth through team-building work, the develop- becoming disconnected from school, might not want toment of strong relationships with adults and peers, and spend any more time in the school building than theyinvolvement in “prosocial” activities.4 have to.12 All of these factors have implications for how But once older youth have enrolled in OST programs, to structure programs for older youth to best meet theirmeaningful and sustained participation is a key factor developmental needs.in attaining positive outcomes.5 Research suggests that There are also real discrepancies in access to andwhen youth are engaged in programs in meaningful participation in OST programs by location and socio-ways, they are likely to learn more, experience better economic status.13 Predictably, youth from lower-incomedevelopmental outcomes, and stay in programs longer.6 families and neighborhoods have fewer OST oppor-While few data are available on exactly how much tunities than their more privileged peers, and manyparticipation is needed for youth to reap the benefits of low-income and minority families report unmet needOST programs, researchers and practitioners do have for high-quality and accessible programming.14 The lacka sense that older youth need exposure to a range of of opportunity for some youth is especially problematichealthy environments, including OST programs, to gain given our nation’s rising dropout rates. If, as research suggests, OST programs have the potential to supporti Throughout the report, “youth,” “older youth,” and “adolescent” are graduation and postsecondary success, then betterused to refer to middle and high school-aged youth. access to quality OST programs may have the potential
Introduction 3to help address educational inequalities, particularly in initiatives provide can help foster youth participation,urban areas. with the assumption that programs can have a potentially In response to the evidence pointing to the benefits greater impact if they are able to work with these youthof out-of-school time, coupled with the lack of access over an extended period of time.in many urban neighborhoods, many cities are creating We examined three key questions:citywide infrastructures to support networks of OST 1. What are the characteristics of high-participationprograms, with one goal being to support participation. OST programs that support sustained participation asThe infrastructures across cities vary: Some take the measured by retention?form of OST partnerships or funding collaboratives, 2. How do these characteristics differ for middle schoolwhile others consist of departments and nonprofit and high school youth?intermediaries dedicated to supporting youth organiza- 3. What strategies are city initiatives implementingtions, their staff, and the youth in the programs. In this to support access to programs and sustainedreport, we use the term “initiative” to refer to efforts participation, and how do OST programs perceive theto create these city-level infrastructures for OST. With usefulness of city-level strategies for achieving theirsupport from The Wallace Foundation and other private participation goals?and public dollars, these nascent OST city initiatives areattempting to build the capacity of programs to deliverbetter-quality programming by engaging in one or more contributions of the research Studyof the following efforts: supporting professional develop- This study builds on and expands the knowledge basement for providers, providing funding, implementing about older youth participation in several importantquality improvement efforts, establishing data-tracking ways. First, while many studies have been conducted onsystems, and connecting OST programs to one another promising retention strategies (by this report’s authorsand to other community institutions. and others), most of these have been based on a small sample of handpicked programs. This study examinesresearch Purpose and Questions the program characteristics (both program practices and structural features) of almost 200 OST programsGiven the potential of city-level OST initiatives to across six diverse cities, as well as a smaller subset ofsupport participation, and against the national backdrop programs chosen for in-depth study, in the contextof concern about access to the benefits of quality OST of a mixed-methods research design (see Chapter 1programs for older youth from disadvantaged communi- for a description of this design). Second, while manyties, The Wallace Foundation commissioned the research studies recently have examined participation of olderstudy reported here. To understand how to engage older youth as a whole—middle and high school—our studyyouth in meaningful ways in OST programs, this study compares and contrasts the program practices that areexamined the program characteristics—both program effective for each of these age groups. Given the profoundpractices and structural features—associated with high developmental differences between middle school- andparticipation and retention in OST programs primarily high school-aged youth, it is not surprising that aserving disadvantaged youth in six cities that have “one-size-fits-all” strategy does not work well. Our studyworked toward building OST initiatives.ii In particular, points to the need for programs to take a more nuancedthis report addresses how OST programs keep middle developmental approach to working with older youth.and high school youth engaged over time (i.e., the Finally, there is emerging knowledge, supported by Theduration of participation) and how the supports that city Wallace Foundation and others, on developing citywide OST initiatives, but no studies have attempted to under-ii As described in Chapter 1, “high participation” was defined using stand the role these initiatives play in improving access toeach city’s management information system (MIS).
4 engaging older youthand sustained participation in individual OST programs.This study begins to explore this important topic. As this study demonstrates, there is a set of programpractices and structural features that distinguishprograms that attain high rates of retention amongolder youth from programs that do not: They are likelyto be community-based programs that enroll a largernumber of youth, offer a greater number of leadershipopportunities, have more ways to keep staff informedabout participants, and hold regular staff meetings todiscuss program-related issues. Although we report onwhat city initiatives indicate they are doing to supportaccess and sustained participation, we did not find anyempirical association between city-level participationsupports and higher rates of long-term retention. Onereason may be that the six city initiatives examined inthis report are relatively young (in existence 5 years orfewer). Another possibility is that the strategies of OSTinitiatives examined in this study affect other key areas ofprogram success, such as enrollment rates, that were notthe focus of this report.Structure of the reportChapter 1 describes our mixed-methods research strat-egy, including information on sample selection and datacollection and analysis. The next three chapters integrateour qualitative and quantitative findings to address ourthree research questions. Chapter 2 presents findings onthe program characteristics of high-retention programs,as well as other commonly used program practices forsustained participation, which together illustrate howto meaningfully engage older youth in OST programs.Chapter 3 examines the differences in OST programmingfor middle and high school youth that correspond todevelopmental changes. Chapter 4 draws on informationcollected from our city-level respondents to presentdata on city-level participation strategies; it then usesinterview and survey data to report on how programsperceive the value of city initiatives in supportingprogram participation goals. Chapter 5 concludes thereport with implications of our results for future OSTprogramming and OST initiative-building efforts.
chaPter 1research Methods and OverviewU sing mixed-methods research strategies, the diversity provides interesting points of comparison and study design brought together both survey data contrast (see Table 1.1). Their population sizes range from a large sample of programs and in-depth from fewer than 1 million to more than 8 million, while interview data. This design allowed for both high school graduation rates vary from 46 percent tobreadth and depth in our understanding of critical 68 percent. San Francisco, interestingly, has the lowestissues related to access to and sustained participation percentage of youth in any large city in the Unitedin OST programs for older youth. We collected and States, which presents its own set of challenges forintegrated qualitative and quantitative data and used an participation.16iterative analytic process, weaving together findings from The six selected initiatives all provide a set of supportsboth sets of data to confirm, augment, and challenge to OST providers in the community, and they are makingour understanding of program characteristics—both efforts to raise the profile and increase understandingprogram practices and structural features—and sup- of out-of-school time in their cities; these efforts will beport from city initiatives. This chapter describes our discussed in Chapter 4. (See Appendix A for descriptionsmixed-methods approach, including city selection, of cities and their OST initiatives.) The OST initiatives indata collection activities, program sample selection and each city are profiled in Table 1.2. They are all relativelycharacteristics, and analysis. new, having been founded between 2004 and 2007, and they are coordinated by different types of organizations— both nonprofit intermediaries and government agencies.city SelectionTo understand how program participation may beaffected by city initiatives’ supports, we selected our six Data collectionsites—Chicago, Cincinnati, New York, Providence, San Five main sources of data were used to develop theFrancisco, and Washington, DC—because they all have findings in the report:• An intermediary or government agency coordinating 1. MIS participation data. Each city selected for inclusion funding and providing services for OST programs in this study provided, at a minimum, individual-level• A management information system (MIS) or database attendance data from its respective MIS to document, to keep track of attendance and participation program by program within its initiative, participation• Extensive programming aimed at middle and high rates over the 2007–2008 school year. In addition, school youth each city provided demographic information on• A focus on low-income youth and distressed participants (most commonly, ethnicity/race, gender, neighborhoods and age or grade level) so that we could calculate approximate participation rates for middle and high All of the cities in this study are contending with school youth separately. There was variability in howissues affecting urban areas, including issues related to these data were recorded by each city and transmittedhigh poverty rates. Providence’s child poverty rate (36.3 to us. The similarities and differences and how wepercent), for instance, is twice that of the United States worked with each data set to calculate participationas a whole, while Cincinnati is among the 10 U.S. cities rates are described more fully in Appendix D.with the lowest median household income.15 The cities’
chapter 1: research methods and overview 7table 1.1Overview of Cities School children in high School city Population* enrollment** Poverty* Graduation rate*** Chicago 2,725,206 467,174 30.7% 49.6% Cincinnati 299,577 46,674 40.3% 45.6% new York City 8,308,163 1,392,232 27.4% 49.9% Providence 169,635 28,614 36.3% 60.3% San Francisco 798,176 76,281 11.6% 67.8% Washington, dC 588,373 80,094 27.0% 48.8%* u.S. Census Bureau, 2006–2008 american Community Survey.** Population enrolled in K–12th grades. data from u.S. Census Bureau, 2006–2008 american Community Survey.*** EPE research Center Diplomas Count 2009 mapping tool.table 1.2City OST Initiatives year city initiative coordinating body children and youth Served* Started Chicago Out-of-School time Chicago department of Family & 2006 Over 175,000 children and (OSt) Project Support Services (FSS) with after youth served in 2008–2009 School Matters, Chicago Public Schools, the Chicago Park district, and the Chicago Public library Cincinnati CincyafterSchool YMCa of greater Cincinnati 2004 3,896 children and youth served in 2008–2009 new York City Out-of-School time nYC department of Youth and 2005 Over 78,000 children and (OSt) initiative Community development (dYCd) youth served in 2006–2007 Providence afterZones Providence after School alliance 2004 Over 1,600 middle school (PaSa) students served annually San Francisco afterschool for all SF department of Children, 2005 27,608 children and youth (aFa) initiative Youth & their Families (dCYF) and served in 2008–2009 San Francisco unified School district (SFuSd) Washington, dC Project My time** Children and Youth investment trust 2007 1,600 middle school Corporation (dC trust)*** students served in 2009*approximate number of youth served; data furnished by initiatives. the Chicago number is the total served by all five partners. the San Francisco number is thetotal funded slots for youth aged 6–13.**Since data collection, Project My time has transitioned to the dC Public Schools.***Founded in 1999.
8 engaging older youth2. Online program survey. Selected programs within each Appendix B for more information about the Com- city (see “Sample Selection” for selection procedures) munity of Practice and a list of members). were asked to complete an online program survey. Additionally, a thorough literature review of OST The survey was designed to generate information participation for older youth as well as a review of the about program activities and features, staffing, youth emerging literature on OST systems deepened our participants, family involvement, use of data, recruit- understanding of the developmental needs of middle and ment and orientation practices, practices for fostering high school youth and helped us develop a theoretical and supporting engagement, and involvement with lens to guide our instrument development, analysis, and the OST initiative in the city. interpretation of findings.3. Site visits to each city. In-person interviews were conducted with OST program leaders at 28 selected programs and with 47 city-level respondents. The Sample Selection interviews with program leaders covered program We used a funnel approach to select our samples: After activities and structure, the youth who participate, we identified the six cities for inclusion in the study, we recruitment practices and challenges, attendance then identified a large number of programs in these cities issues, retention practices and challenges, develop- with high participation rates among middle and high mental issues for older youth, and experience in an school youth based on city-level MIS data, and adminis- OST initiative. The interviews with city-level respon- tered a survey to program leaders. Out of the sample of dents addressed their role in the city OST initiative, programs that returned a survey, we selected a smaller how the initiative supports recruitment and retention, subset of programs to interview in depth.iv Thus we have partnerships to support OST programs, data and two program samples in this study: a survey sample and evaluation, and city contexts for OST. an interview sample. We also selected a group of city-4. Document review. Documents provided to us during level respondents to be interviewed for the study (see our site visits and gathered via online searches were Appendix C). This section of the chapter describes our reviewed to supplement our understanding of the sampling strategies; Appendix D provides more detailed city initiatives and of how programs were working to information on how we developed our survey samples recruit and retain older youth. and on the characteristics of programs that responded to5. Community of Practice.iii The Community of Practice our survey. enabled us to vet and expand on the ideas coming from the survey and interviews. It comprised teams of three or four individuals from 12 cities—the 6 Program survey sample research sites and 6 other cities working on city-level To generate the sample of programs to complete our support for OST—as well as consultants and represen- online survey, we used data from each city’s OST tatives from national organizations. The group met six management information system to calculate average times over the course of the study to discuss themes participation rates for each program in the initiative. In related to participation and emerging findings (see general, we calculated average program participation rates as the proportion of program sessions youthiii A Community of Practice is an intentional, focused, and voluntary attended, averaged across all youth attending thegroup whose members come together around a common interest or program.v For example, a youth who comes to half theproblem to share knowledge, find solutions, improve performance,and discuss and test the transferability and scaling of solutions andinnovations. The Community of Practice convened regularly to discuss iv A few interviewed programs were chosen based on recommenda-topics important to the study and contributed to the overall framing tions and reputation.of the study and to our understanding of specific recruitment and v Some cities track enrollment and exit dates for individuals, allowingretention strategies. for more precise participation rates to be calculated; others do not
chapter 1: research methods and overview 9sessions offered would have a participation rate of 50 in cities in which there were more than 50 programs thatpercent; if a second youth has a 100 percent participation met the criterion, we sampled from all the programs thatrate (attending all the sessions offered), the program’s met the criterion. This process identified 346 programsaverage participation rate across both youth participants from the MIS data that were included in the onlinewould be 75 percent. (See Appendix E for more detailed survey portion of the study. We received a total of 198information on calculations.) completed program surveys (or 57 percent of those After the MIS data were analyzed, programs with a surveyed), which constituted our survey sample forparticipation rate of at least 44 percent were selected quantitative analysis.for inclusion in the survey. vi This cutoff allowed us toidentify a large number of programs within each city Interviewed program samplethat had a range of success at engaging older youth. In Results from the survey data guided in part the selec-order to detect the differences between more successful tion of 28 programs across the six cities for moreprograms and less successful programs, we wanted a in-depth study (see Appendix F for descriptions of thesesample with both strong and moderate older-youth programs). Criteria for the qualitative program sampleparticipation. Because the literature already suggests a included an MIS participation rate of 60 percent orset of practices that seem associated with engaging older higher, geographic distribution across the city, a mix ofyouth, we decided not to include programs with poor program activities and goals, and service to primarilyparticipation rates; we thought we would learn less from low-income youth as defined by percentage of free orthese programs.vii Our goal was to select approximately reduced-priced lunch participants. We also examined50 programs per city that met the minimum participa- retention rates to ensure that we included sometion criterion of at least 44 percent. In some cities, this programs with high retention. Program lists for each citymeant choosing all the programs that met the criterion; were vetted with leaders of the city OST initiatives, who suggested additional programs for the sample based ontrack this information, and therefore we needed to make estimations these programs’ reputation for participation and engag-differently. When a program served both middle and high schoolyouth, we calculated the participation rates for the two groups sepa- ing activities.viiirately. We then asked the program respondents to think about either The interviewed program sample includes 18 school-the middle or the high school youth they serve when responding to based and 10 community-based programs, 14 of whichthe survey questions, depending on which group met the minimumparticipation level. When both age groups met the minimum level, we focus on middle school, 8 on high school, and 6 on aasked the respondent to focus on one or the other age group to get a combination of the two. Examples of program contentsimilar representation of both middle and high school programs. areas include jewelry making, music, theater, collegevi Because Cincinnati’s initiative has fewer programs for older youth prep, law education, and a soccer and writing program.than the other initiatives in this study, we used this participation ratecutoff where data were available and developed a reputational samplefor the rest of the survey and interview sample. City-level respondent samplevii Our goal in selecting programs was to include a sample with a To understand the role of city OST initiatives in middlegreat enough range in participation rates to allow us to explore staffpractices and program features that correlate with higher retention and high school youth participation, we interviewedrates. Given that programs with low participation rates may have a 47 city-level respondents who represented a range ofhost of organizational and infrastructure issues that may be relevantto low participation generally (rather than to participation of olderyouth specifically), we wanted to be careful that the lessons we viii When describing data from the interview sample, we refer ingenerated from the data collection and analysis would be particular to most cases to the programs’ high participation rates because we dounderstanding programs’ effectiveness in attracting older youth and not have retention information from all of them. Some of the cases arenot limited by general program weaknesses such as poor quality or not high-retention programs but were selected because they had highuneven programming. Thus the programs we selected did not include participation and were interesting along another dimension, such asthe very worst performers on participation rates, but rather the more the use of stipends or interactions with families.average programs.
10 engaging older youthtable 1.3MIS Participation Rates by Sample Full-city Database Sample Survey Sample* interview Sample** Middle High Middle High Overall Middle High school school Overall school school school school Overall j=4 j=3 j=4 j=4 j=3 j=4 j=3 j=3 j=4 n=330 n=649 n=979 n=72 n=52 n=124 n=9 n=8 n=17 Program-level average participation rate across 54% 70% 65% 70% 71% 70% 79% 79% 79% all programsSource: City MiS database daily attendance data provided by the four cities with relatively comparable and complete data. the numbers for the survey andinterview samples reported on this table are smaller than for the full sample because they do not include the surveys and interviews from the cities that wereexcluded from this presentation of the combined participation rates. See footnote ix.notes: j=number of cities; n=number of programs across all cities. descriptive data include MiS data provided by four cities. Program-level values reflect theaverage across all participants within each program. City-level values are presented in appendix d and were calculated by taking the average across all programswithin each city and then taking the mean of the city-level averages.*Six programs that completed surveys did not provide MiS participation data.**One interview site did not provide MiS participation data.city-level stakeholders, including lead agency representa- Because programs were selected to receive a surveytives; MIS developers; people responsible for quality only if they met a minimum criterion of 44 percentimprovement and professional development efforts at the participation, the average for the survey sample is highercity level; heads of large community-based organizations; (70 percent). The interview sample is representative ofrepresentatives from parks, recreational facilities, and programs that have even higher rates of participation,libraries; and mayoral staff. These respondents were so the average rate across that sample of programs isselected in consultation with the lead agencies of each still higher (79 percent). The average participation ratescity’s initiative. for the high school youth within programs are similar to those for the middle school youth within programs in both the survey and interview samples; however,Program Sample Descriptions in the full database, high school youth have a higherTable 1.3 displays the participation rates based on the participation rate. More descriptive information on theoverall combined MIS data,ix the full program sample participation rates for each of the samples is presented in(“Survey Sample”), and the subset of 28 programs that Appendix D.took part in our in-depth qualitative study (“Interview Table 1.4 describes other program characteristics ofSample”). the survey sample and the interview sample. As the table Table 1.3 indicates that the participation rates from indicates, the two samples are similar along most of thethe full MIS database are relatively high (65 percent). dimensions, including age of participants, when they operate, whether or not they have been operating 5 or more years, and their service area. One difference standsix The table does not include data from New York City or Cincinnati.The participation calculations in New York City were not comparable out: A greater proportion of the interviewed programsto those of the other cities; because the bulk of the programs surveyed are school-based.from Cincinnati were selected based on nominations, participationdata were often not available.
chapter 1: research methods and overview 11table 1.4Program Characteristics interview Survey Sample characteriSticS Sample n=198 n=28 School-based 34% 46% Serves 100 or More Youth 47% 52% age of Participants Serves elementary school and older students 39% 42% Serves only middle school students 29% 31% Serves middle school and high school students 13% 12% Serves only high school students 12% 12% Serves high school and post-high school students 6% 4% number of Years Program in Operation 1–2 18% 32% 3–4 22% 12% 5+ 60% 56% Operates School-year Only 40% 42% Open 5+ days per Week 61% 69% Only Program in its area with its Particular Focus 26% 23%Source: Program surveys.youth Served the proportion for the interview sample was 87 percent.Through our interviews we learned that the youth On average, more than 90 percent of youth partici-attending the programs we studied in depth are in pants in the survey sample are non-White. Programsschools and neighborhoods with high rates of violence, serve a mix of boys and girls; an average of 52 percent ofcrime, and gang activity, and with few resources for total participants in the survey sample are female. Onlyyouth services and programs. Many of these youth must 4 percent of programs serve girls exclusively; 2 percentconstantly navigate these issues in their neighborhoods, serve only boys. The rates are similar for the interviewmaking OST a low priority for some and a much-needed sample.refuge for others. In the survey sample, an average of 25 percent of Table 1.5 provides a summary of demographic youth participants were estimated to attend other OSTinformation on the youth from both samples. One programs, based on staff responses. On average, almost acommon feature across the programs in this study (in quarter (24 percent) of youth have siblings attending theboth the survey sample and the interview sample) is same program.that participating youth are struggling with poverty. Programs serve as many as 6,400 youth annually, butAcross surveyed programs, an average of 79 percent of only 10 programs serve 1,000 or more youth annually.participants were eligible for free or reduced-price lunch; The median number of youth served annually is 90.
12 engaging older youthtable 1.5 Each program’s retention rate was calculated basedYouth Served on respondents’ answers to a series of questions on the survey. Respondents indicated the proportion of their Survey interview participants who remained in the program for 3, 6, 12, Sample Sample 18, and 24 months or longer. Percentages for youth n=198 n=28 coming for 12, 18, or 24 or more months were summed % Eligible for Free lunch 79% 87% to indicate the proportion of youth in the program who race or Ethnicity were retained for 12 months or longer. Retention rates for both the survey and interview african american 49% 57% samples are presented in Table 1.6. As the table shows, latino/a or hispanic 27% 21% in the survey sample, on average approximately a third asian 10% 10% of youth (34 percent) were retained for 12 months or White 9% 6% more. By design, the average retention rate is higher for Mixed race 4% 4% the interview sample (43 percent) because we wanted to ensure that we learned through our interviews with native american <1% <1% staff about practices used to increase retention; thus Other 1% 1% retention rate was one of the variables we considered % girls Served 52% 51% when selecting programs to be included in the interview % attending Other OSt activities 25% 24% sample. The table also shows the proportion of programs within each sample that reported that 50 percent or more % with Siblings in Program 24% 32% of the older youth served were retained for 12 monthsSource: Program surveys. or longer. Although the results presented in Table 1.3 indicate that the average rate of intensity of participation for high school youth was found to be similar to that of middle school youth across the two samples, as Tableanalysis 1.6 shows, the average rates of retention are significantly and substantially higher for high school youth in theCalculating retention programs in both samples compared with those ofOur quantitative analysis focused on the program middle school youth.practices and structural features associated with reten- The variation in retention rates reflects in part thetion (duration of participation) of youth in programs. nature of how city initiatives are set up. In at least twoRetention was selected as the main outcome of interest, of the cities in the sample (Providence and Chicago),rather than intensity (number of hours per week) of programming for older youth consists of sets of shorter,participation, for both theoretical and pragmatic reasons. more intensive programs and activities that older youthFirst, prior work on the effects of OST programs on would only be expected to attend over a short period ofolder youth suggests that they reap benefits—particularly time (e.g., activities in the Providence AfterZones or athose associated with meaningful relationships with staff session of Afterschool Matters), but they might attendand peers—through participation over a longer period multiple sessions over the course of a year.of time rather than through intense participation overa short period of time.17 Second, the data on intensity Quantitative analysisgathered via the cities’ MIS were not all comparable, To identify characteristics that were significantly associ-whereas the survey questions were asked in the same ated with higher rates of retention among older youthway across all the programs (see Appendix D for more participants, we used a two-step process.explanation).
chapter 1: research methods and overview 13table 1.6Program Retention Rates by Sample Survey Sample interview Sample Middle High Overall* Middle High Overall school school school school j=6 j=5 j=6 j=5 j=4 j=6 n=103 n=72 n=176 n=15 n=8 n=23 Program-level average rate 22% 52% 34% 32% 64% 43% of youth retained 12 or more months across all programs % of programs that retain at 24% 61% 40% 40% 75% 52% least 50% of participants for 12 months or longer (“high retention”)Source: Program surveys.note: j=number of cities; n=number of programs across all cities.*Sample size of middle school and high school programs do not sum to overall sample size due to missing data on age-group focus. First, we examined which of the numerous individual and coding structure based on what our review of theprogram practices and structural features from the literature and early findings indicated were importantsurvey data were significantly more common in high- elements to include in a study of participation andretention programs than in lower-retention programs retention and then refined our codes over time.x(see Appendix G for the usage rates). For our analysis of program interviews, we focused on Next, we conducted a regression analysis of retention the major themes present across programs related to theincluding those practices and features identified in step successes and challenges of achieving high participationone. Regression analysis allowed us to isolate which of and retention rates and what program practices orthe many competing practices and features are uniquely features were linked to these efforts. We also analyzedassociated with the variation in retention rates, even program data to understand how programs participatewhen taking into account other practices and features. in OST initiatives. For our analysis of city-levelResults of regression analyses also provide information interviews, we created detailed city-level descriptions ofon the relative contribution of each factor, above and the initiatives and identified their major efforts relatedbeyond the contribution of other factors, in explaining to participation as well as the challenges they face inretention (see Appendix E for a fuller description of the improving access and participation.regression analyses). Chapter 2 describes the findings of Throughout the analysis, we cross-walked findingsthe regression analyses in detail. from the interviews and the survey against each other to refine our understanding. Sometimes both the regressionQualitative analysis analysis and the qualitative analysis agreed, as was theAnalysis of our interviews and document review enabled case with the importance of leadership opportunities forus both to identify program practices that respondents older youth. Some themes appeared in the qualitativecited as relating to greater retention and to create a data that would not be found in the quantitative datapicture of what it takes in programs and at the city levelto keep youth engaged in programs over time, using a x We used NVivo to organize the qualitative data.grounded-theory approach.18 We developed our codes