IES PRACTICE GUIDE             WHAT WORKS CLEARINGHOUSE   Structuring Out-of-School Time   to Improve Academic Achievement...
The Institute of Education Sciences (IES) publishes practice guides in educationto bring the best available evidence and e...
IES PRACTICE GUIDE            Structuring Out-of-School Time            to Improve Academic Achievement            July 20...
This report was prepared for the National Center for Education Evaluation and RegionalAssistance, Institute of Education S...
Structuring Out-of-School Timeto Improve Academic AchievementContentsIntroduction                                         ...
STRuCTuRIng OuT-Of-SChOOl TIME TO IMPROvE ACADEMIC AChIEvEMEnTList of tablesTable 1. Institute of Education Sciences level...
Introduction                                      when possible. The research base for this                               ...
InTRODuCTIOnwhich equivalence of the groups at pretest            although some of these programs haveis uncertain).      ...
InTRODuCTIOnWe appreciate the efforts of Samina Sat-        their time and expertise to the reviewtar, Virginia Knechtel, ...
InTRODuCTIOnTable 1. Institute of Education Sciences levels of evidence for practice guides                   In general, ...
Structuring out-of-                                 OST programs offer a promising approachschool time to improve         ...
OvERvIEWAlthough it is generally assumed that OST          academically; that time may need to beprograms can provide stud...
Scope of the                                       to improve academic achievement canpractice guide                      ...
SCOPE Of ThE PRACTICE guIDEschool, and SES programs targeting el-          administrators will make that decisionementary ...
SCOPE Of ThE PRACTICE guIDEand practices can impact student behavior.             educators and OST providers alike withIn...
SCOPE Of ThE PRACTICE guIDEmost likely to benefit from a strong aca-          engaging. The panel believes it is particu-d...
Checklist for carrying out the               Recommendation 3.recommendations                              Adapt instructi...
Recommendation 1.                                   outcomes. Although it was common forAlign the OST program             ...
1. AlIgn ThE OST PROgRAM ACADEMICAlly WITh ThE SChOOl DAythat met WWC standards with or with-                 program foun...
1. AlIgn ThE OST PROgRAM ACADEMICAlly WITh ThE SChOOl DAyHow to carry out this                             will reinforce ...
1. AlIgn ThE OST PROgRAM ACADEMICAlly WITh ThE SChOOl DAyExhibit 1. Sample logbook Information from the Classroom Teacher ...
1. AlIgn ThE OST PROgRAM ACADEMICAlly WITh ThE SChOOl DAy2. Designate a school staff person to coordi-     3. Connect OST ...
1. AlIgn ThE OST PROgRAM ACADEMICAlly WITh ThE SChOOl DAyField trips or cultural activities that are        tight or suffi...
1. AlIgn ThE OST PROgRAM ACADEMICAlly WITh ThE SChOOl DAycoordinator, or the school cannot afford to      Roadblock 1.4. T...
Recommendation 2.                                Level of evidence: LowMaximize student                                 Th...
2. MAxIMIzE STuDEnT PARTICIPATIOn AnD ATTEnDAnCEschool day (recommendation 1), adapts                 on attendance rates....
2. MAxIMIzE STuDEnT PARTICIPATIOn AnD ATTEnDAnCEHow to carry out this                          place of worship.68 Parents...
2. MAxIMIzE STuDEnT PARTICIPATIOn AnD ATTEnDAnCEshould consider including lunch or another            staff can follow up ...
2. MAxIMIzE STuDEnT PARTICIPATIOn AnD ATTEnDAnCESuggested Approach. If a large portion       Roadblock 2.4. Communicating ...
Ost pg 072109
Ost pg 072109
Ost pg 072109
Ost pg 072109
Ost pg 072109
Ost pg 072109
Ost pg 072109
Ost pg 072109
Ost pg 072109
Ost pg 072109
Ost pg 072109
Ost pg 072109
Ost pg 072109
Ost pg 072109
Ost pg 072109
Ost pg 072109
Ost pg 072109
Ost pg 072109
Ost pg 072109
Ost pg 072109
Ost pg 072109
Ost pg 072109
Ost pg 072109
Ost pg 072109
Ost pg 072109
Ost pg 072109
Ost pg 072109
Ost pg 072109
Ost pg 072109
Ost pg 072109
Ost pg 072109
Ost pg 072109
Ost pg 072109
Ost pg 072109
Ost pg 072109
Ost pg 072109
Ost pg 072109
Ost pg 072109
Ost pg 072109
Ost pg 072109
Ost pg 072109
Ost pg 072109
Ost pg 072109
Ost pg 072109
Ost pg 072109
Ost pg 072109
Ost pg 072109
Ost pg 072109
Ost pg 072109
Ost pg 072109
Ost pg 072109
Ost pg 072109
Ost pg 072109
Ost pg 072109
Ost pg 072109
Ost pg 072109
Ost pg 072109
Ost pg 072109
Ost pg 072109
Ost pg 072109
Ost pg 072109
Ost pg 072109
Ost pg 072109
Ost pg 072109
Ost pg 072109
Ost pg 072109
Ost pg 072109
Ost pg 072109
Ost pg 072109
Upcoming SlideShare
Loading in …5
×

Ost pg 072109

1,326 views

Published on

Published in: Education
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
1,326
On SlideShare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
Downloads
5
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Ost pg 072109

  1. 1. IES PRACTICE GUIDE WHAT WORKS CLEARINGHOUSE Structuring Out-of-School Time to Improve Academic AchievementNCEE 2009-012U.S. DEPARTMENT OF EDUCATION
  2. 2. The Institute of Education Sciences (IES) publishes practice guides in educationto bring the best available evidence and expertise to bear on the types of challengesthat cannot currently be addressed by a single intervention or program. Authors ofpractice guides seldom conduct the types of systematic literature searches that arethe backbone of a meta-analysis, although they take advantage of such work whenit is already published. Instead, authors use their expertise to identify the most im-portant research with respect to their recommendations and conduct a search ofrecent publications to ensure that the research supporting the recommendationsis up-to-date.Unique to IES-sponsored practice guides is that they are subjected to rigorous exter-nal peer review through the same office that is responsible for independent reviewsof other IES publications. A critical task for peer reviewers of a practice guide is todetermine whether the evidence cited in support of particular recommendations isup-to-date and that studies of similar or better quality that point in a different di-rection have not been ignored. Because practice guides depend on the expertise oftheir authors and their group decisionmaking, the content of a practice guide is notand should not be viewed as a set of recommendations that in every case dependson and flows inevitably from scientific research.The goal of this practice guide is to formulate specific and coherent evidence-basedrecommendations for use by educators using out-of-school time programming toaddress the challenge of improving student academic achievement. The guide pro-vides practical, clear information on critical topics related to out-of-school time andis based on the best available evidence as judged by the panel. Recommendationspresented in this guide should not be construed to imply that no further researchis warranted on the effectiveness of particular strategies for out-of-school time.
  3. 3. IES PRACTICE GUIDE Structuring Out-of-School Time to Improve Academic Achievement July 2009 Panel Megan Beckett (Chair) RAND Geoffrey Borman UNiveRsity of WiscoNsiN–MADisoN Jeffrey Capizzano teAchiNg stRAtegies, iNc. Danette Parsley MiD-coNtiNeNt ReseARch foR eDUcAtioN AND LeARNiNg (McReL) Steven Ross the JohNs hopkiNs UNiveRsity Allen Schirm MAtheMAticA poLicy ReseARch, iNc. Jessica Taylor fLoRiDA DepARtMeNt of eDUcAtioN Staff Samina Sattar Virginia Knechtel Elizabeth Potamites MAtheMAticA poLicy ReseARch, iNc.NCEE 2009-012U.S. DEPARTMENT OF EDUCATION
  4. 4. This report was prepared for the National Center for Education Evaluation and RegionalAssistance, Institute of Education Sciences under Contract ED-07-CO-0062 by theWhat Works Clearinghouse, a project of Mathematica Policy Research, Inc.DisclaimerThe opinions and positions expressed in this practice guide are the authors’ and donot necessarily represent the opinions and positions of the Institute of Education Sci-ences or the U.S. Department of Education. This practice guide should be reviewedand applied according to the specific needs of the educators and education agencyusing it, and with full realization that it represents the judgments of the reviewpanel regarding what constitutes sensible practice, based on the research that wasavailable at the time of publication. This practice guide should be used as a toolto assist in decisionmaking rather than as a “cookbook.” Any references within thedocument to specific education products are illustrative and do not imply endorse-ment of these products to the exclusion of other products that are not referenced.U.S. Department of EducationArne DuncanSecretaryInstitute of Education SciencesJohn Q. EastonDirectorNational Center for Education Evaluation and Regional AssistancePhoebe CottinghamCommissionerJuly 2009This report is in the public domain. Although permission to reprint this publicationis not necessary, the citation should be:Beckett, M., Borman, G., Capizzano, J., Parsley, D., Ross, S., Schirm, A., & Taylor, J.(2009). Structuring out-of-school time to improve academic achievement: A practiceguide (NCEE #2009-012). Washington, DC: National Center for Education Evaluationand Regional Assistance, Institute of Education Sciences, U.S. Department of Education.Retrieved from http://ies.ed.gov/ncee/wwc/publications/practiceguides.What Works Clearinghouse Practice Guide citations begin with the panel chair, fol-lowed by the names of the panelists listed in alphabetical order.This report is available on the IES website at http://ies.ed.gov/ncee and http://ies.ed.gov/ncee/wwc/publications/practiceguides.Alternate FormatsOn request, this publication can be made available in alternate formats, such asBraille, large print, audiotape, or computer diskette. For more information, call theAlternate Format Center at 202–205–8113.
  5. 5. Structuring Out-of-School Timeto Improve Academic AchievementContentsIntroduction 1 The What Works Clearinghouse Standards and their relevance to this guide 2Overview 5Scope of the practice guide 7Status of the research 9Summary of the recommendations 9Checklist for carrying out the recommendations 11Recommendation 1. Align the OST program academically withthe school day 12Recommendation 2. Maximize student participation and attendance 19Recommendation 3. Adapt instruction to individual and smallgroup needs 24Recommendation 4. Provide engaging learning experiences 29Recommendation 5. Assess program performance and use theresults to improve the quality of the program 34Appendix A. Postscript from the Institute of Education Sciences 38Appendix B. About the authors 41Appendix C. Disclosure of potential conflicts of interest 44Appendix D. Technical information on the studies 45References 86 ( iii )
  6. 6. STRuCTuRIng OuT-Of-SChOOl TIME TO IMPROvE ACADEMIC AChIEvEMEnTList of tablesTable 1. Institute of Education Sciences levels of evidencefor practice guides 4Table 2. Recommendations and corresponding levels of evidence 10Table D1. Studies of OST programs that met WWC standardswith or without reservations 46Table D2. Studies and corresponding recommendations 48Table D3. Studies of programs cited in recommendation 1that met WWC standards with or without reservations 51Table D4. Studies of programs cited in recommendation 2that met WWC standards with or without reservations 58Table D5. Studies of programs cited in recommendation 3that met WWC standards with or without reservations 66Table D6. Studies of programs cited in recommendation 4that met WWC standards with or without reservations 74Table D7. Studies of programs cited in recommendation 5that met WWC standards with or without reservations 82List of exhibitsExhibit 1. Sample logbook 15 ( iv )
  7. 7. Introduction when possible. The research base for this guide was identified through a comprehen-This guide is intended to help educators, sive search for studies evaluating academi-out-of-school time (OST) program pro- cally oriented OST interventions and prac-viders, and school and district adminis- tices. An initial search for research on OSTtrators structure academically focused programs conducted in the United Statesout-of-school time programs. OST is an in the past 20 years (1988–2008) yieldedopportunity to supplement learning from more than 1,000 studies. Of these, 130the school day and provide targeted as- studies examined school-based OST pro-sistance to students whose needs extend grams that serve elementary and middlebeyond what they can receive in the class- school students and were eligible for fur-room. With an increasing focus on school ther review. These studies were reviewedaccountability and student performance, by the WWC to determine whether theyOST can play a meaningful role in improv- were consistent with WWC standards. Ofing academic achievement and closing the the 130 studies, 22 met WWC standardsgap between low- and high-performing or met the standards with reservations.students. Although OST programs oper- These 22 studies of 18 different OST pro-ate nationwide, disagreement about which grams represent the strongest evidence ofaspects of these programs are beneficial the effectiveness of OST programs.2for student achievement remains. Thispractice guide includes concrete recom- In keeping with the WWC standards formendations for structuring an effective determining levels of evidence, the panelacademically oriented OST program, and it relied on the following definitions (seeillustrates the quality of the evidence that Table 1):supports these recommendations. Theguide also acknowledges possible imple- A strong rating refers to consistent andmentation challenges and suggests solu- generalizable evidence that an inter-tions for circumventing the roadblocks. vention strategy or program improves outcomes.3A panel of experts in OST programs andresearch methods developed the recom- A moderate rating refers either to evidencemendations in this guide and determined from studies that allow strong causal con-the level of evidence for each recommen- clusions but cannot be generalized withdation. The evidence considered in de- assurance to the population on which aveloping this guide ranges from rigorous recommendation is focused (perhaps be-evaluations of OST programs to expert cause the findings have not been widelyanalyses of practices and strategies in OST. replicated) or to evidence from studies thatIn looking for effective practices, the panel are generalizable but have more causalpaid particular attention to high-qual- ambiguity than that offered by experimentality experimental and quasi-experimental designs (e.g., statistical models of correla-studies, such as those meeting the criteria tional data or group comparison designs forof the What Works Clearinghouse (WWC),1and to patterns of practices that are repli-cated across programs. 2. See Table D2 for a summary of which studies are relevant to each recommendation.As with all WWC practice guides, the rec- 3. Following WWC guidelines, improved out- comes are indicated by either a positive statisti-ommendations in this guide are derived cally significant effect or a positive, substantivelyfrom and supported by rigorous evidence, important effect size (i.e., greater than 0.25). See the WWC guidelines at http://ies.ed.gov/ncee/1. http://www.whatworks.ed.gov/. wwc/pdf/wwc_version1_standards.pdf. (1)
  8. 8. InTRODuCTIOnwhich equivalence of the groups at pretest although some of these programs haveis uncertain). had rigorous evaluations of their impacts, others have not. Furthermore, some of theA low rating refers to expert opinion based programs that have been rigorously evalu-on reasonable extrapolations from re- ated have found positive effects on aca-search and theory on other topics and demic achievement; others have not.5evidence from studies that do not meetthe standards for moderate or strong The What Works Clearinghouseevidence. standards and their relevance to this guideIt is important for the reader to remem-ber that the level of evidence rating is not In terms of the levels of evidence indi-a judgment by the panel on how effective cated in Table 1, the panel relied on WWCeach of these recommended practices will Evidence Standards to assess the quality ofbe when implemented, nor is it a judg- evidence supporting educational programsment of what prior research has to say and practices. WWC addresses evidenceabout their effectiveness. The level of evi- for the causal validity of instructional pro-dence ratings reflect the panel’s judgment grams and practices according to WWCof the quality of the existing literature to standards. Information about these stan-support a causal claim that when these dards is available at http://ies.ed.gov/ncee/practices have been implemented in the wwc/pdf/wwc_version1_standards.pdf. Thepast, positive effects on student academic technical quality of each study is rated andoutcomes were observed. They do not re- placed into one of three categories:flect judgments of the relative strength ofthese positive effects or the relative impor- • Meets Evidence Standards for random-tance of the individual recommendations. ized controlled trials and regressionThus, a low level of evidence rating does discontinuity studies that provide thenot indicate that the recommendation is strongest evidence of causal validity.any less important than other recommen-dations with a strong or moderate rating. • Meets Evidence Standards with Res-Rather, it suggests that the panel cannot ervations for all quasi-experimentalpoint to a body of research that demon- studies with no design flaws and ran-strates its effect on student achievement. domized controlled trials that haveIn some cases, this simply means that the problems with randomization, attri-recommended practices would be diffi- tion, or disruption.cult to study in a rigorous, experimentalfashion; in other cases, it means that re- • Does Not Meet Evidence Screens forsearchers have not yet studied this prac- studies that do not provide strong evi-tice, or that there is ambiguous evidence dence of causal validity.of effectiveness.4 Following the recommendations and sug-Citations in the text refer to studies of gestions for carrying out the recommen-programs that have implemented vari- dations, Appendix D presents more in-ous practices. Not all of these programs formation on the research evidence thatcontribute to the level of evidence rating: supports each recommendation.4. For more information, see the WWC FrequentlyAsked Questions page for practice guides, http:// 5. Table D1 summarizes the details and effective-ies.ed.gov/ncee/wwc/references/idocviewer/ ness of studies consulted for the evidence ratingDoc.aspx?docId=15&tocId=3. of this guide. (2)
  9. 9. InTRODuCTIOnWe appreciate the efforts of Samina Sat- their time and expertise to the reviewtar, Virginia Knechtel, Liz Potamites, and process. We would like to thank KristinClaire Smither, MPR staff members who Hallgren, Scott Cody, Shannon Monahan,participated in the panel meetings, charac- and Mark Dynarski for their oversight andterized the research findings, and drafted guidance during the development of thethe guide. We also appreciate the help of practice guide, and for helpful feedbackthe many WWC reviewers who contributed and reviews of its earlier versions. Megan Beckett Geoffrey Borman Jeffrey Capizzano Danette Parsley Steven Ross Allen Schirm Jessica Taylor (3)
  10. 10. InTRODuCTIOnTable 1. Institute of Education Sciences levels of evidence for practice guides In general, characterization of the evidence for a recommendation as strong requires both studies with high internal validity (i.e., studies whose designs can support causal conclu- sions) and studies with high external validity (i.e., studies that in total include enough of the range of participants and settings on which the recommendation is focused to sup- port the conclusion that the results can be generalized to those participants and settings). Strong evidence for this practice guide is operationalized as • A systematic review of research that generally meets WWC standards (see http://ies. ed.gov/ncee/wwc/) and supports the effectiveness of a program, practice, or approach Strong with no contradictory evidence of similar quality; OR • Several well-designed, randomized controlled trials or well-designed quasi-experi- ments that generally meet WWC standards and support the effectiveness of a program, practice, or approach, with no contradictory evidence of similar quality; OR • One large, well-designed, randomized controlled, multisite trial that meets WWC stan- dards and supports the effectiveness of a program, practice, or approach, with no contradictory evidence of similar quality; OR • For assessments, evidence of reliability and validity that meets the Standards for Educational and Psychological Testing.a In general, characterization of the evidence for a recommendation as moderate requires studies with high internal validity but moderate external validity or studies with high external validity but moderate internal validity. In other words, moderate evidence is derived from studies that support strong causal conclusions but generalization is uncer- tain or studies that support the generality of a relationship but the causality is uncertain. Moderate evidence for this practice guide is operationalized as • Experiments or quasi-experiments generally meeting WWC standards and supporting the effectiveness of a program, practice, or approach with small sample sizes and/ or other conditions of implementation or analysis that limit generalizability and no contrary evidence; OR Moderate • Comparison group studies that do not demonstrate equivalence of groups at pretest and, therefore, do not meet WWC standards but that (1) consistently show enhanced outcomes for participants experiencing a particular program, practice, or approach and (2) have no major flaws related to internal validity other than lack of demonstrated equivalence at pretest (e.g., only one teacher or one class per condition, unequal amounts of instructional time, highly biased outcome measures); OR • Correlational research with strong statistical controls for selection bias and for dis- cerning influence of endogenous factors and no contrary evidence; OR • For assessments, evidence of reliability that meets the Standards for Educational and Psychological Testingb but with evidence of validity from samples not adequately repre- sentative of the population on which the recommendation is focused. In general, characterization of the evidence for a recommendation as low means that the recommendation is based on expert opinion derived from strong findings or theories in Low related areas and/or expert opinion buttressed by direct evidence that does not rise to the moderate or strong level. Low evidence is operationalized as evidence not meeting the standards for the moderate or high level.a. American Educational Research Association, American Psychological Association, and National Council on Measurement in Education (1999).b. Ibid. (4)
  11. 11. Structuring out-of- OST programs offer a promising approachschool time to improve to enhancing students’ academic skills and to closing the achievement gap. In recog-academic achievement nition of this promise, funding for OST programs and services has grown in theOverview past few years. As part of the No Child Left Behind (NCLB) legislation, districts are re-Over the past three decades, changing quired to spend 5 percent to 20 percentlabor force patterns in the United States of all Title I funds on supplemental educa-have significantly increased the need for tional services (SES).10 Further, some stateschild care for school-age children. In 2000, expanded funding for OST programs, evennearly half of school-age children with in the face of reduced budgets. For exam-working mothers spent time in non-pa- ple, in 2002, California voters approvedrental supervised settings when they were the addition of approximately a half bil-not in school, including before- and after- lion dollars annually to existing state after-school programs, family child care homes, school programs.11and the homes of relatives.6 Commonlyknown as out-of-school time (OST), this Similarly, the number of OST programs, in-period outside of the school day when cluding after-school, weekend, and summerchildren are not with their parents has re- programs and SES, has been increasing. Inceived extensive policy attention, focused 1995, a U.S. Census Bureau study of childon both the risks of negative influences care arrangements showed that 5.6 percentduring this time and the potential benefits of children ages 5 to 14 received care in athe time holds for the positive develop- before- or after-school program accord-ment of school-age children. ing to parents in the sample.12 In 2005, 20 percent of K–8 students participated in aAlthough many OST settings are designed before- or after-school program.13primarily to provide a safe place for chil-dren to be outside of the traditional schoolday while parents work, there is now a 10. Supplemental educational services (SES) are tutoring or other academic support services of-broader movement toward using OST to fered outside the regular school day, at no chargebridge the gap between high- and low- to students or their families, by public or privateachieving students and to give students providers that have been approved by the state.more time to learn if they need it.7 Aca- Districts are required to offer SES to low-incomedemically oriented out-of-school programs students in schools that have fallen short of adequate yearly progress (AYP) standards for aand services are promising because stu- third time (after missing AYP for two consecutivedents spend twice as much of their wak- years). Students and their parents are permitteding hours outside of the classroom as in to choose among state-approved SES providers,it,8 and OST periods, especially summer which come in all varieties, including national for-profit firms, local nonprofits, faith-basedbreaks, are the times when the achieve- organizations, institutions of higher education,ment gap widens.9 and local school districts (which are permitted to become approved providers unless they are themselves identified for improvement under No Child Left Behind [NCLB]). 11. Administration for Children & Families (n.d.).6. Capizzano, Tout, and Adams (2000). State afterschool profiles: California. Washington,7. Halpern (1999). DC: U.S. Department of Health and Human Ser-8. Hofferth and Sandberg (2001). vices. Retrieved May 29, 2009, from http://nccic. org/afterschool/ca.html.9. Heyns (1978); Alexander, Entwisle, and Olson(2007a, 2007b); Downey, von Hippel, and Broh 12. Smith (2000).(2004); Cooper et al. (1996). 13. Carver and Iruka (2006). (5)
  12. 12. OvERvIEWAlthough it is generally assumed that OST academically; that time may need to beprograms can provide students with posi- carefully orchestrated to facilitate learn-tive, academically enriching experiences, ing and retention of academic material.it is not necessarily known how to struc- Additionally, the average amount of totalture programs to effectively improve stu- instructional time received by students indent academic outcomes. Although many a typical OST program may be too low tostudies lacking comparison groups sug- generate meaningful academic effects.17gest that OST programs can benefit stu-dents academically,14 those with more rig- The findings from the evaluation of En-orous evaluation designs raise questions hanced Academic Instruction in After-about these findings. For example, find- School Programs, sponsored by the Insti-ings from the national evaluation of the tute of Education Sciences (IES), provide21st Century Community Learning Centers some evidence for what works in OST in-(CCLC) program, which is the largest after- struction.18 The elementary school pro-school program in the United States, show grams delivered school-day math andthat, on average, students participating in reading curricula adapted to after-schoolthe programs had no improvement in aca- settings.19 Students, who received an av-demic achievement.15 erage of 57 hours of enhanced math in- struction (more than the 30–40 hours SESThe evaluation found that 21st CCLC pro- students might receive20), had modest butgrams were not consistently focused on statistically significant improvements inacademics and often placed more empha- math achievement after one year comparedsis on sports or extracurricular activities with students in a regular after-school pro-because they thought those activities were gram.21 No differences were found betweenmore popular with students and would students who received enhanced readingencourage participation in the program.16 instruction and those in a regular after-Students in 21st CCLC programs may not school program. This first year of findingshave been spending enough time engaged provides some indication that instructionin academic content to produce measurable in OST can improve student achievementgains in achievement. Simply adding time when delivered in a structured, focusedto students’ days may not benefit them format with adequate dosage.14. Center for Applied Linguistics (1994); Fash- 17. Kane (2004).ola (1998); Ferreira (2004); Sheldon and Hopkins 18. Black et al. (2008).(2008). 19. Ibid.15. U.S. Department of Education (2003). 20. Ross et al. (2008).16. James-Burdumy, Dynarski, and Deke (2007). 21. Black et al. (2008). (6)
  13. 13. Scope of the to improve academic achievement canpractice guide benefit from this guide. State education agencies may find the recommendations useful for assessing the quality of prospec-The purpose of this practice guide is to tive OST programs such as SES providersprovide recommendations for organiz- or 21st CCLC programs. Other types ofing and delivering school-based OST pro- programs, such as before-school or non–grams to improve the academic achieve- school-based programs also may benefitment of student participants. School-based from the recommendations in this guide.programs include those that are adminis- However, these programs were not thetered by a school or school district, as well focus of the panel’s discussions.as programs that are contracted by theschool or school district and provided by The panel assumed that the basic structureother organizations. The panel has limited of an academically focused OST programthe scope of this practice guide to pro- would include the following components:grams that (1) serve elementary and mid-dle school students, (2) are organized by • a place to meet—often this is the school,or conducted in partnership with a school but it also may be a community centeror school district, and (3) aim to improve or other facilityacademic outcomes. • regular hours of operationThe structure and objectives of OST pro-grams vary. Some exist as a place where • transportation (if necessary)students can be safe and occupied for thetime from school dismissal until parents • administrative and instructional staffare able to pick them up. Other programsare designed to provide social and cultural • instructional materials or curriculaenrichment opportunities or to promotehealthy outcomes such as grade promotion Staffing needs will vary by program butand reduction in risky behaviors. This guide typically include a program director whotargets programs whose primary goal is to supervises the operation of the programprovide academic instruction to improve (or program sites if there are multiple loca-participants’ achievement.22 The guide tar- tions) and manages OST instructors. Thegets the following types of programs: OST instructors are the front-line staff members who interact with students. Ad-• after-school or weekend programs ditionally, either the OST program or a school may employ a coordinator who is• summer school responsible for maintaining the relation- ships among the school, the program site,• SES and other partners (see recommendation 1). School districts that have contractedTeachers, principals, district administra- with multiple OST programs may employtors, and other staff who seek guidance for a district coordinator to monitor all pro-structuring these types of OST programs grams across schools. The evidence base in this guide for the22. In a meta-analysis of 73 after-school pro- effectiveness of OST programs and ser-grams, Durlak and Weissberg (2007) found that“the presence of an academic component” was vices on student achievement is based onthe largest predictor of a program producing what is known about after-school, summersignificant academic improvements. (7)
  14. 14. SCOPE Of ThE PRACTICE guIDEschool, and SES programs targeting el- administrators will make that decisionementary and middle school students in based on their available resources and thelow-income, high-needs communities. needs and preferences of the communitiesAlthough many of the recommendations they serve. They may decide that one or amay look similar to those for high school combination of these types of programsprograms, the objectives of OST programs will suit their objectives.may differ in high school, and the actionsteps and roadblocks will be different Second, the guide does not address specificbecause of the wider range of activities, instructional practices or teaching strate-transportation issues, and instructional gies that are unique to reading, math, orneeds of high school students compared other content areas. The research basewith those of younger students. Similarly, that would be required to support con-although the recommendations may be ap- tent-specific recommendations is beyondplicable to other populations, the literature the scope of this guide. For reference, theused in this guide largely looks at students What Works Clearinghouse (WWC) has pub-in urban, low-income, and low-achieving lished a number of practice guides andschools. Table D1 in Appendix D includes intervention reports devoted to content-details on the populations studied by the specific areas such as adolescent literacy,literature mentioned in this guide. beginning reading, and elementary and middle school math.23The recommendations in this guide can beused singly or in combination. The panel Third, the guide does not provide detailsrecommends that readers consider imple- on the costs of organizing or operating OSTmenting all recommendations, but any one programs or of implementing the recom-recommendation can be implemented inde- mendations of the panel. The panel rec-pendently. For example, recommendations ognizes that cost is a huge element in the1 and 2 will be most useful to school ad- decisions that programs make about theministrators who want to address low stu- services they provide, but also that costsdent attendance and ensure that students can vary considerably by factors such aswho need academic help will benefit from the type of program, days of operation, andthe OST program. Recommendations 3 and geographical area.24 Some of the roadblocks4 will be useful for teachers who are strug- found at the end of each recommendationgling with addressing the academic needs provide suggestions for ways to minimizeof their students. Recommendation 5 is costs, but the panel directs readers to re-intended for school, district, and program cent reports by Public/Private Ventures (P/administrators who want to ensure that the PV), RAND, and others for more informationout-of-school programming offered to stu- on the costs of OST programs.25dents is of high quality and that studentsare benefiting from those services. Finally, the guide does not address behav- ioral management in the OST context. TheIn writing the guide, the panel chose not panel acknowledges that in the OST arena,to address the following four areas: as in the school-day classroom, programsFirst, the guide does not address whether 23. Institute of Education Sciences. (n.d.). Whatto provide services after school, before Works Clearinghouse: Practice Guides. Washing-school, on weekends, or during the sum- ton, DC: U.S. Department of Education. Retrievedmer months. The panel does not believe May 29, 2009, from http://ies.ed.gov/ncee/wwc/that there is a consensus in the literature publications/practiceguides.about how the timing of a program affects 24. Grossman et al. (2009).student outcomes. The panel assumes that 25. Grossman et al. (2009); Beckett (2008). (8)
  15. 15. SCOPE Of ThE PRACTICE guIDEand practices can impact student behavior. educators and OST providers alike withIn OST, this includes the sites participat- more definitive information about effec-ing in the national evaluation of 21st CCLC, tive practices.which were found to have an adverse ef-fect on student behavioral outcomes.26 Al- In offering these recommendations, thethough the panel recognizes that creating panel reminds readers that the evidencea safe and orderly environment is a neces- base in support of these recommenda-sary condition for students to learn, inter- tions, when available, is based on expe-ventions that address behavioral manage- riences in OST programs. The relativelyment issues were judged to be out of the small amount of literature on academicallyscope of this guide and recommendations focused programs limited the depth of in-that are targeted to academic improve- formation on effective instructional prac-ment. The panel directs readers to other tices in the OST context. To account forpublications that may be helpful in this this, the panel also incorporated relevantarea, including the WWC practice guide, literature in the broader education fieldReducing Behavior Problems in the Elemen- and used its expert judgment to identifytary School Classroom.27 practices that strengthened the OST learn- ing environment.Status of the research Summary of the recommendationsOverall, the panel believes that the exist-ing research on OST practices is not at a This practice guide offers five recommen-level to provide conclusive evidence of dations to improve the ability of OST pro-best practices. Studies of OST programs grams to benefit students academicallytend to examine combined effects of a va- (see Table 2). Recommendations 1 and 2riety of practices and procedures on stu- address how to design an OST program bydent achievement, making it difficult to considering its relationship with schoolsdetermine the specific practices contrib- and the components that can maximizeuting to achievement gains.28 Likewise, the appeal of the program. Recommenda-the panel encountered varying impacts tions 3 and 4 focus on the delivery of aca-across OST programs with ostensibly simi- demic instruction, and how it can be usedlar practices. Many studied interventions purposely to improve student engage-are practiced on a small scale, necessitat- ment and performance. Recommendationing a small sample size and often making 5 addresses evaluation of OST programs,it difficult to find an appropriate compari- which is essential for maintaining highson group. Low levels of participation or standards of quality as well as continuousattendance, even in large-scale programs, improvement of the program design andalso make it difficult to interpret evalua- instruction.tion results. The panel believes that theOST field would benefit from additional, Recommendations 1 and 2 (Design). OSTrigorous research on OST programs that programs should include design featuresserve a large number of students and have that ultimately strengthen academic prog-achieved high levels of participation. Im- ress while fulfilling the needs of parentsproving the research base will provide and students. Recommendation 1 empha- sizes the importance for OST programs to26. James-Burdumy, Dynarski, and Deke (2008). connect with school and classroom activi-27. Epstein et al. (2008). ties to achieve a shared mission of improv-28. In these cases, the panel members exercised ing academic performance. Further, thetheir expert judgment to identify practices likely panel recognizes that sometimes studentsto produce achievement gains for students. (9)
  16. 16. SCOPE Of ThE PRACTICE guIDEmost likely to benefit from a strong aca- engaging. The panel believes it is particu-demic program may be especially unlikely larly important to engage students whento enroll in or attend OST programs regu- they may be fatigued after a long day oflarly. Thus, the panel recommends that school; on Saturdays; during the summerOST programs focus on recruiting and months; or when they are drawn to partici-retaining targeted students so that they pate in other, nonacademic activities. Toreceive the dosage necessary to realize avoid the pitfalls of other programs thatacademic benefits (recommendation 2). failed to demonstrate positive academic ef- fects, the panel suggests that all activitiesRecommendations 3 and 4 (Instruction). To have a specific learning objective.maximize the educational benefits for stu-dents, OST programs should deliver aca- Recommendation 5 (Evaluation). Finally,demic instruction in a way that responds program improvement depends on theto each student’s needs and engages them articulation of goals and expectations, ef-in learning. Recommendation 3 presents fective management, and the performancestrategies for the structuring of instruc- and experience of staff. Recommenda-tional practices and program content to ad- tion 5 presents strategies for schools anddress the needs of students and effectively districts to use to identify programs thatimprove academic outcomes. The recom- are most likely to result in academic im-mendation provides suggestions for orga- provement and to monitor existing pro-nizing instructional time in the classroom grams to ensure that the highest-qualityand for facilitating individualized teaching services are being provided to students.by assessing student needs. Recommenda- The earlier recommendations in this guidetion 4 encourages OST programs to capital- should provide programs with a solidize on programming flexibility by offering starting point from which to evaluate anactivities that students may find especially OST program.Table 2. Recommendations and corresponding levels of evidence Recommendation Level of evidence Design 1. Align the OST program academically with the school day. Low 2. Maximize student participation and attendance. Low Instruction 3. Adapt instruction to individual and small group needs. Moderate 4. Provide engaging learning experiences. Low Evaluation 5. Assess program performance and use the results to improve the qual- Low ity of the program. ( 10 )
  17. 17. Checklist for carrying out the Recommendation 3.recommendations Adapt instruction to individual and small group needs.Recommendation 1.Align the OST program academically  formal and informal assessment usewith the school day. data to inform academic instruction. use OST program coordinators to de-  use one-on-one tutoring if possi-velop relationships and maintain ongoing ble; otherwise, break students into smallcommunication between schools and the groups.OST program.  Provide professional development Designate a school staff person to and ongoing instructional support to allcoordinate communication with OST pro- instructors.grams and help them support schoolneeds. Recommendation 4. Provide engaging learning Connect OST instruction to school experiences.instruction by identifying school-basedgoals and learning objectives.  learning relevant by incorpo- Make rating practical examples and connect- Coordinate with the school to identify ing instruction to student interests andstaff for OST programs. experiences.Recommendation 2.  learning active through op- MakeMaximize student participation portunities for collaborative learning andand attendance. hands-on academic activities. Design program features to meet the  Build adult-student relationshipsneeds and preferences of students and among OST program participants.parents. Recommendation 5. Promote awareness of the OST pro- Assess program performance and usegram within schools and to parents. the results to improve the quality of the program. attendance data to identify stu- usedents facing difficulties in attending the  Develop an evaluation plan.program.  Collect program and student perfor- mance data.  Analyze the data and use findings for program improvement.  Conduct a summative evaluation. ( 11 )
  18. 18. Recommendation 1. outcomes. Although it was common forAlign the OST program programs to include some components of the panel’s recommendations, none testedacademically with the effectiveness of this recommendationthe school day individually, only in combination with the other components of OST programs.The panel believes that academic In the panel’s opinion, collaboration canalignment with the school day is improve academic outcomes and in thenecessary for OST programs to improve studies reviewed for this guide, two in-academic performance. OST programs dependent evaluators recommended thatand schools have the shared mission of collaboration between schools and OSThelping students achieve success, and programs be strengthened if possible.31collaboration between the two can be However, we acknowledge that more re-mutually beneficial. Although alignment search is required to demonstrate the ef-requires additional effort from staff, fects of stronger alignment.teachers and principals in schools withexisting OST programs have voiced Brief summary of evidence tosupport for this sort of collaboration.29 support the recommendation Fifteen OST programs endeavored toAn OST program coordinator can collaborate with school-based staff orensure alignment through regular initiatives,32 but, in general, these effortscommunication with school staff, were not core components of the pro-and schools can help by designating grams. Three programs also expresseda school-based coordinator to work difficulty or reluctance to coordinate morewith the OST coordinator. This sort fully.33 Of the 11 programs with studiesof cooperation helps OST programsevaluate their students’ needs and 31. Schacter and Jo (2005) and Center for Appliedprovide the most effective instruction Linguistics (1994) suggested the use of more col-and services. Both the program and laboration when appropriate.the school-based coordinators should 32. Challenging Horizons Program (CHP)—Lang- berg et al. (2006); Early Risers—August et al.work to align the instructional activities (2001); Enhanced Academic Instruction—Black etof the OST program with state and al. (2008); Teach Baltimore—Borman and Dowl-local content standards, the school ing (2006); Chicago Summer Bridge—Jacob andcurriculum, and district- and/or school- Lefgren (2004); Los Angeles’s Better Educated Stu-based learning initiatives.30 dents for Tomorrow (L.A.’s BEST)—Goldschmidt, Huang, and Chinen (2007); Youth Services—Child Care, Academic Assistance, Recreation, and En-Level of evidence: Low richment (YS-CARE)—Bissell et al. (2002); 21st CCLC—U.S. Department of Education (2003); LeapThe level of evidence for this recommen- Frog—McKinney (1995); Nurturing Development Partnerships (NDP)—Udell (2003); SES—McKay etdation is low. There is no direct evidence al. (2008); SES—Ross et al. (2008); SES—Muñoz,that practices outlined in this recommen- Potter, and Ross (2008); Title I supplementarydation contribute to improved academic education—Borman (1997); The After-School Corporation (TASC)—Reisner et al. (2004); Project Adelante—Center for Applied Linguistics (1994);29. Goldschmidt, Huang, and Chinen (2007); Bis- After-school tutoring—Leslie (1998).sell et al. (2002). 33. In James-Burdumy et al. (2005), the authors30. Borman and Dowling (2006); Langberg et al. noted that 21st CCLC programs struggled to effec-(2006); Roderick, Jacob, and Bryk (2004); Borman tively coordinate homework help with the school;(1997). and in U.S. Department of Education (2003), 21st ( 12 )
  19. 19. 1. AlIgn ThE OST PROgRAM ACADEMICAlly WITh ThE SChOOl DAythat met WWC standards with or with- program found significant and persistentout reservations,34 three programs docu- effects on both math and reading for 3rdmented practices that closely corresponded graders but not for 6th graders.39to the panel’s recommendations.35 In twoof these, coordination between school- The remaining eight programs includedteachers and OST instructors was frequent some components similar to this recom-and structured.36 Content and skills taught mendation, but studies indicated that theduring OST were intentionally designed to degree of coordination was lower or notsupport students during their school-day enough information was provided to de-instruction. One program showed positive termine the level of alignment betweenacademic effects,37 and the other did not.38 programs.40 Of these, one showed positiveThe purpose of the third program, which effects,41 two showed mixed effects,42 andwas a summer school, was to help stu- five programs showed no detectable aca-dents achieve proficiency on state exami- demic effects.43 Although more of thesenations they had not mastered during the programs failed to demonstrate effective-school year. The curriculum was designed ness, Table D2 shows that there were sevenby the district with the express purpose effective programs that did not attempt co-of helping students meet state standards ordination with schools. Given the absenceand was closely linked to that goal, but of a clear pattern of effectiveness based onby nature of its being a summer school the level of coordination with schools andprogram, coordination with individual the small sample of programs with highteachers was limited. The evaluation of the levels of coordination, the panel decided that the level of evidence was low.CCLC programs were found to be supportivebut not “integrated” (p. 39) with the school. InProject Adelante (Center for Applied Linguistics1994), program directors recommended closercoordination with school-day staff to collect dataand share information on student progress butwere concerned about the appropriateness of theschool-day curriculum for the students that theirprogram served. Similarly, Morris, Shaw, and Per-ney (1990) expressed reluctance to align theirafter-school tutoring program, Howard StreetTutoring, to the school curriculum given thattheir students’ classroom instruction was often 39. Chicago Summer Bridge—Jacob and Lefgrenbeyond the students’ current reading levels. (2004).34. CHP—Langberg et al. (2006); Early Risers— 40. Early Risers—August et al. (2001); EnhancedAugust et al. (2001); Enhanced Academic Instruc- Academic Instruction—Black et al. (2008); Teachtion—Black et al. (2008); Teach Baltimore—Borman Baltimore—Borman and Dowling (2006); L.A.’sand Dowling (2006); Chicago Summer Bridge—Jacob BEST—Goldschmidt, Huang, and Chinen (2007);and Lefgren (2004); L.A.’s BEST—Goldschmidt, YS-CARE—Bissell et al. (2002); 21st CCLC—U.S.Huang, and Chinen (2007); YS-CARE—Bissell et al. Department of Education (2003); NDP—Udell(2002); 21st CCLC—U.S. Department of Education (2003); SES—McKay et al. (2008); SES—Ross et al.(2003); Leap Frog— McKinney (1995); NDP—Udell (2008); SES—Muñoz, Potter, and Ross (2008).(2003); SES—McKay et al. (2008); SES—Ross et al. 41. Early Risers—August et al. (2001).(2008); SES—Muñoz, Potter, and Ross (2008). 42. Enhanced Academic Instruction—Black et35. CHP—Langberg et al. (2006); Chicago Sum- al. (2008); Teach Baltimore—Borman and Dowl-mer Bridge—Jacob and Lefgren (2004); Leap ing (2006).Frog—McKinney (1995). 43. L.A.’s BEST—Goldschmidt, Huang, and Chinen36. CHP—Langberg et al. (2006); Leap Frog— (2007); YS-CARE—Bissell et al. (2002); 21st CCLC—McKinney (1995). U.S. Department of Education (2003); NDP—Udell37. CHP—Langberg et al. (2006). (2003); SES—Ross et al. (2008); SES—McKay et al.38. Leap Frog—McKinney (1995). (2008); SES—Muñoz, Potter, and Ross (2008). ( 13 )
  20. 20. 1. AlIgn ThE OST PROgRAM ACADEMICAlly WITh ThE SChOOl DAyHow to carry out this will reinforce and complement the schoolrecommendation curriculum.45 Regular communication can help identify the needs and strengths of in-1. use OST program coordinators to develop dividual students and those strategies thatrelationships and maintain ongoing com- are most effective in raising achievement.munication between schools and the OST Some examples of steps follow:program. • The OST coordinator can develop a log-An OST program coordinator can play book that students carry back and fortha critical role in ensuring that instruc- daily to share information about OST ac-tional components of an OST program are tivities with classroom teachers.46 Thealigned with the school day. Coordinators logbook can contain information fromshould work directly with teachers and classroom teachers about homeworkadministrators from the school to obtain assignments, concepts the student isinformation that can be used to guide in- struggling with during the school day,struction in the OST program. This can or the instructional strategies that arebe accomplished through regular com- most effective with the student.47 Themunication with key school staff, and also coordinator should work closely withthrough participation in school meetings school staff in developing this logbook.and committees. For example, coordina- For a sample logbook, see Exhibit 1.48tors can attend staff meetings, participatein common planning periods, serve on • The OST coordinator can arrange forschool leadership teams, and participate OST instructors to periodically attendin parent-teacher organizations. OST co- common planning periods with class-ordinators also can promote the OST pro- room teachers to align programminggram to staff and families by posting in- or collaborate with effective teachersformation about the program on bulletin to identify best practices and materi-boards or holding OST events during the als for meeting curriculum goals andschool day to expose other students to the raising student achievement.49OST program. • The OST coordinator can collaborateWhen possible, the OST coordinator should with school-based staff to identify rele-be housed within the school, spending vant professional development that OSTtime during daily school hours to be visi- instructors can attend with schoolteach-ble to both students and teachers. The OST ers to align instructional strategies andcoordinator can use these types of oppor- to provide funding when possible.tunities to maintain an open relationshipwith teachers, principals, and counselors, 45. Leslie (1998); U.S. Department of Educationadvocating for and gathering data about (2003).OST students as necessary.44 46. Morris, Shaw, and Perney (1990). 47. Langberg et al. (2006).The OST coordinator can take key steps to 48. For other examples of logbooks, see SEDLfacilitate regular communication between National Center for Quality Afterschool (n.d.) andOST instructors and classroom teachers that Region VII After School Programs (n.d.). 49. Bott (2006) described the use of this strategy in the Gardner Extended Services School; U.S.44. Center for Applied Linguistics (1994). Department of Education (2003). ( 14 )
  21. 21. 1. AlIgn ThE OST PROgRAM ACADEMICAlly WITh ThE SChOOl DAyExhibit 1. Sample logbook Information from the Classroom Teacher Today’s Date Student attended class?  Yes  No Tardy  By subject: Subject 1a Subject 2 Subject 3 Subject 4 Topics covered in class today Today’s homework assignment Areas in which the student needs additional help Instructional strategies that were useful Behavior or discipline issues Other comments Information from the OST Instructor Today’s Date Student attended  Yes  No Tardy  OST program? By subject: Subject 1 Subject 2 Subject 3 Subject 4 Percentage of homework completed during OST Homework items that were challenging for the student Topics covered during OST instruction Instructional strategies or activities that were useful Behavior or discipline issues Other commentsa. This form can be used for single subject classrooms or elementary classrooms. ( 15 )
  22. 22. 1. AlIgn ThE OST PROgRAM ACADEMICAlly WITh ThE SChOOl DAy2. Designate a school staff person to coordi- 3. Connect OST instruction to school instruc-nate communication with OST programs and tion by identifying school-based goals andhelp them support school needs. learning objectives.Schools can designate a staff member (the Information gathered from the school andschool-based coordinator) to work with the the district can be used to prioritize effortsOST program coordinator (or with OST co- to raise academic achievement and sup-ordinators from multiple OST programs, port student learning during the schoolif relevant). The panel believes that when day. OST programs should align activi-OST programs are well aligned to school- ties, instruction, and any formal curricu-day goals and instruction, they can support lum with the state and local standards, asthe school in raising student achievement. well as the content and curriculum of theA school-based coordinator can serve as a school day.50 State and local curriculumfirst point of contact in the school for OST standards are often available online, butprograms and can help ensure that OST in- school and district officials also should di-struction is well aligned with school goals. rect OST staff to relevant resources suchThis person will play an important role as school improvement plans or other spe-in program-school relations, but it is not cific school- or district-based objectives.imperative that the position is full time, The OST program need not repeat class-and an existing teacher or counselor may room instruction, but it can use differentbe suited to this role. Key functions of the methods to support and reinforce whatschool-based staff designee include students learn in school.• Developing a set of standard operat- OST programs can help students develop ing procedures for distributing student skills that support classroom instruction, data to OST program staff. Relevant such as learning how to plan, take notes, data can include results from district- develop an outline, or study for an upcom- or state-wide standardized testing, ing test. For example, OST providers could student progress reports from teach- explicitly teach a skill and require students ers, or even brief informal comments to practice that skill using an assignment from teachers on student strengths from the school day. They should follow and weaknesses. up by checking assignments and confer- encing with the classroom teacher. When• Preparing relevant information on promoting the use of skills from OST dur- school operations, academic standards, ing the school day, instructors should be improvement plans, and curricula. careful to coordinate with the classroom teacher first to ensure that the relevant• Observing and communicating with OST skill will align with classroom instruction staff to identify opportunities for greater and will not disrupt the teacher’s routine. coordination with the school day. This can be useful for helping students de- velop note-taking, planning, and studyingIt also can be beneficial for school districts skills that will help them achieve successto designate a key staff person to work with with the school-day curriculum.51OST coordinators. The district designee canprovide OST coordinators with informationon standards and curricula. The district 50. Borman and Dowling (2006); Langberg et al.designee also can play a key role in coor- (2006); Roderick, Jacob, and Bryk (2004); Bormandinating and supervising the activities of (1997); Udell (2003).multiple SES and other OST providers. 51. Langberg et al. (2006). ( 16 )
  23. 23. 1. AlIgn ThE OST PROgRAM ACADEMICAlly WITh ThE SChOOl DAyField trips or cultural activities that are tight or sufficient numbers of experiencedpart of the OST program should be ex- teachers are not available.54plicitly linked to school content and statestandards. The panel believes that these Although little is known about the methodsactivities need to connect to something or characteristics that define effective teach-the students are learning in school to help ers, researchers have discovered that somethem see how what they learn in school teachers are much better than others at help-relates to their real-life experiences. The ing students achieve significant achievementresult can maximize the gains from both gains during the school day.55 For directthe OST program and the school day and instruction or supervisory roles, the panelmake academic content more relevant to recommends hiring classroom teachers whostudents’ lives (see recommendation 4 for demonstrate success during the school day,information on connecting engaging in- and the school can support these efforts. Tostruction to academic content). identify effective teachers to employ as the OST coordinator or as an OST instructor,4. Coordinate with the school to identify OST programs can seek out award-winningstaff for OST programs. teachers or work with administrators to identify effective teachers.56OST programs have several roles for whicheffective classroom teachers are well Potential roadblocks and solutionssuited. The panel recommends that pro-grams evaluate how classroom teachers Roadblock 1.1. The principal does notcan be useful to their programs and hire have time to coordinate with OST staff.them when appropriate to meet programgoals. For example, teachers can serve as Suggested Approach. OST programs sup-OST coordinators, particularly for sum- port the school by providing additionalmer programs in which teachers might academic assistance to students. To max-not face conflicting demands on their time imize their effectiveness, the panel sug-from their regular teaching schedules.52 gests that the principal designate someoneWhen funding is available to hire effective at the school to be responsible for com-teachers from the school to serve as OST munications regarding OST programming.instructors, these teachers can use their The OST program can help by clearly com-experience and knowledge of instructional municating the benefits of collaboration tomethods to maximize academic gains for the principal.participating students.53 Finally, teacherscan use their experience to advise and Roadblock 1.2. The OST program doesmentor less-experienced OST instructors not have enough money to hire a programor volunteers, especially when budgets are 54. In KindergARTen (Borman, Goertz, and Dowl- ing 2008), Teach Baltimore (Borman and Dowling52. In 21st CCLC programs, about 67 percent of 2006), NDP (Udell 2003), and Howard Street Tutor-coordinators had experience as classroom teach- ing (Morris, Shaw, and Perney 1990), schoolteachersers, and 34 percent were currently school-day served as advisors to less-experienced instructorsteachers (U.S. Department of Education 2003, or volunteers.p. 36). 55. See, for example, Hanushek (1992).53. In Chicago Summer Bridge (Jacob and Lef- 56. Roderick, Jacob, and Bryk (2004). Research-gren 2004), 21st CCLC (U.S. Department of Educa- ers have demonstrated that principals are good attion 2003), and Enhanced Academic Instruction identifying their best and worst teachers, but they(Black et al. 2008), for example, schoolteachers are not as good at distinguishing among thosefrequently served as OST instructors. who fall in between (Jacob and Lefgren 2008). ( 17 )
  24. 24. 1. AlIgn ThE OST PROgRAM ACADEMICAlly WITh ThE SChOOl DAycoordinator, or the school cannot afford to Roadblock 1.4. The OST instructor has somehire a school-based coordinator. concerns about aligning instruction with the school day, or the classroom teacher hasSuggested Approach. Depending on size some concerns about the OST instruction.and scope of the program, a full-time pro-gram coordinator might not be necessary. Suggested Approach. Concerns aboutThe panel believes that programs can be alignment may signal a need for greatersuccessful with a part-time coordinator, communication between the two programs.particularly when the program is small Use the OST coordinator and the school-or works with only a few students. What based coordinator to communicate con-is important is that both the program and cerns and ensure that students receivethe school designate a person who is re- the instruction that will benefit them thesponsible for managing the coordination most. This may mean sharing informationbetween program and school staff and about a particular student’s progress, forthat this person’s role in maintaining com- example.munication is clearly established. Anotherpossibility is for a volunteer to take on the Roadblock 1.5. Privacy concerns prohibitcoordinator role. the transfer of data from school to OST program.Roadblock 1.3. It is hard to get high-qual-ity teachers to work after school because Suggested Approach. OST programs canof their commitments and responsibilities establish a secure system for transferringduring the school day. data that meets the security concerns of the school and can ensure that only a lim-Suggested Approach. OST programs can ited number of staff members have accessconsider devising flexible staff schedules to the data. Programs should get formalthat allow teachers to work for shorter written consent from parents and studentsperiods (e.g., tutoring small groups of for their data to be released. OST staff alsostudents for one hour). Programs also can schedule meetings with teachers andcan consider involving busy but qualified staff to gather informal information onteachers in important support roles, such student performance in school.as coaching less-experienced instructorsor providing instructional training. ( 18 )
  25. 25. Recommendation 2. Level of evidence: LowMaximize student The panel judged the level of evidenceparticipation and supporting this recommendation to be low,attendance because there is no conclusive evidence that following the action steps in this rec- ommendation will lead to higher atten-To attract and retain participants, dance or increased academic achievement.OST programs should determine Given the voluntary nature of most OSTwhich factors prevent students from programs and other barriers discussed inparticipating in the program and work this recommendation, regular attendancewith schools and parents to ensure that appears to be a difficult goal for many pro-the program is addressing those factors. grams to reach. Some programs have de-Parents are critical to this process voted considerable resources and seem tobecause they are co-decisionmakers have made efforts to implement the actionabout students’ participation in OST steps described in this recommendationprograms, and children generally value and still have trouble getting students totheir parents’ judgment about which attend regularly (see Table D2). However,programs may be beneficial to them.57 given that attendance is a precursor to anImportant factors include location, OST program’s promoting student learn-transportation, timing, length, program ing, the panel believes it is particularlyofferings, and frequency of services. important for programs to enhance theirResearchers have found that student efforts to get students in the door.participation is affected by issues ofaccess and convenience, as well as Brief summary of evidence toby the adequacy and attractiveness support the recommendationof the services and features providedin the program.58 The importance of emphasizing partici- pation has been pointed out by many ex- perts in OST.59 Although it seems logicalThe challenge for OST program that students need to attend to receive theorganizers is to design program benefits of a program, there is no rigorousfeatures to minimize the barriers to evidence demonstrating that the steps rec-participation, especially for the students ommended here will lead to increased par-most in need of program services and ticipation, and limited evidence that aca-most likely to benefit from them. OST demic achievement is increased throughprograms also can increase awareness more exposure to OST programs. A meta-and acceptance by promoting the analysis of 53 OST programs by Lauer etprogram among school staff and al. (2004) found larger effect sizes in bothfamilies. greater awareness is likely to math and reading for programs that con-facilitate communication with schools sisted of at least 45 hours of program-and parents regarding students who ming.60 The panel believes that if a pro-struggle to attend and could help gram is aligned academically with thein actively persuading students toparticipate in program activities. 59. Cooper et al. (2000); Granger and Kane (2004); Lauver, Little, and Weiss (2004). 60. The meta-analysis also found that, on average, programs with very high durations (more than 10057. Duffett et al. (2004). hours for math and 210 hours for reading) did not58. Ibid. have effects significantly different from zero. ( 19 )
  26. 26. 2. MAxIMIzE STuDEnT PARTICIPATIOn AnD ATTEnDAnCEschool day (recommendation 1), adapts on attendance rates.64 Even when pro-instruction to individuals and groups (rec- grams report attendance, the panel cannotommendation 3), and provides engaging isolate which, if any, components of thelearning experiences (recommendation 4), program affected attendance, forcing thegreater exposure to that program will yield panel to use its judgment regarding whichhigher academic achievement. practices contributed to increased atten- dance and improved academic outcomes.Since a student’s actual program atten- In terms of overall academic effects, 7 ofdance, as a percentage of hours of pro- the 14 showed positive effects,65 2 showedgramming offered, may be correlated with mixed effects,66 and 5 others showed nounobservable characteristics such as mo- effects.67 Despite the lack of consistenttivation or family circumstances, it is not evidence linking the panel’s suggestionsadvisable to draw causal conclusions from to increased academic achievement, themost studies on the relationship between panel believes these recommendations,attendance and outcomes. Four evalua- faithfully implemented and taking intotions met WWC standards with or with- consideration the unique constraints andout reservations for their impact studies student populations of each program, canand also looked at the possible relation increase student attendance and, there-between level of program attendance and fore, contribute to achievement gains.academic achievement.61 Only one found apositive correlation between higher atten- Only one study provided direct evidencedance and greater program effects.62 on increasing attendance. Black et al. (2008) randomly assigned students to ei-The other evidence for this recommen- ther a less-structured, business-as-usualdation is less direct. Fourteen programs after-school program or an enhanced mathused practices similar to the action steps or reading program that included moni-recommended by the panel (such as toring and incentive systems to increaseusing teachers to recruit students, locat- attendance. Students in the enhanced pro-ing within schools, offering snacks, and gram attended significantly more daysincluding enrichment activities) and also than did control students.had evaluations that met WWC standardswith or without reservations.63 Of the 14programs, 6 reported some information 64. Early Risers—August et al. (2001); Kinder- gARTen—Borman, Goetz, and Dowling (2008);61. Teach Baltimore—Borman and Dowling (2006); Enhanced Academic Instruction—Black et al.Early Risers—August et al. (2001); L.A.’s BEST— (2008); Teach Baltimore—Borman and DowlingGoldschmidt, Huang, and Chinen (2007); 21st (2006); L.A.’s BEST—Goldschmidt, Huang, andCCLC—U.S. Department of Education (2003). Chinen (2007); 21st CCLC—U.S. Department of Education (2003).62. Teach Baltimore—Borman and Dowling(2006). 65. CHP—Langberg et al. (2006); Fast ForWord— Slattery (2003); Howard Street Tutoring—Morris,63. CHP—Langberg et al. (2006); Fast ForWord— Shaw, and Perney (1990); SMART—Baker, Gersten,Slattery (2003); Howard Street Tutoring—Morris, and Keating (2000); Summer Reading Day Camp—Shaw, and Perney (1990); Start Making a Reader Schacter and Jo (2005); KindergARTen—Borman,Today (SMART)—Baker, Gersten, and Keating Goetz, and Dowling (2008); Early Risers—August(2000); Summer Reading Day Camp—Schacter et al. (2001).and Jo (2005); KindergARTen—Borman, Goetz,and Dowling (2008); Early Risers—August et al. 66. Teach Baltimore—Borman and Dowling(2001); Enhanced Academic Instruction—Black et (2006); Enhanced Academic Instruction—Blackal. (2008); Teach Baltimore—Borman and Dowl- et al. (2008).ing (2006); L.A.’s BEST—Goldschmidt, Huang, and 67. L.A.’s BEST—Goldschmidt, Huang, and ChinenChinen (2007); YS-CARE—Bissell et al. (2002); 21st (2007); YS-CARE—Bissell et al. (2002); 21st CCLC—CCLC—U.S. Department of Education (2003); Leap U.S. Department of Education (2003); Leap Frog—Frog—McKinney (1995); NDP—Udell (2003). McKinney (1995); NDP—Udell (2003). ( 20 )
  27. 27. 2. MAxIMIzE STuDEnT PARTICIPATIOn AnD ATTEnDAnCEHow to carry out this place of worship.68 Parents often prefer therecommendation use of school facilities for services, which eliminates the need to move from school1. Design program features to meet the needs to another location after school.69 If theand preferences of students and parents. program is not located at the school, or if the program is serving students from mul-The panel recommends that the OST pro- tiple schools, schools and districts shouldgram gather information about parent ensure that transportation to and from thepreferences with a survey or seek the program is readily available and affordableadvice of school staff in identifying the (or provided at no cost), and that adult su-needs of parents and students. The survey pervision is provided while transportingcould be distributed to parents through students.70 The panel believes that OSTthe school and could ask a short series of programs also should try to operate dur-questions, such as ing hours that are convenient for families, particularly for working parents.• Do you prefer an after-school/summer program that is located at your child’s Program features should reflect the con- school or one that is located at a com- tent that students and parents want, both munity center? in academic and nonacademic areas. Par- ents are likely to be looking for academic• Would you be able to transport your or homework help that reflects the areas child from school to an after-school in which their children need additional program? help.71 Since academically oriented OST programs may be competing with other• What times would you like the pro- recreational activities in the same time gram to start and end? period,72 programs should offer enrich- ment and recreational activities in addi-• Which academic subjects does your tion to academic instruction. These ac- child need additional support in? tivities can include theater, music, arts and crafts, sports, board games, fitness,• How many days per week would you and martial arts, but they should reflect send your child to an after-school/ student interests.73 To satisfy students’ summer program? nutritional needs, after-school programs should incorporate a snack time into theThe program could consider a similar short schedule, whereas summer programssurvey to gauge the preferences of stu-dents, especially when serving the highergrade levels in which students are more 68. August et al. (2001); Langberg et al. (2006);likely to choose a program on their own. Chaplin and Capizzano (2006); Reisner et al. (2004).The responses should guide how the pro- 69. U.S. Department of Education (2009).gram organizes and provides its services. 70. August et al. (2001); Langberg et al. (2006);This includes working with schools and dis- Morris, Shaw, and Perney (1990); Center for Ap- plied Linguistics (1994).tricts to ensure that design features make 71. Duffett et al. (2004).the program accessible. For example, theOST program should consider a location 72. Carver and Iruka (2006); Brown (2002); Coo- per et al. (2000).that is well situated and easy to get to, 73. Bissell et al. (2002); Schacter and Jo (2005);whether it is a school, community center, or August et al. (2001); Langberg et al. (2006); Bor- man and Dowling (2006); Goldschmidt, Huang, and Chinen (2007). ( 21 )
  28. 28. 2. MAxIMIzE STuDEnT PARTICIPATIOn AnD ATTEnDAnCEshould consider including lunch or another staff can follow up with school staff to seeappropriate meal.74 if the problem extends to the school day. OST staff also could coordinate with school2. Promote awareness of the OST program staff to contact parents to determine thewithin schools and to parents. reason for the absences.76 Programs can consider using reward incentives, positiveIt is the opinion of the panel that OST pro- reinforcement, or special privileges to en-grams, whether organized by schools, dis- courage regular attendance. For example,tricts, or private providers, should consis- incentives can be in the form of monthlytently inform parents, teachers, and other prizes or a point system that rewards stu-school staff about programs and their ben- dents with points for good attendance orefits. Programs can use various methods behavior that can be redeemed for specialof promotion, including websites, flyers benefits such as field trips, school sup-distributed at parent meetings, notices on plies, small prizes, or books.77school bulletin boards or in school news-letters, and word of mouth. Information Potential roadblocks and solutionsshould include program location, hoursof operation, and contact numbers so that Roadblock 2.1. As students get older, ad-parents can raise questions or concerns, ditional options for after-school or summerand the information should be available in recreational activities become availablemultiple languages when appropriate. and increase the competition for students’ after-school time.78Schools can join OST providers in promot-ing participation in the programs. For ex- Suggested Approach. An OST coordi-ample, teachers and administrators can nator should be aware of other extracur-identify and recruit students who might ricular activities when making schedulingbenefit from OST program services. Teach- or timing decisions so that it is easier forers can provide referrals or informational students to participate in both academicmaterials during parent-teacher meetings or and recreational programs and, thereby,give the program a list of students in need minimize competition. OST programs alsoof academic assistance.75 Teachers or school can make participation in sports practiceadministrators also can remind students at or other activities a privilege contingentthe end of the school day about attending on attendance in the academic portion ofthe after-school program and, if needed, es- the OST program.79cort students directly to the program. Roadblock 2.2. Students may have un-3. use attendance data to identify students avoidable circumstances that prevent themfacing difficulties in attending the program. from attending OST programs, such as tak- ing care of a sibling after school.Program coordinators should systemati-cally collect OST program attendance data 76. August et al. (2001); Black et al. (2008); Centerand use the data to identify students with for Applied Linguistics (1994).recurring absences or low attendance. OST 77. Black et al. (2008); Udell (2003); U.S. Depart- ment of Education (2005); Langberg et al. (2006); Brown (2002).74. Bissell et al. (2002); Schacter and Jo (2005); 78. Studies have found a drop-off in student par-Morris, Shaw, and Perney (1990); Borman and ticipation after 3rd grade (Grossman et al. 2002)Dowling (2006). and also between elementary and middle school75. Langberg et al. (2006); Goldschmidt, Huang, (U.S. Department of Education 2003; Grossmanand Chinen (2007); Baker, Gersten, and Keating et al. 2002; Reisner et al. 2002).(2000); McKinney (1995). 79. U.S. Department of Education (2003). ( 22 )
  29. 29. 2. MAxIMIzE STuDEnT PARTICIPATIOn AnD ATTEnDAnCESuggested Approach. If a large portion Roadblock 2.4. Communicating withof an OST program’s target group is fac- families involved in the program may being these types of barriers, the program difficult.should consider devising options for stu-dents who cannot be onsite. Programs Suggested Approach. Schools and OSTmight consider providing tutoring for providers should use multiple methods tothese students in the home or implement- communicate with parents. Some parentsing a system that can provide the instruc- may move or change phone numbers fre-tion in an online or electronic format. quently, which can make phone and mail communication difficult. Work with theRoadblock 2.3. The number of slots in OST school to provide information throughprograms is limited. school staff or teachers via flyers sent home with students. Recruitment efforts shouldSuggested Approach. If the demand for extend beyond the school site to locationsOST programming is greater than the ca- that families frequent. These may includepacity of the program, organizers should grocery stores, laundromats, and com-consider ways to expand their capacity munity and faith-based centers. Schoolsor partner with other programs that have should consider the common languagesavailable space. Organizers may need to spoken in the area and provide translatedseek additional funding sources, such as materials or hire bilingual staff.foundations or other federal grant pro-grams, for which they might consider hir-ing an external evaluator to demonstratethe value of the program model. ( 23 )

×