Is Our Program Working? How to Partner with Evaluators and Get Results
One day the educational opportunities available to children will be determined solely by their talent, motivation, and effort.
Presentation and Workshop AgendaIs Our Program Working?How to Partner with Evaluators and Get Results ► I: Presentation: Overview of evaluation partnership and key learnings ► II: Small group exercise: Overcoming potential evaluation challenges ► III: Question and Answer Period and Discussion Traci Kirtley, Director of Programming and Evaluation, College Possible Dr. Chris Avery, Roy E. Larsen Professor of Public Policy and Management, Malcolm Wiener Center for Social Policy, Harvard Kennedy School of Government
Introduction to College PossibleCollege Possible is a national college access and success organization currentlyserving 12,000 students in Minnesota’s Twin Cities; Milwaukee, Wisconsin;Omaha, Nebraska and Portland, Oregon, with a growth plan to expand to serving20,000 students in 10 metro areas by 2020. 2019-202020000 10 markets 20,000 students 2012-13 4 markets15000 12,000 students10000 2000-01 1 market 5000 35 students 0 2000 2002 2004 2006 2008 2010 2012 2014 2016 2018 2020
Program ModelCollege Access Program College Success Program College Success Program Structured after-school program Ongoing support through college during junior & senior years persistence & graduation320+ hours time on task in Weekly touch point with sessions & one-on-one college coach College application Academic and campus assistance resource connections Standardized test prep Social network support Financial aid consulting AmeriCorps coach Financial aid consultingCollege transition guidance ‘near-peer mentor’ Leadership development College planning workshops for 9th and 10th grade students are led by coaches and high school seniors to develop a college-going school culture early on.
ResultsCollege Possible students are 10 times more likely to graduate from collegethan their low-income peers. College Access Program College Success Program Percentage of students who Percentage of students who enroll in college graduate from college All Low 94 Income 58 College 55 74 Possible 59 All Income 11 "Tom Mortensen, Postsecondary Education Opportunity, 2010"
Introduction to Dr. Christopher Avery► Roy E. Larson Professor of Public Policy at Harvard Kennedy School► Extensive research in college admissions► Current focus on college application and enrollment patterns, especially among low-income students► Recent studies have highlighted the issue of “undermatching” of high- achieving, low-income students in applying to and enrolling in more selective institutions
Why Evaluate?3 main reasons to devote resources to program evaluation: - Shows if program is working, and for whom and in what conditions - Learn how to improve programmatic practices - Increases organization’s credibility with funders and other constituentsAs College Possible grows to serve more students, rigorous andsystematic evaluation of program implementation and outcomesbecomes increasingly important to ensure the organization’s credibilityand sustainability.
Experience with Evaluations Showed College Possible to be “very 2005 effective in helping the students who Wilder complete the program gain admission Research and financial assistance to college.” 2010 ICF External evaluation verified CollegeInternational Possible’s reported student ACT score Program increases and college acceptance Audit outcomes. 2011 Study by Dr. Chris Avery demonstrated thatComparative College Possible students were more than Historical twice as likely to enroll in 4-year instituation Analysis as similarly situated peers.
Case Study: Avery study of College Possible- During spring 2010 student recruitment, College Possible identified 8high schools with significantly more eligible sophomore applicants thanavailable spots in cohort.- School teams selected 80% of cohorts (32/40) as they usually would,leaving a “last-in, first out” group of about 240 students.- Among “last-in, first-out” group, the evaluator randomly assignedstudents to enroll in the program or be on waitlist. This created treatmentand control groups.- Treatment and control groups were solidified in fall 2010, when somestudents were admitted off waitlist to replace students who had moved ortransferred. Ultimately, around 130 students were in treatment group.
Case Study: Avery study of College Possible- From their junior year onward, evaluator compared followingoutcomes of treatment and control groups using data fromindependent sources. - ACT attendance and score increases - Number of college applications sent - Types of colleges applied to (2 year or 4 year, selectivity, etc.) - Admissions decisions - College enrollment rates in fall and spring- Plans to continue following students to study rates of collegepersistence and graduation.
Lessons Learned: Evaluation Challenges► Conducting an evaluation takes resources and attention away from running the program► Evaluations can challenge core beliefs when you “just know it works”► Self-selection issues may exist – How are participants recruited? – How are participants selected?► Evaluation needs must be balanced with program design needs or preferences► Find an evaluator you trust!
Lessons Learned: Evaluation prerequisites► Measurable outcomes► Logical explanation for how results are achieved► Historical evidence – Observational study/regression discontinuity – As a program, are you ready to be evaluated?► Data collection systems and process► Staff buy-in
Special Issues: Randomized Controlled TrialsBenefits: Clean comparison of program effects “Gold standard” establishes strong evidence base Evidence of effectiveness helps build supportChallenges: Consent process – organizations and participants Paperwork vs. program activities Institutional Review Boards Staff understanding – design issues, interpreting resultsUncertainty about Results