The document outlines the efforts and successes of College Possible, a national college access and success organization serving low-income students across multiple states. It highlights their rigorous program evaluation which demonstrates that College Possible students are significantly more likely to graduate from college compared to their peers. The document also discusses challenges and prerequisites for effective program evaluation, emphasizing the importance of evidence-based practices for organizational credibility and sustainability.

1. One day the educational opportunities available to
children will be determined solely by their
talent, motivation, and effort.

2. Presentation and Workshop Agenda
Is Our Program Working?
How to Partner with Evaluators and Get Results
► I: Presentation: Overview of evaluation partnership and key
learnings
► II: Small group exercise: Overcoming potential evaluation
challenges
► III: Question and Answer Period and Discussion
Traci Kirtley, Director of Programming and Evaluation, College Possible
Dr. Chris Avery, Roy E. Larsen Professor of Public Policy and Management,
Malcolm Wiener Center for Social Policy, Harvard Kennedy School of
Government

3. Introduction to College Possible
College Possible is a national college access and success organization currently
serving 12,000 students in Minnesota’s Twin Cities; Milwaukee, Wisconsin;
Omaha, Nebraska and Portland, Oregon, with a growth plan to expand to serving
20,000 students in 10 metro areas by 2020.
2019-2020
20000 10 markets
20,000 students
2012-13
4 markets
15000 12,000 students
10000
2000-01
1 market
5000
35 students
0
2000 2002 2004 2006 2008 2010 2012 2014 2016 2018 2020

4. Program Model
College Access Program College Success Program
College Success Program
Structured after-school program Ongoing support through college
during junior & senior years persistence & graduation
320+ hours time on task in Weekly touch point with
sessions & one-on-one college coach
College application Academic and campus
assistance resource connections
Standardized test prep Social network support
Financial aid consulting AmeriCorps coach Financial aid consulting
College transition guidance ‘near-peer mentor’ Leadership development
College planning workshops for 9th and 10th grade students are led by coaches
and high school seniors to develop a college-going school culture early on.

5. Results
College Possible students are 10 times more likely to graduate from college
than their low-income peers.
College Access Program College Success Program
Percentage of students who Percentage of students who
enroll in college graduate from college
All Low
94 Income 58
College 55
74 Possible
59
All Income
11
"Tom Mortensen, Postsecondary Education Opportunity, 2010"

6. Introduction to Dr. Christopher Avery
► Roy E. Larson Professor of Public Policy at Harvard Kennedy School
► Extensive research in college admissions
► Current focus on college application and enrollment
patterns, especially among low-income students
► Recent studies have highlighted the issue of “undermatching” of high-
achieving, low-income students in applying to and enrolling in more
selective institutions

7. Why Evaluate?
3 main reasons to devote resources to program evaluation:
- Shows if program is working, and for whom and in what conditions
- Learn how to improve programmatic practices
- Increases organization’s credibility with funders and other
constituents
As College Possible grows to serve more students, rigorous and
systematic evaluation of program implementation and outcomes
becomes increasingly important to ensure the organization’s credibility
and sustainability.

8. Experience with Evaluations
Showed College Possible to be “very
2005
effective in helping the students who
Wilder complete the program gain admission
Research and financial assistance to college.”
2010 ICF External evaluation verified College
International Possible’s reported student ACT score
Program increases and college acceptance
Audit outcomes.
2011 Study by Dr. Chris Avery demonstrated that
Comparative College Possible students were more than
Historical twice as likely to enroll in 4-year instituation
Analysis as similarly situated peers.

9. Case Study: Avery study of College Possible
- During spring 2010 student recruitment, College Possible identified 8
high schools with significantly more eligible sophomore applicants than
available spots in cohort.
- School teams selected 80% of cohorts (32/40) as they usually would,
leaving a “last-in, first out” group of about 240 students.
- Among “last-in, first-out” group, the evaluator randomly assigned
students to enroll in the program or be on waitlist. This created treatment
and control groups.
- Treatment and control groups were solidified in fall 2010, when some
students were admitted off waitlist to replace students who had moved or
transferred. Ultimately, around 130 students were in treatment group.

10. Case Study: Avery study of College Possible
- From their junior year onward, evaluator compared following
outcomes of treatment and control groups using data from
independent sources.
- ACT attendance and score increases
- Number of college applications sent
- Types of colleges applied to (2 year or 4 year, selectivity, etc.)
- Admissions decisions
- College enrollment rates in fall and spring
- Plans to continue following students to study rates of college
persistence and graduation.

11. Lessons Learned: Evaluation Challenges
► Conducting an evaluation takes resources and attention away from
running the program
► Evaluations can challenge core beliefs when you “just know it works”
► Self-selection issues may exist
– How are participants recruited?
– How are participants selected?
► Evaluation needs must be balanced with program design needs or
preferences
► Find an evaluator you trust!

12. Lessons Learned: Evaluation prerequisites
► Measurable outcomes
► Logical explanation for how results are achieved
► Historical evidence
– Observational study/regression discontinuity
– As a program, are you ready to be evaluated?
► Data collection systems and process
► Staff buy-in

13. Special Issues: Randomized Controlled Trials
Benefits: Clean comparison of program effects
“Gold standard” establishes strong evidence base
Evidence of effectiveness helps build support
Challenges: Consent process – organizations and participants
Paperwork vs. program activities
Institutional Review Boards
Staff understanding – design issues, interpreting results
Uncertainty about Results

14. Small Group Discussion

15. Question and Answer/
Discussion