- The document presents findings from an evaluation of the Rhode Island GEAR UP program that used propensity score matching to select a comparison group and measured outcomes for GEAR UP participants and comparisons through high school and college. - Before matching, GEAR UP participants differed from their classmates on characteristics like gender, race, academic performance, and economic status. Propensity score matching created a balanced comparison group. - Results found GEAR UP participants were more likely to be on-time in their grade level and less likely to drop out of school compared to the matched comparison group. Higher levels of GEAR UP participation were also linked to higher rates of on-time grade progression.

1. Evaluation of the Rhode Island
GEAR UP Program:
Propensity Score Matching, Initial
Impact and Integration of Findings into
Program Management
NCCEP/GEAR UP Annual Conference
July 19-22, 2015
Presenters:
Neeta Fogg, Ph.D., Center for Labor Markets and Policy, Drexel University
DavidVillegas, M.A., RI GEAR UP/The College Crusade of Rhode Island
1

2. Background and RI GEAR UP
Support Services Summary
2

3. Background
RI GEAR UP:
• Implemented byThe College
Crusade of Rhode Island (founded
in 1989), one of the nation’s first
statewide college access and
scholarship programs
• First GEAR UP grant awarded in
1999
• Secured 2 additional, 6-year GEAR
UP grants
• 2005-2011
• 2011-2017
3

4. Participant Demographics
• 3,900 RI GEAR UP urban students in
grades 6-12 are served annually
• 100% eligible for free or reduced lunch
• 92% from minority families
• 77% will be first in families to graduate
from college
• 39% living in homes where English is not
the primary language
• 1,400 parents served annual; total
attendance at 4,999
• 1,412 former participants in college
• Of those receiving scholarships, 93% are
eligible for Pell Grants
4

5. Educational Landscape in RI
• RI GEAR UP support services
rendered to students attending 15 of
17 priority middle and high schools
• RI is 34th in nation for 4-year
graduation rate
• RI is 5th highest high school event
dropout rate in the nation
• Large disparities in college enrollment
and degree attainment rates for
Blacks and Latinos
• RI is ranked among the worst states
for student debt
5

6. • Comprehensive advisory
• Over 60 student programs/services
rooted in:
• Academic enrichment
• Postsecondary preparation
• Social and personal development
• Career awareness/exploration
• Parent engagement
• 48 workshops offered multiple times
throughout the year
• Core offerings + Parenting Strategies and
Financial Literacy
• College scholarships
GEAR UP Support Services
6

7. Non-GEAR UP Support Services
Postsecondary transition counseling and college
advisory services
• Pre-transition
• Accuplacer test preparation
• Dual Enrollment initiatives
• Financial aid counseling
• Referrals to additional state institution’s access
and persistence programs
• Transition
• Summer outreach
• Post-transition
• Continued financial aid counseling
• Student enrollment status and academic
tracking
• Workshops
• Civic engagement
• Coordination of services with higher education
partners
7

8. Center for Labor Markets and Policy | Drexel University
Neeta Fogg, Ph.D.
Center for Labor Markets and Policy
Drexel University
2015 NCCEP/GEAR UP Annual Conference, San Francisco, CA
July 20, 2015
Evaluation of the Rhode Island GEAR UP
Program:
Propensity Score Matching, Initial Impact, and
Integration of Findings into Program Management
8

9. Center for Labor Markets and Policy | Drexel University
OUTLINE
 Evaluation Plan
 Selection of Matched Comparison Groups
 Characteristics of RI GEAR UP Participants and their
Classmates, Before and After Propensity Score
Matching
 On-Time Grade Promotion and Attrition
 On-Time High School Graduation
 College Enrollment Among High School Graduates
 Summary Outcome: Share of all 6th Graders in 2007-
08 Enrolled in College in 2014
9

10. Center for Labor Markets and Policy | Drexel University
OUTLINE
 Evaluation Plan
 Selection of Matched Comparison Groups
 Characteristics of RI GEAR UP Participants and their
Classmates, Before and After Propensity Score
Matching
 On-Time Grade Promotion and Attrition
 On-Time High School Graduation
 College Enrollment Among High School Graduates
 Summary Outcome: Share of all 6th Graders in 2007-
08 Enrolled in College in 2014
10

11. Center for Labor Markets and Policy | Drexel University
 Select a matched comparison group of students from the
6th grade (non-participating) classmates of each entering
6th grade cohort RI GEAR UP participants.
 In each successive school year from 6th grade to high
school graduation, track each cohort of RI GEAR UP
participants and matched comparison group students.
 After high school graduation track RI GEAR UP participants
and comparison group students into colleges and
universities across the nation.
 Measure differences in key secondary and postsecondary
schooling outcomes of RI GU participants and the
comparison group.
Plan for Quasi-Experimental Evaluation
11

12. Center for Labor Markets and Policy | Drexel University
Evaluation Time-Table for Each Cohort
On-Time Grade Level in Middle and High School
Cohorts
6th
grade
7th
Grade
8th
grade
9th
grade
10th
grade
11th
grade
12th
grade
On-time
High School
Graduation
Immediate
College
Enrollment
Freshman
Year
Retention
Cohort 1
(249) 2007-08 2008-09 2009-10 2010-11 2011-12 2012-13 2013-14
Spring
2014
Fall
2014
Fall
2015
Cohort 2
(205) 2008-09 2009-10 2010-11 2011-12 2012-13 2013-14 2014-15
Spring
2015
Fall
2015
Fall
2016
Cohort 3
(291) 2009-10 2010-11 2011-12 2012-13 2013-14 2014-15 2015-16
Spring
2016
Fall
2016
Fall
2017
Cohort 4
(405) 2010-11 2011-12 2012-13 2013-14 2014-15 2015-16 2016-17 Spring 2017
Fall
2017
Fall
2018
Cohort 5
(560) 2011-12 2012-13 2013-14 2014-15 2015-16 2016-17 2017-18 Spring 2018 Fall 2018
Fall
2019
Cohort 6
(449) 2012-13 2013-14 2014-15 2015-16 2016-17 2017-18 2018-19 Spring 2019 Fall 2019
Fall
2020
12

13. Center for Labor Markets and Policy | Drexel University
Evaluation Outcomes
Progress through Middle and High School
 On-time grade promotion
 Attrition
High School Graduation
 On-time high school graduation
 Delayed high school graduation
College Enrollment and Persistence
 Immediate college enrollment
 Delayed college enrollment
 College freshman year retention
13

14. Center for Labor Markets and Policy | Drexel University
Data for Evaluation
 Rhode Island GEAR UP Program: Administrative data on
treatment, dosage, etc.
 Rhode Island Department of Education: Annual data for
each school year for each cohort from 6th grade to high school
graduation across all school districts in Rhode Island.
 National Student Clearinghouse: College enrollment and
persistence data for each cohort for each semester after
graduating high school
 Other publicly available data sources: as needed
Merge these data to build longitudinal data files for each cohort
of RI GEAR UP participants and comparison group students to
track them through middle school, high school, and college.
14

15. Center for Labor Markets and Policy | Drexel University
OUTLINE
 Evaluation Plan
 Selection of Matched Comparison Groups
 Characteristics of RI GEAR UP Participants and their
Classmates, Before and After Propensity Score
Matching
 On-Time Grade Promotion and Attrition
 On-Time High School Graduation
 College Enrollment Among High School Graduates
 Summary Outcome: Share of all 6th Graders in 2007-
08 Enrolled in College in 2014
15

16. Center for Labor Markets and Policy | Drexel University
The Need for Propensity Score Matching
 Pre-program traits of GEAR UP participants were very
different from their classmates.
 Propensity score matching is used to establish
baseline (pre-program) equivalence.
 Matching propensity scores of individuals in the
treatment group with the non-treatment group will
yield a comparison group that is similar to the
treatment group on all covariates leaving the
treatment as the only difference between them.
 A balanced match based on propensity scores
provides pre-program equivalence and unbiased
estimates of impact.
16

17. Center for Labor Markets and Policy | Drexel University
Propensity Score Matching
Comparison
17

18. Center for Labor Markets and Policy | Drexel University
List of Covariates Included in Propensity Score
Models Used to Create the Comparison Group
Covariates
Demographic
traits
Gender, race-ethnicity, IEP status
Economic status Free or reduced price school lunch status
Academic traits Performance on the NECAP standardized
reading test
Behavioral traits Attendance rate
School
Climate/Quality
Composite index of school climate/quality
Propensity Score Probability of being in the RI GEAR UP program
based on above characteristics
18

19. Center for Labor Markets and Policy | Drexel University
OUTLINE
 Evaluation Plan
 Selection of Matched Comparison Groups
 Characteristics of RI GEAR UP Participants
and their Classmates, Before and After
Propensity Score Matching
 On-Time Grade Promotion and Attrition
 On-Time High School Graduation
 College Enrollment Among High School Graduates
 Summary Outcome: Share of all 6th Graders in 2007-
08 Enrolled in College in 2014
19

20. Center for Labor Markets and Policy | Drexel University
Gender
BEFORE MATCHING
RI GEAR UP
Participants
All Non-
Participating
Classmates
Difference
(Percentage
Points)
Total Number 249 3,136 ---
Percent of Total:
Male 44.6 53.4 -8.8
Female 55.4 46.6 +8.8
AFTER MATCHING
RI GEAR
UP
Participants
Matched
Comparison
Group of
Non-
Participating
Classmates
Difference
(Percentage
Points)
249 249 ---
44.6 45.0 -0.4
55.4 55.0 +0.4
20

21. Center for Labor Markets and Policy | Drexel University
Race-Ethnicity
BEFORE MATCHING
RI GEAR UP
Participants
All Non-
Participating
Classmates
Difference
(Percentage
Points)
Total Number 249 3,136 ---
Percent of Total:
Asian, Pacific
Islander or Native
American 7.2 5.8 1.4
Black, non-Hispanic 23.3 18.4 4.9
Hispanic 54.2 48.4 5.8
White, non-Hispanic 15.3 27.4 -12.1
AFTER MATCHING
RI GEAR
UP
Participants
Matched
Comparison
Group of
Non-
Participating
Classmates
Difference
(Percentage
Points)
249 249 ---
7.2 7.2 0.0
23.3 23.3 0.0
54.2 54.2 0.0
15.3 15.3 0.0
21

22. Center for Labor Markets and Policy | Drexel University
Academic Proficiency:
6th Grade NECAP (Standardized) Reading Test
BEFORE MATCHING
RI GEAR UP
Participants
All Non-
Participating
Classmates
Difference
(Percentage
Points)
Total Number 249 3,136 ---
% Dist. by level of 6th grade NECAP reading test
Proficient with
Distinction 8.4 3.8 4.6
Proficient 51.4 32.8 18.6
Partly Proficient 27.3 32.0 -4.7
Substantially below
proficient 12.9 31.4 -18.5
Did not take the
test/waiver/missing 1.2 5.8 -4.6
AFTER MATCHING
RI GEAR
UP
Participants
Matched
Comparison
Group of
Non-
Participating
Classmates
Difference
(Percentage
Points)
249 249 ---
8.4 8.4 0.0
51.4 51.4 0.0
27.3 27.3 0.0
12.9 12.9 0.0
1.2 1.2 0.0
22

23. Center for Labor Markets and Policy | Drexel University
FRL, Attendance, and IEP status in the 6th grade
BEFORE MATCHING
RI GEAR UP
Participants
All Non-
Participating
Classmates
Difference
(Percentage
Points)
Total Number 249 3,136 ---
Percent of Total:
With free or
subsidized school
lunch in 6th grade 91.6 82.1 9.5
Attendance rate in 6th
grade 94.7 91.8 2.9
With IEP in 6th grade 11.7 23.2 -11.5
AFTER MATCHING
RI GEAR
UP
Participants
Matched
Comparison
Group of
Non-
Participating
Classmates
Difference
(Percentage
Points)
249 249 ---
91.6 92.0 -0.4
94.7 94.8 -0.1
11.7 11.2 0.5
23

24. Center for Labor Markets and Policy | Drexel University
Distribution of 6th Graders by Middle School
Climate/Quality
School Quality Index was constructed from five indicators for 49 middle schools in the GEAR UP districts: school
stability rate (low turnover of students), school attendance rate, 6th grade NECAP reading test score, share of
students with free or reduced price school lunch, and share of race-ethnic minority students.
BEFORE MATCHING
RI GEAR UP
Participants
All Non-
Participating
Classmates
Difference
(Percentage
Points)
Total Number 249 3,136 ---
Percent of Total by Quartile of School Climate/Quality
Index:*
Schools in the lowest
quartile 40.9 36.1 4.8
Second quartile 41.8 41.0 0.8
Third quartile 15.3 18.8 -3.5
Highest quartile 2.0 4.1 -2.1
AFTER MATCHING
RI GEAR
UP
Participants
Matched
Comparison
Group of
Non-
Participating
Classmates
Difference
(Percentage
Points)
249 249 ---
40.9 40.9 0.0
41.8 41.0 0.8
15.3 16.1 -0.8
2.0 2.0 0.0
24

25. Center for Labor Markets and Policy | Drexel University
OUTLINE
 Evaluation Plan
 Selection of Matched Comparison Groups
 Characteristics of RI GEAR UP Participants and their
Classmates, Before and After Propensity Score
Matching
 On-Time Grade Promotion and Attrition
 On-Time High School Graduation
 College Enrollment Among High School Graduates
 Summary Outcome: Share of all 6th Graders in 2007-
08 Enrolled in College in 2014
25

26. Center for Labor Markets and Policy | Drexel University
On-Time Grade Promotion and Attrition
 On-time promotion to successive grade levels
keeps students on track to graduate high school
on-time.
 Students who fall behind below modal grade are
at a higher risk of dropping out and less likely to
graduate high school
26

27. Center for Labor Markets and Policy | Drexel University
Status During School Year 2013-14
On-time
grade**
81%
Behind
Grade
8%
Not
enrolled
**
11%
RI GEAR UP Participants
On-time
grade**
72%
Behind
Grade
10%
Not
enrolled
**
18%
Matched Comparison
Group
Significance level of differences between participants and comparison
group students: *** p < .01 ** p< .05 * p < .10
27

28. Center for Labor Markets and Policy | Drexel University
Percent who Entered 12th Grade On Time (in SY 2013-14) by
Total (Lifetime) Hours of GEAR UP Program Participation
56%
82%
92% 92%
81%
72%
Lowest Quartile Second Third Highest Quartile GEAR UP
Participants (All)
Comparison
Group
GEAR UP Participants by Quartile of Lifetime Service Hours
28

29. Center for Labor Markets and Policy | Drexel University
OUTLINE
 Evaluation Plan
 Selection of Matched Comparison Groups
 Characteristics of RI GEAR UP Participants and their
Classmates, Before and After Propensity Score
Matching
 On-Time Grade Promotion and Attrition
 On-Time High School Graduation
 College Enrollment Among High School Graduates
 Summary Outcome: Share of all 6th Graders in 2007-
08 Enrolled in College in 2014
29

30. Center for Labor Markets and Policy | Drexel University
On-Time High School Graduation
On-time high school graduation rate measures the
share of 6th graders in 2007-08 who had graduated
high school during or before school year 2013-14
30

31. Center for Labor Markets and Policy | Drexel University
Percent of 6th Graders Who Graduated High School On Time (SY 2013-
14 ): Gender
*** p <.01 ** p < .05 * p < .10
RI GEAR UP
Participants
Total Number 249
Percent who Graduated
High School On Time
All 74.7
Male 71.2
Female 77.5
Matched
Comparison Group
of Non-Participating
Classmates
249
66.7
63.4
69.3
Difference
(Percentage Points)
---
+8.0**
+7.8
+8.2*
31

32. Center for Labor Markets and Policy | Drexel University
Percent of 6th Graders Who Graduated High School On Time (2013-14
SY):
6th Grade Free and Reduced Price Lunch (FRL) Status
RI GEAR UP
Participants
Total Number 249
Percent who Graduated
High School On Time
All 74.7
FRL in 6th Grade 73.7
No FRL in 6th Grade 85.7
Matched
Comparison Group
of Non-Participating
Classmates
249
66.7
65.5
80.0
Difference
(Percentage Points)
---
+8.0**
+8.2**
+5.7
*** p <.01 ** p < .05 * p < .10
32

33. Center for Labor Markets and Policy | Drexel University
Percent of 6th Graders Who Graduated High School On Time (2013-14
SY):
Race-Ethnicity
RI GEAR UP
Participants
Total Number 249
Percent who Graduated High
School On Time
All 74.7
Black 77.6
Hispanic 75.6
White 60.5
Matched
Comparison
Group of Non-
Participating
Classmates
Difference
(Percentage
Points)
249 ---
66.7 +8.0**
66.7 +10.9*
70.4 +5.2
53.8 +6.7
All Non-
Participating
Classmates
3,136
55.8
50.5
55.3
57.8
*** p <.01 ** p < .05 * p < .10
33

34. Center for Labor Markets and Policy | Drexel University
Percent of 6th Graders Who Graduated High School On Time (2013-14
SY):
6th Grade Attendance Rate Quartile
RI GEAR UP
Participants
Total Number 249
Percent who Graduated High School On Time
All 74.7
6th Grade Attendance Rate Quartile
Lowest Quartile 50.0
Second 70.7
Third 72.3
Highest Quartile 87.2
Matched Comparison
Group of Non-
Participating
Classmates
Difference
(Percentage Points)
249 ---
66.7 +8.0**
34.6 +15.4
60.9 +9.8
70.1 +2.2
77.2 +10.1**
*** p <.01 ** p < .05 * p < .10
34

35. Center for Labor Markets and Policy | Drexel University
Percent of 6th Graders who Graduated High School On Time
by Total (Lifetime) Hours of GEAR UP Program Participation
52%
77% 81%
89%
75%
67%
Lowest Quartile Second Third Highest Quartile GEAR UP
Participants (All)
Comparison
Group
GEAR UP Participants by Quartile of Lifetime Service Hours
35

36. Center for Labor Markets and Policy | Drexel University
OUTLINE
 Evaluation Plan
 Selection of Matched Comparison Groups
 Characteristics of RI GEAR UP Participants and their
Classmates, Before and After Propensity Score
Matching
 On-Time Grade Promotion and Attrition
 On-Time High School Graduation
 College Enrollment Among High School Graduates
 Summary Outcome: Share of all 6th Graders
in 2007-08 Enrolled in College in 2014
36

37. Center for Labor Markets and Policy | Drexel University
SUMMARY OUTCOME: COLLEGE ENROLLMENT
AMONG THE ENTIRE COHORT OF 2007-08 SIXTH
GRADERS IN FALL 2014
This outcome is a summary measure of all
outcomes. It measures the percent of the entire
2007-08 6th grade cohort of RI GEAR UP
participants (and their classmates) who had
enrolled in college in Fall 2014.
This measure includes their grade promotion,
persistence (attrition), high school graduation, and
college enrollment outcomes.
37

38. Center for Labor Markets and Policy | Drexel University
Pathway from Sixth Grade (2007-08) to College
Enrollment (Fall 2014)
RI GEAR UP Participants Comparison Group
249
233
220
221
203
198
201
186
140
249
230
217
220
193
182
180
166
102
38

39. Center for Labor Markets and Policy | Drexel University
Pathway from Sixth Grade (2007-08) to
College Enrollment (Fall 2014)
56%
75%
81%
80%
82%
89%
88%
94%
100%
41%
67%
72%
73%
78%
88%
87%
92%
100%
0% 20% 40% 60% 80% 100% 120%
Enrolled in college in fall 2014
Graduated high school in spring 2014
Grade 12 or higher in 2013-14
Grade 11 or higher in 2012-13
Grade 10 or higher in 2011-12
Grade 9 or higher in 2010-11
Grade 8 or higher in 2009-10
Grade 7 or higher in 2008-09
Grade 6 in 2007-08
Comparison Group RI GEAR UP Participants
39

40. Center for Labor Markets and Policy | Drexel University
Percent of all 6th Graders in 2007-08 who had Enrolled in College in
Fall 2014:
Gender
RI GEAR UP
Participants
Matched
Comparison Group
of Non-Participating
Classmates
Difference
(Percentage Points)
Total Number 249 249 ---
Percent of the Entire Cohort
who had Enrolled in College
All 56.2 41.0 15.3***
Male 50.5 37.5 13.0**
Female 60.9 43.8 17.1***
*** p <.01 ** p < .05 * p < .10
40

41. Center for Labor Markets and Policy | Drexel University
Percent of all 6th Graders in 2007-08 who had Enrolled in College in
Fall 2014:
6th Grade Free or Reduced Price Lunch (FRL) Status
RI GEAR UP
Participants
Matched
Comparison Group
of Non-Participating
Classmates
Difference
(Percentage Points)
Total Number 249 249 ---
Percent of the Entire Cohort
who had Enrolled in College
All 56.2 41.0 15.3***
FRL in 6th grade 54.8 39.3 15.5***
No FRL in 6th grade 71.4 60.0 11.4
*** p <.01 ** p < .05 * p < .10
41

42. Center for Labor Markets and Policy | Drexel University
Percent of all 6th Graders in 2007-08 who had Enrolled in College in
Fall 2014:
Race-Ethnicity
RI GEAR UP
Participants
Matched
Comparison
Group of Non-
Participating
Classmates
Difference
(Percentage
Points)
Total Number 249 249 ---
Percent of the Entire Cohort
who had Enrolled in College
All 56.2 41.0 15.3***
Black 65.5 42.1 23.4***
Hispanic 53.3 41.5 11.8**
White 47.4 25.6 21.7**
*** p <.01 ** p < .05 * p < .10
42

43. Center for Labor Markets and Policy | Drexel University
Percent of all 6th Graders in 2007-08 who had Enrolled in College in
Fall 2014:
6th Grade Attendance Rate Quartile
RI GEAR UP
Participants
Matched
Comparison Group
of Non-
Participating
Classmates
Difference
(Percentage Points)
Total Number 249 249 ---
Percent of the Entire Cohort
who had Enrolled in College
All 56.2 41.0 15.3***
6th Grade Attendance Rate
Quartile
Lowest Quartile 43.8 23.1 20.7**
Second 46.6 34.4 12.2*
Third 55.4 41.8 13.6*
Highest Quartile 67.0 50.0 17.0***
*** p <.01 ** p < .05 * p < .10
43

44. Center for Labor Markets and Policy | Drexel University
Percent of all 6th Graders in 2007-08 who had Enrolled in
College in Fall 2014 by Total (Lifetime) Hours of GEAR UP
Program Participation
39%
50%
65%
71%
56%
41%
Lowest Quartile Second Third Highest Quartile GEAR UP
Participants (All)
Comparison
Group
GEAR UP Participants by Quartile of Lifetime Service Hours
44

45. Center for Labor Markets and Policy | Drexel University
Summary of Impact Estimates
45

46. Center for Labor Markets and Policy | Drexel University
Estimates of the Impact of the RI GEAR UP Program on
Various Outcomes of the 2007-08 Sixth Grade Cohort of
Participants
8.8***
-7.2**
8.0**
13.9***
15.3***
-15 -10 -5 0 5 10 15 20 25
Grade 12 on Time
Attrition by 2003-14
On-time high school graduation
College enrollment rate
Share of all 6th graders enrolled in college
Percentage Point Difference between
GEAR UP Participants and Comparison Group
*** p <.01 ** p < .05 * p < .10
46

47. Integration of Findings into
Program Management
47

48. Integration of Findings into Program
Management:
Who is benefiting the least?
On-time High School
Graduation
College Enrollment
All (N=249) 74.7% 56.2%
Gender
Male 71.2% 50.5%
Female 77.5% 60.9%
48

49. Integration of Findings into Program
Management:
Participation Hours by Gender
303
256
101
155
47
25
22
349
291
126
164
58
31
27
0.0
50.0
100.0
150.0
200.0
250.0
300.0
350.0
400.0
Total Hours Programs HS Programs MS Programs Advisory HS Advisory MS Advisory
MeanHours
Type
Male
Female
49

50. Integration of Findings into Program
Management:
Who is benefiting the least? (cont.)
On-time High School
Graduation
College Enrollment
All (N=249) 74.7% 56.2%
Race/Ethnicity
White 60.5% 47.4%
Black 77.6% 65.5%
Hispanic 75.6% 58.3%
50

51. Integration of Findings into Program
Management:
Participation Hours by Race/Ethnicity
388
332
144
188
56
33
23
322
266
112
154
56
29
27
293
250
90
161
42
20
22
0.00
50.00
100.00
150.00
200.00
250.00
300.00
350.00
400.00
450.00
Total Programs HS Programs MS Programs Advisory HS Advisory MS Advisory
MeanHours
Type
Blacks
Hispanics
Whites
51

52. Integration of Findings into Program
Management:
Who is benefiting the least? (cont.)
On-time High School
Graduation
College Enrollment
Attendance
1st Quartile (Low) 50.0% 43.8%
2nd Quartile 70.7% 46.6%
3rd Quartile 72.3% 55.4%
4th Quartile (High) 87.2% 67.0%
School Quality
1st Quartile (Low) 74.5% 43.8%
2nd Quartile 75.0% 46.6%
3rd Quartile 71.1% 55.4%
4th Quartile (High) 100% 67.0%
52

53. Integration of Findings into Program
Management:
Targeted Outreach and Recruitment
• Increase participation of males,
Whites and Hispanics in program
and advisory services
• Focus groups with non-participating
parents and students
• Identify barriers/solutions
• More concerted efforts to
engage parents living outside of a
5-mile radius of our
headquarters
• Host workshops/events in local
communities
• Improve communication
53

54. Integration of Findings into Program
Management:
Early Identification of At-risk Students
• EarlyWarning System (EWS)
• Newly enrolled Grade 6 students
• Indicators
• Quarter 1 Academic Standing in English
• Quarter 1 Academic Standing in Math
• Grade 5 Proficiency Level in Reading
• Grade 5 Proficiency Level in Math
• Grade 5 unexcused absences
GEAR UP
ID
Math AS
(< C)
English AS
(< C)
Math PL
(<3)
Reading
PL (<3)
Absences
(> 18)
EWS
Score
2013068 D+ B- 1 1 65 4
2013125 C B+ 3 3 9 0
54

55. Integration of Findings into Program
Management:
Early Identification of At-risk Students (cont.)
• Incorporate the School Quality
Index in EWS Contracts
• 24 middle schools classified in the 1st
and 2nd lowest school quality quartiles
• Require Advisors to establish EWS
contracts with 100% of students
attending schools in the 1st and 2nd
school quality quartiles
• Require Mobile Advisory to prioritize
EWS contracts for student attending
schools in the 1st and 2nd school quality
quartiles
55

56. • Follow-up analyses exploring the relationship(s) between
student outcomes and specific program/service participation
• Comparative analyses of GEAR UP and non-GEAR UP
matched participants on outcomes beyond college
enrollment:
• Persistence
• Degree attainment
• Replication of findings in successive cohorts of GEAR UP and
non-GEAR UP matched participants
• Continue to integrate findings into program management
Next Steps
56

57. Other Evaluation Efforts
• Annual GEAR UP Item Analyses:
• Student Assessment
• School by item analysis
(Program Managers)
• School specific grade by item
analysis (Advisors)
• Parent Survey
• Grade by item analysis
• Individual program surveys (60+)
• Pre/post for multi-day programs
• Satisfaction surveys for single day
support services
57

58. GEAR UP Student Assessment:
Ex.) School by Item Analysis
Has anyone from your school or RI GEAR UP ever spoken with you about
the availability of financial aid to help you pay for college?
0
20
40
60
80
100
Percent
School Name
No
Yes
58

59. GEAR UP Parent Survey:
Ex.) Grade by Item Analysis
Has anyone from your child’s school or RI GEAR UP ever spoken with you
about the availability of financial aid to help you pay for you child’s college
education costs?
0
10
20
30
40
50
60
70
Yes Yes, but I need to
learn more
Yes, and I undertand
the FA process
No Missing
Percent
6
7
8
9
10
11
12
59

60. Pre/Post Program Surveys:
Ex.) Ways to A’s –
21st Century Study Skills
• Pre/PostTest:
• 14 items
• Multiple choice
• Graded
• PostTest Includes:
• 9 survey items
• Rate the instructor
• General program satisfaction
• Express interest in topic areas
60

61. Pre/Post Program Evaluation:
Ex.) Ways to A’s –
21st Century Study Skills (Cont.)
Instructor N Mean (STD)
Pre
Mean (STD)
Post
Diff t df p
Christian 25 4.5 (2.1) 6.8 (2.6) 2.3 5.43 24 < .0001
Fran 25 4.0 (2.1) 8.0 (3.0) 4.0 7.30 24 < .0001
Hilary 26 4.4 (2.1) 8.3 (2.6) 3.9 7.46 25 < .0001
Lindsay 28 4.1 (2.2) 8.5 (2.9) 4.4 7.54 27 < .0001
Total 104 4.2 (2.1) 7.9 (2.8) 3.7 13.53 103 < .0001
Participant/Instructor Performance
61

62. Pre/Post Program Evaluation:
Ex.) Ways to A’s –
21st Century Study Skills (Cont.)
Of the 11 topics, which area would you like to spend more time in?
15.32
7.21
9.91
5.41
9.01
9.91
12.61
6.31 6.31
2.7
4.5
0.0
2.0
4.0
6.0
8.0
10.0
12.0
14.0
16.0
18.0
20.0
Percent
62

63. Repeated Measurement Program
Evaluation:
Ex.) Scholastic’s ReadAbout Program
• 12-month reading program provided
to newly enrolled RI GEAR UP
participants
• Poor academic standing in 5th grade
reading/ELA course
• Performing partially proficient or below
proficient on 5th grade NECAP
• Program is designed to improve
word comprehension and
vocabulary
• Participants complete the Scholastic
Reading Inventory (SRI) at 4 time
points
63

64. Repeated Measurement Program
Evaluation:
Ex.) Scholastic’s ReadAbout Program (cont.)
• Repeated measurement allows for
assessment of change in Lexile
score over time
• Individual level reports include:
• Baseline reading ability
• Final reading ability relative to all
classmates
• Reading ability relative to grade level
proficiency range
• Reports generated for full class
summarize average change in
vocabulary, word comprehension
and Lexile score.
64

65. Questions
DavidVillegas
Associate Director for Evaluation and IT Systems
RI GEAR UP/The College Crusade of Rhode Island
dvillegas@thecollegecrusade.org
Neeta Fogg
Research Professor
Center for Labor Markets and Policy
Drexel University
npf29@drexel.edu
65