The document summarizes evaluations of college access, readiness, and career pathway initiatives. It discusses evaluating implementation of a Texas GEAR UP state grant program through longitudinal student tracking, surveys, site visits, and analysis of costs and lessons learned. Initial analysis found variation in services across schools and opportunities to increase student and parent knowledge. Forthcoming analysis will examine relationships between implementation factors and outcomes. The document also describes assessing implementation fidelity in the Diplomas Now school turnaround program evaluation through documentation of variation across sites and challenges.

1. icfi.com |
Evaluation 2013: The State of Evaluation
Practice in the Early 21st Century
27th Annual Conference of the American Evaluation Association
Washington, DC, USA
Design Considerations in Evaluating
the Implementation of College
Access, College Readiness, and
Career Pathway Initiatives
Thomas Horwood, Chair
Barbara O’Donnel, Discussant
October 18, 2013 | 4:30 - 6:00 PM

2. icfi.com |
Determining Dosage: Evaluating
the Implementation of a State
GEAR UP Initiative
Ashley Briggs
Charles Dervarics
October 18, 2013
Presented to:
American Evaluation Association
2013 Conference

3. icfi.com | 3
GEAR UP Nationally
 First grants funded in 1999 – Originally based on I Have a Dream program
concept
 Cohort approach – Most grantees follow a cohort of students from 7th grade
through to postsecondary education, with support services (tutoring,
mentoring, college visits) available
 FY12 – 132 federal grants serving 647,772 students
 Multiple award types – State grants and local partnership grants
OVERVIEW

4. icfi.com | 4
About the Texas GEAR UP State Grant (SG)
OVERVIEW
 FY12 state grant from USED – approx. $5M per year (2012 to 2019)
 Focused on a single cohort of students starting in Grade 7 (students are in
Grade 8 in 2013–14)
 Includes district and statewide services
 District services
– Support schools in four districts (7 MS  5 HS) to increase academic rigor
– Increase number of Grade 8 students succeeding in Algebra I (short-term goal)
– Provide teacher professional development to support delivery of rigorous courses
(such as Pre-AP training)
– Provide teacher professional development to support postsecondary goals (financial
literacy)
– Promote vertical alignment of core subject teachers across the grades
– Support college visits, summer learning opportunities, and tutoring services

5. icfi.com | 5
About the Texas GEAR UP State Grant (SG) (cont’d)
OVERVIEW
 Statewide Services
– Postsecondary information dissemination to students and families statewide
– Active, in-depth web site with information for students and families
– Online communication and teaching platform available statewide
– Statewide coalition of GEAR UP grantees (including local partnership grants not
directly under the SG)
 TEA GEAR UP Partners
– The University of Texas at Austin’s Institute for Public School Initiatives [IPSI]
– TG [Texas Guaranteed Student Loan Corporation]
– College Board
– AMS Pictures

6. icfi.com | 6
About the Texas GEAR UP SG Evaluation
EVALUATION DESIGN AND METHODOLOGY
The external evaluation is a longitudinal 7-year study using a quasi-
experimental design that started in January 2013 to:
 Provide ongoing formative evaluation of facilitators/barriers, promising
practices, and recommended next steps
 Explore implementation status, trends in the mix of implementation, and
relationships between implementation and outcomes
 Determine impact including short, intermediate, and long-term student
outcomes
 Identify impact on relevant family, school, and community partnership
outcomes
 Examine access to and use of statewide opportunities
 Understand cost, spending, and sustainability

7. icfi.com | 7
Data Sources
EVALUATION DESIGN AND METHODOLOGY
 Extant Data
– Documents: Texas GEAR UP SG Grant Application, Notice and Grant Awards
(NOGAs), and implementation plans
– Student level data: Demographics, attendance, high school course
completion and high school completion, school personnel, and district
organizational information
– School level data: Profile information about campus-level performance,
staff, finances, and programs
 Student Tracking System (Annual Performance Report – APR)
– Format: Submission by 4 subgrantee districts using a prepopulated
spreadsheet
– Topics: Advanced course-taking; Academic services; Student services;
Student events and attendance; Parent events and attendance; Teacher
professional development and enrollment; Community partners

8. icfi.com | 8
Data Sources (cont.)
EVALUATION DESIGN AND METHODOLOGY
 Surveys with Parents and Students
– Format: Online and paper-based versions in English and Spanish
– Topics: Aspirations and expectations; Knowledge of financial aspects;
Knowledge of college requirements; Perceptions of Texas GEAR UP SG
 Site Visits to Texas GEAR UP SG Schools
– Format: 1-1.5-day visits including interviews and focus groups with school
staff, teachers, students, parents, and community partners
– Topics: GEAR UP activities and events (school and statewide); Knowledge of
college requirements and financial aspects; Perceptions of Texas GEAR UP
SG; Readiness for success in college
 Interviews with Key Leaders from TEA and Partner Organizations
– Format: Telephone interviews
– Topics: Level of partner involvement; Perceptions of program; Progress on
statewide implementation

9. icfi.com | 9
Initial Analysis: Implementation
ANALYSIS
 Data Source: Student tracking system (APR) and site visits
 Primary Analysis: Descriptive statistics on participation, dosage (number of
hours, events), and mix (range of services/activities); disaggregation by school,
subject area, and format (virtual or in-person)
Implementation Strategy A B C D E F G
Adv. Course X X X X X X X
SSS: Tutoring X X X X X (math) X (math) X
SSS: Mentoring X X
SSS: Counseling/Advising X
SSS: Other Activities X (math) X (math)
College Visit X X X X
Job Site Visit X
Student Events X X X X X X
Parent Events X X X X X
Teacher PD X X X X
Community Partners X X X X
Use Statewide Services X X X
Total 4 6 5 5 8 7 11

10. icfi.com | 10
Initial Analysis: Plans, Knowledge, and Perceptions
ANALYSIS
 Data Source: Student and parent surveys
 Primary Analysis
– Descriptive statistics (frequencies, averages, ranges)
– Crosstabs (chi-square analyses comparing frequency distribution by
subgroup)
– Analysis of variance (ANOVA) comparing means by subgroup
– Correlation
 Key Baseline Takeaway: Both parent and student aspirations often exceeded
expectations, suggesting they are concerned about being able to achieve their
educational dreams.
 Key Baseline Takeaway: Few students or parents perceive themselves as very
knowledgeable, which can potentially be changed by participation in Texas
GEAR UP SG.
 Key Baseline Takeaway: Student overall satisfaction with Texas GEAR UP SG
was highest at one school, where 41% of students indicated they were very
satisfied.

11. icfi.com | 11
Initial Analysis: Costs and Lessons Learned
ANALYSIS
 Cost
– Data Source: Budgets and reported draw downs
– Primary Analysis: Descriptive statistics, breakdown by cost categories
 Facilitators and Barriers
– Data Source: Survey and site visit data
– Primary Analysis: Descriptive statistics, analysis of open-ended survey
responses, qualitative analysis
– Key Baseline Takeaway: Parents reported that engagement in activities is
facilitated when topics are of interest to them, when events are held at
times appropriate for their schedule, and when their student is also
engaged.
 Potentially Promising Practices
– Data Source: Site visit data
– Primary Analysis: Qualitative analysis
– Key Baseline Takeaway: Early successes at some schools related to
afterschool mathematics programs, enhanced college visits, and family
events.

12. icfi.com | 12
Forthcoming Analysis beyond Year 1
ANALYSIS
 Level and Mix of Implementation: Analysis of various service factors
– Provision type (virtual or on-line)
– Frequency of delivery (number of hours, number of sessions)
– Mix of services (e.g., enrollment in and tutoring in an advanced course)
– Quality of implemented activities
 Plans, Knowledge, and Perceptions: Disaggregation by student characteristics
– Gender, race/ethnicity, LEP status, special education status
– Participation in advanced coursework
 Cost
– Descriptive analysis of actual expenditures (annual and cumulative) by cost
category
 Types of Analysis
– HLM (with student, school, and district levels) and cluster analysis
– Impact analysis using extant outcome data
– Comparisons using PSM
– Linkages between outcomes and implementation
– Change in implementation over time
– Relationship of actual implementation and proposed plans

13. icfi.com | 13
Lessons About This Evaluation from Year 1
LESSONS
 Caution interpretation based on the period of data collection.
 Use crosswalk to address 60+ evaluation questions.
 Ensure common definitions of program services.
 Consider ways to verify information across data sources.
 Maximize the use of online surveys.
 Leverage various strategies to obtain sufficient parent response rates.
 Analyze data at multiple levels (school and district).
 Utilize district-level case studies to understand the context in which
implementation occurs.

14. 14
O’Donnel, B., Briggs, A., Dervarics, D., Horwood, T., Sun, J., Alexander, A.,
Zumdahl, J., & Rhodes, J. (2013, September). Annual Implementation Report #1:
Texas GEAR UP State Grant Evaluation. Report prepared for the Texas Education
Agency by ICF International. Available online at:
http://www.tea.state.tx.us/WorkArea/linkit.aspx?LinkIdentifier=id&ItemID=2576980765
9&libID=25769807662#25769807659
For more information, the report is publicly available:

15. icfi.com | 15
Assessing the Fidelity of
Implementation in the
Diplomas Now Evaluation
October 18, 2013
Presented to:
American Evaluation Association
2013 Conference
Felix Fernandez
Aracelis Gray

16. icfi.com | 16
Overview of Diplomas Now Study
Diplomas Now is school turnaround model that unites three
organizations – Talent Development, City Year, and Communities
In Schools
A random assignment design
Sixty-two schools in 11 districts across the country participating
in the study.
Study will compare student outcomes in the 32 middle and high
schools that implement DN to those in the 30 schools that do
not.

17. icfi.com | 17
Overview of Diplomas Now Implementation Study
 Overall goal: document implementation in the 32 DN schools.
 Research Questions:
– How much variation in implementation fidelity was there across sites?
– What were the largest challenges to implementing the DN model?
– What were the most important ways in which the intervention as implemented
differed from the intervention as planned?

18. icfi.com | 18
DN Fidelity of Implementation
 Fidelity of implementation is based on the DN Logic Model and measured by
the Fidelity of Implementation Matrix.
 The matrix is made up of 111 separate components, 62 of which were
identified as critical to adequate implementation.
 The fidelity matrix consist of 9 inputs ranging from program staff training,
family and community involvement and student supports.

19. icfi.com | 19
DN Fidelity of Implementation
 That is…
Input
1
Component
X
Component
Y
Component
Z

20. icfi.com | 20
DN Fidelity of Implementation
 And…
Overall
Fidelity
Input
1
Input
2
Input
3
Input
4
Input
5 Input
6
Input
7
Input
8
Input
9

21. icfi.com | 21

22. icfi.com | 22
Overview of Fidelity Matrix
– Program Staff Training and Professional Development
• 18 individual components, 15 of which are critical
– Integrated On-Site Support (Critical Input)
• 11 individual components, 9 of which are critical
– Family and Community Involvement
• 6 individual components, 1 of which is critical
– Tiered Intervention Model (Critical Input)
• 3 individual components, 2 of which are critical
– Strong Learning Environments (Critical Input)
• 6 individual components, 4 of which are critical
– Includes 1 MS and 1 HS specific critical component

23. icfi.com | 23
Overview of Fidelity Matrix
– Professional Development and Peer Coaching
(Critical Input)
• 5 individual components, 2 of which are critical
– Includes 1 HS specific component
– Curriculum for College Readiness
• 24 individual components, 4 of which are critical
– Includes 7 MS and 17 HS specific components
– Student Supports (Critical Input)
• 24 individual items, 19 of which are critical
– Student Case Management (Critical Input)
• 14 individual items, 5 of which are critical

24. icfi.com | 24
DN Fidelity of Implementation
 Divided into two metrics, a categorical rating and a continuous score:
1. Implementation Rating (categorical measure): focus on critical components
2. Implementation Score (continuous measure): allow for assessment of greater
variability between sites
 Together they provide the flexibility to:
– Look at categories that emerge
– See if scores vary by rating, amount of overlap, etc.
– Study relationships between implementation and outcomes

25. icfi.com | 25
Data Sources
 Fidelity of Implementation Data stem from the following sources:
– Diplomas Now Implementation Support Team (DNIST) Survey
– School Transformation Facilitator (STF) Survey
– Citi Year Program Manager (CYPM) Survey
– Communities In Schools (CIS) Site Coordinator (SC) Survey
– Communities In Schools (CIS) Site Records

26. icfi.com | 26
Fidelity of Implementation: Strong Learning Environments
Component Operational definition Fidelity Scale Criterion Critical Sample Response
Strong Learning Environments
Small Learning
Communities
Teams of Teachers working with the
same small group of students
0: No
1: Yes
1: Adequate/
High Fidelity
YES 1:Yes
Interdisciplinary
Teams
Frequency of Interdisciplinary team
meeting
0: do not /rarely occur
1: occur monthly
2: occur bi-weekly
3: occur weekly
4:occur multiple times a
week
5: occur daily
4: Adequate
5: High Fidelity
YES 4:occur multiple
times a week
DN Site-Based
Meeting
Admin, STF, Program Manager, Site
Coordinator hold brief review of
program implementation (approx. 30
minutes)
0: Once a month or less
1: Biweekly
2: Weekly or more
frequently
1=Adequate,
2= High
Fidelity
YES 0: Once a month
or less
DN Site-Based
Collaboration
Site based collaborative (Admin, STF,
PM, Site Coordinator) have norms for
collaboration, standards for
communication, and frameworks for
decision making
0: No
1: Partially/In Process
2: Yes
1=Adequate,
2= High
Fidelity
No 1: Partially/In
Process
4x4 Block (High
School Only
Question)
Four 75-90 minute class periods that
meet daily
0: No
1: Hybrid/Acceptable
Alternative
2: Yes
1= Adequate
2= Highly
Fidelity
YES 0: No

27. icfi.com | 27
Implementation Rating
The implementation rating focused on the “Critical to
Fidelity/Adequate Rating” column within the fidelity matrix.
Using this column each input (e.g., program staff professional
development) of the DN model was provided with one of two
ratings:
1. “Successful” - have met all components identified as critical
2. “Developing” - did not meet one or more critical components
In addition to critical components, critical inputs have also
been identified (i.e., inputs critical to an adequate
implementation).

28. icfi.com | 28
Implementation Rating
 Individual input ratings served as the basis for the site-level fidelity rating,
which has been broken up into four parts:
1. Low: successful on less than 3 critical inputs
2. Moderate: successful on at least 3 critical inputs
3. Solid: successful on at least 5 critical inputs
4. High: successful on 8 or more inputs including 5 critical inputs

29. icfi.com | 29
Example: Implementation Rating
 Implementation Rating only takes into account components identified as
critical. In this case:
– Teams of teachers working with the same small group of students
– Frequency of interdisciplinary team meetings
– DN site-based meeting
– 4x4 Block
 Given our sample responses this site has met the criterion for adequate
implementation for teams of teachers and frequency of interdisciplinary team
meetings but not DN site-based meetings or 4x4 classroom blocks.
 It would therefore receive an implementation rating of “Developing” on this
input.

30. icfi.com | 30
Program Staff Professional Development
DN Fidelity Implementation Rating Flowchart
Successful
Developing
Integrated On-Site Support*
Tiered Intervention Model*
Professional Development and Peer Coaching*
Student Supports*
Student Case Management*
Family and Community Involvement
Strong Learning Environments*
Curriculum for College Readiness
Successful
Successful
Successful
Successful
Successful
Successful
Successful
Successful
Developing
Developing
Developing
Developing
Developing
Developing
Developing
Solid Implementation
(successful on 5 critical inputs)
High Implementation
(successful on 8+ inputs)
ModerateImplementation
(successfulonatleast3criticalinputs)
LowImplementation
(successfulonlessthan3criticalinputs)
* indicates critical inputs
Developing

31. icfi.com | 31
Implementation Score
The implementation score focused on the “Fidelity Scale”
column within the fidelity matrix.
Using this column each site received an input score,
calculated as the equally weighted sum of the site’s
fidelity scales divided by the total number of
components.
The average of the 9 individual input scores then
formed the site-level implementation score.

32. icfi.com | 32
Example: Implementation Score
 Implementation scores are calculated by taking the sum of the weighted
response divided by the total number of components.
– Scale scores are equally weighted, for example, a component scaled 0-2 would be
recoded 0=0, 1=.5, and 2=1
 Adding up the weighted fidelity scale responses would equal 2.3
(1+.8+0+.5+0).
 There are 5 Strong Learning Environments components.
 The site’s implementation score for this input would then equal 2.3 divided 5
or .46.

33. icfi.com | 33
Program Staff Professional Development
Integrated On-Site Support
Tiered Intervention Model
Professional Development and Peer Coaching
Student Supports
Student Case Management
Family and Community Involvement
Strong Learning Environments
Curriculum for College Readiness
X / 18
Inputs
Input
Score
(X / 18) / 9
X / 11 (X / 11) / 9
X / 6 (X / 6) / 9
X / 3 (X / 3) / 9
X / 5 (X / 5) / 9
X / 5 (X / 5) / 9
X / 17 (X / 17) / 9
X / 24 (X / 24) / 9
X / 14 (X / 14) / 9
Site-Level Score
+
+
+
+
+
+
+
+
Note: X is the equally weighted sum of fidelity scale components. Sample calculations provided are only for
HS data.
DN Fidelity Implementation Score Flowchart

34. icfi.com | 34
Fidelity of Implementation
 Independently, each measure provides useful but different information.
 Together, they provide flexibility in understanding implementation, allow for
detailed discussion of site fidelity, and help to shape an implementation
story.

35. icfi.com | 35
Evaluating a Career Pathways
Research Initiative
PathTech: Successful Academic and
Employment Pathways in Advanced
Technology
Kristen Peterson
October 18, 2013
Presented to:
American Evaluation Association
2013 Conference

36. icfi.com | 36
The PathTech Program

37. icfi.com | 37
Background on the PathTech Project
 Funded through a grant from the National Science Foundation (NSF) under
the Advanced Technological Education (ATE) Program
 ATE promotes the improvement of education for science and engineering
technicians entering high-technology fields
 The ATE program supports many different types of activities including:
– Articulation between two-year and four-year programs
– Career pathways
– Curriculum development
– Educator professional development
– General research advancing the understanding of educating technicians for careers in
high-technology fields

38. icfi.com | 38
Background on the PathTech Project
 Successful Academic and Employment Pathways in Advanced Technologies
(PathTech)
– A research study examining the progression of students from high school into
advanced technology programs and into the workforce
– A four-year study currently entering the third year of the project
 Collaborative grant awarded to the University of South Florida (USF)
– Grant partnership includes USF researches, the Florida Advanced Technological
Education Center (FLATE) and four south Florida Community Colleges
• Hillsborough Community College
• Polk State College
• St. Petersburg College
• State College of Florida

39. icfi.com | 39
PathTech Research Design and Evaluation

40. icfi.com | 40
PathTech Research Questions
1. Who enrolls in engineering technology (ET) programs out of high school?
– How are student demographic and academic characteristics related to ET
enrollment?
– How do students learn about ET programs?
– How can the pathway from high school into ET programs be improved?
2. How do ET students benefit from enrolling (in degree programs) and
earning degrees through these programs?
– What are the most critical steps in ET degree attainment from enrollment through
gatekeeper courses and to the degree?
– How do these students become ET program graduates?
– How do the ET students differ from comparable students in their degree and
employment outcomes?

41. icfi.com | 41
Design Considerations for PathTech
 Mixed-methods study
 Quantitative Data and Analysis:
– Descriptive statistics and empirical analysis with quantitative data from state
databases
 Qualitative Data and Analysis:
– Ethnographic and qualitative analyses of engineering technology programs
– Three data sources:
• interviews with community college students,
• Interviews with students at feeder high schools,
• Interviews with local industry partners

42. icfi.com | 42
Evaluation Approach
 External evaluation of PathTech complements and supports the efforts of the
PathTech research team and involves:
1. Monitoring the progress of various aspects of the project
2. Providing objective reviews of project instruments, plans, reports and
other materials
3. Serving as an external resource for technical advice
 Designed a flexible evaluation model to account for a dynamic work plan
– Developed an annual workplan tracker
– Update the workplan tracker monthly
– Facilitate monthly status calls with the research team

43. icfi.com | 43
PathTech Evaluation Challenges

44. icfi.com | 44
Evaluation Challenges: Quantitative Data Analysis
 Using data from the Florida Department of Education’s Florida Education and
Training Placement Information Program (FETPIP)
– Initial requests for data went unanswered
– New policy to not release FETPIP employment data in conjunction with supplemental
educational data that includes demographic data
 Pursuing data from other sources including from the National Academy
Foundation (NAF)
– The NAF is a network of over 500 career-themed academies across the country which
include engineering as a career theme.
– The data would provide longitudinal, national, state and regional student-level data
including data on academic performance, demographic characteristics and academy
assessments.

45. icfi.com | 45
Evaluation Challenges: Qualitative Data Collection
 Multiple sources of qualitative data including high schools with a focus on
engineering technology, community colleges and engineering technology
industry partners
– Initial pilot study at one community college, high school, and industry
– Data collection and analyses are underway for community college and industry
employers and recruiters
– Challenge recruiting high schools with relevant engineering technology programs
 Participant and stakeholder buy-in across sites
– Particular challenge among area high schools

46. icfi.com | 46
Evaluation Challenges: External Evaluation Considerations
 Challenge: Monitoring progress with a flexible project design and dynamic
work plan
– Tracking changing timelines
– Reporting on initial and updated benchmarks
 Response: Task management and evaluation tools
– Monthly progress meetings
– Flexible workplan task tracker, designed to accommodate high-level and individual
task changes

47. icfi.com | 47
Workplan Tracker

48. icfi.com | 48
Current Project Status and Next Steps
 Qualitative pilot studies have been completed with high school students,
community college students, and industry partners
– Pilot interviews were transcribed and thematically coded
– Initial findings indicate three main factors influencing engineering technology pathways
and several pathways leading into engineering technology fields
 Interview protocols have been finalized and community college students have
been recruited into the study
 20 in-depth interviews with industry members were completed, and
transcriptions and analysis are underway
 Ongoing challenges obtaining quantitative data; pursing alternate data sources
 Multiple publications in progress
 Continue to monitor the research team’s progress and help overcome barriers

49. icfi.com | 49
Contact Information
Program Evaluation
 Barbara O’Donnel, Principal, Barbara.ODonnel@icfi.com
 T.J. Horwood, Senior Manager, Thomas.Horwood@icfi.com
Texas GEAR UP State Grant
 Ashley Briggs, Senior Associate, Ashley.Briggs@icfi.com
 Chuck Dervarics, Expert Consultant, Chuck.Dervarics@icfi.com
Diplomas Now
 Felix Fernandez, Technical Specialist, Felix.Fernandez@icfi.com
 Aracelis Gray, Senior Manager, Aracelis.Gray@icfi.com
PathTech
 Kristen Peterson, Senior Associate, Kristen.Peterson@icfi.com