The Value of Evaluation

312 views

Published on

Explains the process and benefits of conducting different program evaluations, and the Heldrich Center's work in evaluating over 30 education and workforce programs.

Published in: Education
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
312
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
0
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • So what is evaluation?
  • Note that the only data we have for both treatment and comparison groups is on graduates from training.
  • We matched the 230 or so individuals who participated in the program in 2004 and 2005 with their records in the New Jersey Unemployment Insurance Wage Database to which almost all employers in the state report their employees’ wages. In none of the 12 quarters before enrollment was the average wage greater than $3000 in any quarter. In last four quarters before enrollment the average wage was below $2500 a quarter. In the 12 quarters before enrollment less than 50 percent of program participants were employed in any given quarter. The point is that the program does not engage in creaming.
  • The first bullet compares the absolute post-graduation wages of the Newark Essex graduates with the post-training wages of the One-Stop trainees. The second shows that the Newark Essex graduates experience greater wage growth from before training to after training than the One-Stop trainees. These results are extremely positive, but we then wondered whether they were being driven by the fifty percent of adults who managed to obtain apprenticeships. Maybe the graduates who didn’t get apprenticeships were doing very poorly. So what we did was we re-ran the comparison and excluded the individuals who had earned apprenticeships. And what did we find …
  • The first bullet compares the absolute post-graduation wages of the Newark Essex graduates with the post-training wages of the One-Stop trainees. The second shows that the Newark Essex graduates experience greater wage growth from before training to after training than the One-Stop trainees. These results are extremely positive, but we then wondered whether they were being driven by the fifty percent of adults who managed to obtain apprenticeships. Maybe the graduates who didn’t get apprenticeships were doing very poorly. So what we did was we re-ran the comparison and excluded the individuals who had earned apprenticeships. And what did we find …
  • The Value of Evaluation

    1. 1. The Val ue of Eval uat i on John J. Heldrich Center for Workforce Development April 8, 2011 Presentation to the New Jersey County Colleges’ Institutional Advancement Affinity Group William Mabe, PhD, Director of Research and Evaluation
    2. 2. John J. Heldrich Center for Workforce Development 22 About the Heldrich Center  The Heldrich Center is a research and policy institute at Rutgers dedicated to raising the effectiveness of the American workplace by strengthening workforce education and training programs  Founded in 1997, the center employs 18 full-time professional staff and faculty representing an array of disciplines, from economics and human services to business and public policy
    3. 3. John J. Heldrich Center for Workforce Development 3 Heldrich Center Research and Evaluation  Have evaluated over 30 education and workforce programs  Selected evaluation projects include: – A profile of performance outcomes of New Jersey’s community colleges – Evaluation of Kessler Foundation funded grantees – Evaluation of US Department of Education financed Parent Information and Resource Centers
    4. 4. John J. Heldrich Center for Workforce Development 44 Benefits of Evaluation • Learn what works and what does not work so resources can be directed to most effective programs • Identify barriers to success and program weaknesses • Provide evidence for program sustainability • Document best practices for replication
    5. 5. John J. Heldrich Center for Workforce Development 5 What Evaluation is NOT
    6. 6. John J. Heldrich Center for Workforce Development 66 Evaluation Is Not • Auditing • A gotcha • Needs assessment • Just about the numbers • Customer satisfaction
    7. 7. John J. Heldrich Center for Workforce Development 77 Evaluation Is • Systematic and objective process by which a researcher assesses the quality, effectiveness, or value of a program  Systematic--follows established rules of scientific inquiry  Objective--any neutral observer would arrive at the same conclusions about the program if she were to use the same methods  Assess program quality by studying how the program operates  Effectiveness--whether the program achieves its intended goal  Value assessment places the effectiveness of the program in the context of its costs
    8. 8. John J. Heldrich Center for Workforce Development 88 Types of Evaluations • Process Evaluations:  Identify barriers to successful implementation  Identify which program components are effective and which are ineffective at helping program achieve its goals  Recommend strategies for modifying program to strengthen it  Tend to use more qualitative data • Outcome Evaluations:  How well is the program achieving it goals? Is it effective?  Tend to use more quantitative data
    9. 9. John J. Heldrich Center for Workforce Development 99 When Programs Should Be Evaluated • Before the program begins. In the early stages of developing any program, staff should plan the program to accommodate evaluation • Data collection should be ongoing • Program should be evaluated at regular intervals
    10. 10. John J. Heldrich Center for Workforce Development 1010 Life-Cycle Model of Program Evaluation • Like products, programs have life cycles: • conceptualization • piloting • widespread implementation • maturity • (possibly) phase-out • Type of evaluation to be conducted is a function of where a program is in the life-cycle
    11. 11. John J. Heldrich Center for Workforce Development 1111 How to Approach Evaluation • At what stage is the program in its life-cycle? • What evidence would be needed to convince a neutral observer that the program is effective? That it’s being implemented well? • If this program were successful what would I expect to observe?  Look for the observable implications of program success  Observable implications can be measured, either qualitatively or quantitatively • Focus on both process and outcomes
    12. 12. John J. Heldrich Center for Workforce Development 1212 How to Evaluate • Identify stakeholders • Types of data: qualitative and quantitative • Methodologies of data collection • Individual interviews • Focus groups • Surveys • Observation of program activities • Collection of programmatic and administrative data • Some methodologies can be used to collect qualitative and quantitative data, e.g., surveys • When to collect data • Design program so that needed data can be collected from day one
    13. 13. John J. Heldrich Center for Workforce Development 1313 Data Heldrich Center Has Used for Program Evaluations • New Jersey Unemployment Insurance Wage Record Data • Quarterly census of earnings of over nearly all workers in NJ • Heldrich Center has access to UI wage records through data sharing agreement with NJ Dept of Labor and Workforce Development • Permitted to use for approved research purposes • Employment Service OSOS Data • Records of people who received training through public workforce system • Heldrich has data sharing agreement with NJLWD • Valuable for creating comparison groups • New Jersey Student Unit Record Data System • Centralized database of student enrollment and graduation records that CHE collects from you and all public and some independent institutions in the state • Heldrich has data sharing agreement with NJCHE to use for approved research purposes
    14. 14. John J. Heldrich Center for Workforce Development 14 Case Study #1: Essex County College Math Initiative
    15. 15. John J. Heldrich Center for Workforce Development 15 Program Overview Semester Key Program Elements Summer I Pilot Semester •Linked developmentalpre-algebracourse and scientific reasoningcourse •STEM specific tutors •Financialsupport Fall2009 Semester & Spring 2010 Semester •Collegelevel Math course with a supported recitation •STEMspecific tutors •Financialsupport •Access to a dedicatedcomputer lab •Feedbackoriented homework structure.
    16. 16. John J. Heldrich Center for Workforce Development 16 Key Differences: STEM Algebra Course versus College’s Regular Algebra Course • Revised curriculum: – Focus on concepts – New textbook – Coverage of more difficult topics • Mandatory recitation • Teaching Assistant • Financial support for students • Mandatory homework • Different grading structure
    17. 17. John J. Heldrich Center for Workforce Development 17 Evaluation Design • Process and outcome evaluations • Use qualitative and quantitative data collection methods • Interdisciplinary evaluation team • Submitted interim reports to provide actionable information in a timely manner • Focused on understanding the culture and politics of the school as part of the evaluation process • Considered the views of all stakeholders
    18. 18. John J. Heldrich Center for Workforce Development 18 Qualitative Data Collection Method Time Frame Number of Participants Course Observations Three times per course per semester Eighteen observations Instructor Interviews Pre-Post Semester Twelve interviews with five different instructors Student Focus Groups Post Semester Five groups consisting of 5-8 students Tutor Focus Groups Post Semester Four groups consisting of 5-10 tutors Meetings, phone calls and e-mails with administrative staff Regularly throughout the semester – (Approx. two to three times per month) Two
    19. 19. John J. Heldrich Center for Workforce Development 19 Quantitative Data Collection • Administrative Data – Program level data is collected from the program administrator – Includes program participant information • Preliminary College level data is collected from the college data administrator – Includes grade information for ALL students enrolled in College Algebra – Retention information, and follow up grade information is forthcoming
    20. 20. John J. Heldrich Center for Workforce Development 20 Notable Findings from Process Evaluation • Increased social integration helped students establish a sense of community. • The STEM Program promoted small group interactivity and encouraged students to prioritize self study. • Students sought opportunities to interact with peers and more advanced students as a learning strategy. • Students began to see themselves and their peers as resources and active participants in a dynamic learning process • Poor faculty collaboration and planning led to an ineffective implementation of the interdisciplinary component of the learning community.
    21. 21. John J. Heldrich Center for Workforce Development 21 Outcomes Comparison • 76 percent of students in STEM Algebra course passed in Spring 2010 • Compared with 54 percent of all non-STEM majors • Compared with 56 percent of STEM majors
    22. 22. John J. Heldrich Center for Workforce Development 22 Study Limitations • Results may be an indicator of the redesigned course’s effectiveness, but it may also be a product of: – Students self-selected into the STEM program – Financial incentives may have impacted student behavior in the STEM class – The grading structure for the STEM section consisted of more elements than the grading structure of the other MTH-100 courses – The curriculum in the STEM section was more difficult than the curriculum of the other MTH-100 courses
    23. 23. John J. Heldrich Center for Workforce Development 23 Case Study #2: Newark Pre-Apprenticeship Program
    24. 24. John J. Heldrich Center for Workforce Development 24 Program Overview • Program for low-income women and minorities in Newark that prepares them for a union apprenticeship in the construction industry • Mostly basic skills preparation but also addresses other barriers to employment, such as suspended driver’s licenses • Participants’ earnings low. Demographics: predominantly (89 percent) black, some Latinos • Research Question: Does this program increase participants’ earnings?
    25. 25. John J. Heldrich Center for Workforce Development 25 How to Answer the Research Question?  What we really want to know: what would have happened to participants had they not participated • Just look at what participants earn after the program? – No way to know if that’s a “lot” • See if they make more after the program than before? – Many entered the program because they were unemployed • See if they’re doing better than similar people who did not participate in the program? – Promising but potentially problematic: applicants more motivated than non-applicants – Need a way to compare to similarly motivated individuals – Need a way to identify similar people
    26. 26. John J. Heldrich Center for Workforce Development 26 Creating a Valid Comparison Group • Taking motivation into account: – Comparison pool: individuals who graduated from training programs through the One Stop in Newark at the same time as program graduates • Thousands of people completed training in Newark at this time, which ones are most similar? – Similar age – Same race – Same sex – Similar employment and earnings history • Use statistical software to match program graduates to most similar completers of training programs in Newark
    27. 27. John J. Heldrich Center for Workforce Development 27 Study Design • Data – New Jersey Unemployment Wage Record data – Americas One-Stop Operating System • Methodology – Matched program participants by social security number with their records in the NJ UI Wage Record and OSOS data – Probabilistic matching to generate a comparison group – Parametric statistical models to estimate the program’s effect • Sample – Treatment: 129 individuals who completed the program and who were 22 years old or older at enrollment – Comparison: Matched individuals from ASOS data
    28. 28. John J. Heldrich Center for Workforce Development 28 Work Histories of Program Graduates • Program graduates tend to have: – Limited employment histories – Low average earnings – Limited academic skills (as measured by TABE) • Twelve percent of graduates had formerly been incarcerated
    29. 29. John J. Heldrich Center for Workforce Development 29 • Program graduates earn more than similar individuals who receive One-Stop training. • Program graduates experience greater wage growth after training than this comparison group. • Subgroup analysis: Youth participants in program do no better than comparison group Program Graduates Out Earn Comparison Group Members
    30. 30. John J. Heldrich Center for Workforce Development 30 Earnings Progression (Post-Matching)
    31. 31. John J. Heldrich Center for Workforce Development 31 • Study looked at graduates from 2004/2005 and earnings through mid-2006, during real estate boom • Graduates versus participants • Selective nature of program means cannot rule out some differences, though pre-training earnings help control for ability and motivation Study Limitations
    32. 32. John J. Heldrich Center for Workforce Development 3232 Questions and Discussion
    33. 33. John J. Heldrich Center for Workforce Development 33 Heldrich Center Contact Information William Mabe, PhD, John J. Heldrich Center for Workforce Development, Rutgers University – billmabe@rutgers.edu or 732-932-4100, ext. 6210 Heldrich Center Website: www.heldrich.rutgers.edu

    ×