Finding what works helping young adults transition into adulthood
The Need for Rigorous Evaluation ofInterventions to Improve theTransition to Adulthood for Youth inState CareMark E. CourtneySchool of Social Service Administration and Chapin HallUniversity of Chicago
My Purpose TodayPresent recent research on foster youths’ transitions to adulthoodDescribe the focus of social policy on this populationSummarize the weak evidence regarding the effectiveness of interventions for this population
How do foster youth fare during the transition?
Midwest Study Design and SampleLargest prospective study of foster youth making the transition to adulthood since the Foster Care Independence Act of 1999Collaboration between state child welfare agencies and the research teamFoster youth in Iowa, Wisconsin and Illinois who: Were still in care at age 17 Had entered care before their 16th birthday Had been placed in care because they were abused, neglected or dependent Not originally placed because of delinquencyData from in-person interviews (structured and in-depth qualitative) and government program administrative data
Study Design and Sample (continued)Wave Year Number Response Age at Interviewed Rate interview 1 ’02 – ’03 732 96% 17 – 18 2 ‘04 603 82% 19 3 ‘06 591 81% 21 4 ‘08 602 82% 23-24 5 ’10 – ’11 596 83% 26
Young Men’s and Young Women’s Employment72% employed during year; mean earnings among employed = $13,989
Family Formation Among Young Women 19% of women with children have a nonresident child
Family Formation Among Young Men 66% of men with children have a nonresident child
Young Women’s Criminal Justice SystemInvolvement
Young Men’s Criminal Justice SystemInvolvement
Summary of What We Know About EarlyAdult Outcomes Post ChafeeOutcomes are relatively poor across a variety of domainsTrends are generally problematic: Declining engagement in education, though some are still in school Gradually increasing but poor engagement in the workforce Many non-resident children Troubling levels of justice system involvement continuing through mid 20s Functioning in other domains (e.g., mental and behavioral health, risk behaviors, victimization) is also poorOutcomes vary by gender; males fare worseDespite a sobering picture overall, many young people leaving the care of the state do well
16US Social Policy and the Transition toAdulthood for Foster Youth
U.S. Demographic, Developmental, andPolicy ContextThe transition to adulthood in the U.S. is taking longer Markers of the transition are happening later; half of young people between 18-24 live with a parent $38k in direct support between 18-34Developmental psychologists describe a new period of “emerging adulthood”Yet, U.S. policy provides little support for young adults
U.S. Policy on Foster Youth in TransitionResearch from 1990s continues to show poor outcomes1999 Foster Care Independence Act $140 million per year allocated to states Funds a broad range of services Up to 30% of funds can be used for room and board Allows states to extend Medicaid to foster youth through age 21 Amendment to law allows appropriation up to $60 million per year to fund education/training vouchers for up to $5000 per year through age 23 Creates outcome reporting requirements and devotes 1.5% of funds to rigorous evaluation of promising programs
A Brave New World: The Fostering Connectionsto Success Act of 2008Among its provisions, the law: Extends Title IV-E funding (including guardianship and adoption subsidies), at state option, to age 21 Youth must be 1) completing high school or an equivalency program; 2) enrolled in post-secondary or vocational school; 3) participating in a program or activity designed to promote, or remove barriers to, employment; 4) employed for at least 80 hours per month; or 5) incapable of doing any of these activities due to a medical condition Existing IV-E protections remain, including ongoing court oversight of state foster care provision
But…Evidence of What Works is LackingCochrane collaboration review of evaluation research on IL programs (Montgomery et al, 2006) found no rigorous studies: “Further research incorporating randomized designs is both feasible and necessary”Recent ACF-funded randomized evaluations: No impact of life skills training, tutoring/mentoring, and employment support Massachusetts Adolescent Outreach had some positive effects, but those appear to be mediated by the program’s impact on youth remaining in care past age 18.The bottom line: Too many programs are poorly targeted, have poorly developed logic models, and are not intensive enough to influence outcomes for youth making the transition to adulthood from foster care.WE NEED RIGOROUS EVALUATION RESEARCH!!!
Evaluation of Youth Villages’ Transitional Living Program John MartinezDeputy Director, Health and Barriers to Employment Policy Area
Overview of Session Introduction to MDRC and evaluation Who is in the study Evaluation Status2
Who is conducting the evaluation? MDRC, a non-profit, non-partisan education and social policy research organization and intermediary Based in New York City with a regional office in Oakland, CA Dedicated to learning what works best to improve the lives of low-income families Nearly 40 years of experience evaluating social policy programs3
Who is funding the evaluation? The Edna McConnell Clark Foundation The Annie E Casey Foundation The Bill and Melinda Gates Foundation4
Components of the Evaluation (I) Impacts: To what extent does TL improve outcomes for youth? Which approaches are most effective for whom? On what outcomes (e.g. housing stability, educational outcomes, labor market success, reduced crime) Over what period of time (e.g. length of follow-up) To what magnitude? For what subgroups? Use Random Assignment Research Design Data sources: public records data, baseline forms, and surveys5
Components of the Evaluation (II) Program implementation: What services are provided? How are they delivered? What challenges are encountered? Are the challenges related to serving youth aging out of state care? Measuring enrollment and participation rates Measuring implementation fidelity Measuring the contrast Data sources: surveys, field research, YV MIS, in-depth interviews with youth Costs and Benefits: What are the costs? Do benefits outweigh the costs?6
Impact Analysis Design Evaluation uses random assignment, the most reliable way to assess what difference a program makes Youth eligible for TL were assigned, by chance, to one of two groups: TL Program Group: receives TL services Community Services(CS) Group (Control Group): not eligible to receive TL services for up to five years MDRC will follow both groups for at least three years (and possibly longer, depending on funding)7
Random Assignment Participants meet program criteria Participants give consent Baseline data collected Random Assignment Program group Control group Enroll in program Receive other services in the community8
Why Random Assignment? Gold standard of research as it is the most reliable way to measure impacts: Ensures motivation levels and personal characteristics of youth in program and control groups are same on average at beginning of program Youth in the control group illustrate what would have happened if not for the program Any subsequent difference in outcomes can be attributed to the program with the highest confidence Widely used in public service settings Endorsed by OMB, DOE and other federal agencies Fair and equitable way to determine who receives the program9
What is the baseline information form? Youth completed a baseline survey prior to random assignment Baseline data provides a “picture” of these youth at study entry Multiple uses Allows researchers to assess whether random assignment worked Could be used in impact analysis models Can also be used to determine whether TL works differently for different types of youth Presenting data today on 1,225 study participants
Did RA work? Goal: two research groups in which the only difference is that one group was eligible to receive TL and one was not Baseline data can help support that the two groups were equivalent Data suggests this was the case: of 22 variables, only 2 had a SS difference between the 2 groups (and we would expect that to happen by chance)
DemographicsCharacteristic Full Sample (%)Gender Male 52 Female 48Race Hispanic 5 White/non-Hispanic 51 Black/non-Hispanic 38 Other/non-Hispanic 6
Age at random assignment Age at RA 9% 20% 18 19 20-24 71%
Age at 1st custody placement 1% 6% 6% LT 1 1-5 23% 6-10 11-14 64% 15-18
Contact with biological parentContact with biological Contact with biologicalmother father Every day Every day At least 16% At least 25% 1X/wk 1X/wk 43% At least At least 12% 8% 1X/mo 53% 1X/mo 8% LT 1X/mo LT 1X/mo 11%8% 16% Never Never
Other characteristicsCharacteristic Full Sample (%)Contact with any other relative at least 1X/mo 88Pregnant at baseline 4Has any children 17Enrolled in school 54Ever repeated a grade or held back 43Ever suspended from school 81Ever arrested 64
Current Status Random assignment complete 1322 youth randomly assigned within two years Most field work associated with the implementation study is complete Survey fielding is ongoing Very high response rates (about 85 percent)
What’s Next Survey fielding will continue (survey firm should wrap up winter/spring 2014) Implementation report slated for publication in January 2014 Impact report slated for publication in spring/summer 2015
Thank You 20 John Martinezjohn.email@example.com 212-340-8690 www.mdrc.org
Discussion Topics• Embarking on a random assignment evaluation• Confronting the ethical issues• Recruiting study subjects• Monitoring program fidelity• Controlling study costs• Awaiting results
Why do a random assignment evaluation? PRO CON• Provides a benchmark • Frightening• Value in having • Time consuming independent evaluation • Referral sources/ service• Exciting purchasers don’t require it• Opportunity for program • Youth Villages already has improvement an on-going outcome evaluation process• Push toward evidence- based practices from govt. and foundation funders• Meets organizational goal of increasing use of evidence-based services
Confronting the Ethical Issues of Random AssignmentForty percent of youth in thestudy are denied entry into the TL Program. How do we justify that?
Recruiting Study SubjectsThe expectation: The reality:
It wasn’t as easy as we thought!1,3001,2001,100 Original goal – 1,6001,000 Adjusted goal – 1,300 900 800 700 600 500 400 300 200 100 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 Month Goal Actual
Monitoring Program FidelityIn some ways, this has been the easy one! • Program Model Adherence Reviews • Balanced Scorecard Is that enough?
Controlling Study CostsThe evaluation is funded by the Edna McConnell Clark Foundation and the Bill and Melinda Gates FoundationWhat are the provider’s costs? • Study coordinator (full time) • Leadership staff time • Assessors (to determine eligibility and appropriateness for program) • Training and travel
Awaiting Study Results Planning Begins – July 2008 Study Recruitment Begins – October 2010 One Year Follow-up Begins – November 2011 Study Recruitment Ends – October 2012 One Year Follow-up Completed – January 2014 SixYears! Preliminary Report on One Year Outcomes – July 2014
Lessons learned?• Go in with eyes wide open• Take time to prepare• Be ready for the unexpected
A particular slide catching your eye?
Clipping is a handy way to collect important slides you want to go back to later.