This slide should run before the presentation begins.
Making justice systems fairer and more effective through research and innovation.
Helps policymakers get clear and accessible information on the economic pros and cons associated with criminal and juvenile justice investments. Bridges the gap between research and policy by putting evidence in context. What works? Is “what works” worth it? What should we do?
You can ask questions at any time by typing a question or comment in the chat box feature to the left of your screen. A CBKB staff member will respond your question or queue up your question to the speaker’s attention.
Can be matched statistically
Say what dosa is (reduces prison sentences for some drug offenders, offers community-based drug treatment)
considered the “gold standard”, but it’s expensive, often impractical (or unethical) in CJ or social services
considered the “gold standard”, but it’s expensive, often impractical (or unethical) in CJ or social services
Need meta-analysis because you can’t always do random assignment.
Comments: This transition provides an opportunity to survey the audience for questions and to quickly review what was covered in the last section. *This transition provides an opportunity to set-up how crime reduction programs might fit into cost-benefit analysis. We know that incarceration reduces crime. We know that there are diminishing marginal returns to incarceration. We also know that crime costs society. Costs are influenced by type of crime and how resources are used to arrest, convict and sentence offenders. Programs that reduce crime affect justice system costs. Some programs increase taxpayer costs in the short-term while saving taxpayer money in the long-term. Programs also generate significant benefits (both tangible and intangible). Take aways: This section will discuss how to start a cost-benefit analysis on a program or policy in your state. It will also provide some insight to how cost-benefit analysis can be a helpful decision-making tool.
Linking Evaluation and Cost-Benefit Analysis in Criminal Justice: A Practical Introduction
Linking Evaluation and Cost-Benefit Analysisin Criminal Justice: A Practical IntroductionMarch 24, 2011Elizabeth Drake, Senior Research Associate, Washington State Institute for Public PolicyValerie Levshin, Policy Analyst, Cost-Benefit Analysis Unit, Vera Institute of Justice Slide 1
WelcomeLinking Evaluation and Cost-Benefit Analysisin Criminal Justice: A Practical IntroductionThe webinar will start at 2pm.Call 1-866-740-1260 for the audio portion of thewebinar. Slide 2
Linking Evaluation and Cost-Benefit AnalysisCriminal Justice: A Practical Introduction Elizabeth Drake Valerie Levshin Slide 3
The Cost-Benefit Knowledge Bank for Criminal Justice (CBKB) is aproject of the Vera Institute of Justice funded by the U.S.Department of Justice’s Bureau of Justice Assistance. • Website (cbkb.org) • Cost-Benefit Analysis Toolkit • Snapshots of CBA Literature • Podcasts, Videocasts, and Webinars • Roundtable Discussions • Community of Practice Slide 4
What You Will Learn Today• Why evaluation is an important element of a cost-benefit analysis (CBA).• Different ways to assess program/policy impacts.• What meta-analysis is and how it can be used in a CBA. Slide 5
Today’s AgendaIntroduction and Housekeeping 5 minutesThe Role of Evaluation in CBA 5 minutesWays to Assess Program/Policy Impacts 10 minutesMeta-Analysis and CBA 30 minutesQ&A 10 minutes Slide 6
Housekeeping items Questions Use the chat feature to send us your questions at any time during the webinar. We will address your questions after each section of the presentation. Slide 7
Housekeeping itemsWebinar support and troubleshooting Call: (800) 843-9166 Email: email@example.comThis webinar is being recordedThe recording and PowerPoint will be posted to cbkb.org Slide 8
The Role of Evaluation inCost-Benefit Analysis Slide 9
What is Cost-Benefit Analysis (CBA)?• A tool to assess the pros and cons of policies and programs• A method for finding out what will achieve the greatest net benefit to society• An approach to policymaking Slide 10
CBA in Five Steps1. Determine the impact of the initiative2. Determine whose perspectives matter3. Measure costs4. Measure benefits (in dollars)5. Compare costs and benefits Slide 11
Evaluation and CBA• CBA is only possible if there is information about program/policy impact.• If you don’t know the outcome, you can’t measure the benefits and some of the costs.• You need to evaluate the initiative or draw on research to predict program/policy outcomes. Slide 12
Ways to Assess Program andPolicy Impacts Slide 13
Ways to Assess Program/Policy Impacts• Evaluate the initiative Comparison group design Random assignment design• Draw on evaluations of similar initiatives Literature-based estimate Meta-analysis Slide 14
Comparison Group DesignCompare the outcomes for people in your program to asimilar group of people not in your program For example: matching groups, pre-post. Need to match groups to make sure they’re similar. Need to consider outside factors (such as the economy) that could influence the outcome. If the groups are similar, and other factors are ruled out, then we can assume that the program affected the outcome. Slide 15
Comparison Group ExampleExample: CBA of Washington’sDrug Offender SentencingAlternative (DOSA) WSIPP’s evaluation compared the recidivism rates before/after DOSA was implemented in 1999. Evaluation results: DOSA reduced recidivism for drug offenders, not drug-involved property offenders. CBA results: benefits > costs for drug offenders. Benefit / Cost ratio = $7.25 - $9.94 for drug offenders Benefit / Cost ratio = $0.93 for drug-involved property offenders. Slide 16
Random Assignment DesignConduct a randomized assignment study to assess theimpact Participants are randomly assigned into program or “control” groups. Best way to create very similar groups, where the only difference between them is program participation. Differences in outcomes can be attributed directly to the program. e.g., if program participants have lower recidivism rates that non-participants, then we know the program reduced recidivism rates. Slide 17
Random Assignment ExampleExample: CBA of the Center forEmployment Opportunity (CEO) MDRC random assignment evaluation showed that CEO reduced recidivism rates. CBA showed that recidivism reduction generated taxpayer, victim and offender benefits. Benefit / Cost ratio = about 3 to 1 Slide 18
Make a Literature-Based Estimate• Review the literature to determine the impact of similar initiatives• Example: CBA of Raising the Age in North Carolina How will trying 16- and 17-yr-olds in the juvenile instead of the adult system affect their recidivism rates? 6 studies show that the recidivism rates are 0%-50% lower than in the adult system. CBA assumes that trying youth in the juvenile system will reduce recidivism rates by 10%. Sensitivity analysis shows how using a different recidivism reduction affects CBA results. Slide 19
Meta-Analysis• Review the literature to estimate the average effect of a program/policy on outcomes • Review all evidence • Give more weight to results of stronger evaluations• Details are up next Slide 20
Which Approach to Use?• Depends on time, resources, staff expertise• Aim for stronger research designs: the more rigorous the evaluation, the more accurate the CBA results• Random assignment, comparison group evaluation are sometimes unfeasible; need to draw on the literature instead Increasing Random assignment difficulty, Comparison group time, and Meta-analysis expertise Literature-based estimate Slide 21
Washington State Institute for Public Policy (WSIPP)Washington legislature has asked WSIPP this question:Are There Evidence-Based Policy Options That Improve Public Outcomes,but at Less Cost? Exhibit 4 Reducing Crime With Evidence-Based Options: What Works, and Benefits & Costs WSIPPWashington State Institute for Public Policy Effect on Crime Benefits and CostsEstimates as of October, 2006 Outcomes (Per Participant, Net Present Value, 2006 Dollars). Percent change in crime Benefits to Benefits to Costs Benefits (total)Notes: outcomes, & the number of Crime Victims Taxpayers (marginal program Minus “Consumer Reports” Lists:"n/e" means not estimated at this time. evidence-based studies on (of the reduction (of the reduction cost, compared to CostsPrevention program costs are partial program costs, pro-rated to which the estimate is based in crime) in crime) the cost of (per participant)match crime outcomes. (in parentheses) alternative) (1) (2) (3) (4) (5)Programs for People in the Adult Offender System Vocational education in prison -9.0% (4) $8,114 $6,806 $1,182 $13,738 Intensive supervision: treatment-oriented programs -16.7% (11) $9,318 $9,369 $7,124 $11,563 General education in prison (basic education or post-secondary) -7.0% (17) $6,325 $5,306 $962 $10,669 Cognitive-behavioral therapy in prison or community -6.3% (25) $5,658 $4,746 $105 $10,299 Drug treatment in community -9.3% (6) $5,133 $5,495 $574 $10,054 Correctional industries in prison -5.9% (4) $5,360 $4,496 $417 $9,439 Drug treatment in prison (therapeutic communities or outpatient) -5.7% (20) $5,133 $4,306 $1,604 $7,835 Adult drug courts -8.0% (57) $4,395 $4,705 $4,333 $4,767 Employment and job training in the community -4.3% (16) $2,373 $2,386 $400 $4,359 Electronic monitoring to offset jail time 0% (9) $0 $0 -$870 $870 Sex offender treatment in prison with aftercare -7.0% (6) $6,442 $2,885 $12,585 -$3,258 Intensive supervision: surveillance-oriented programs 0% (23) $0 $0 $3,747 -$3,747 Washingtons Dangerously Mentally Ill Offender program -20.0% (1) $18,020 $15,116 n/e n/e What Works? Drug treatment in jail -4.5% (9) $2,481 $2,656 n/e n/e Adult boot camps 0% (22) $0 $0 n/e n/e Domestic violence education/cognitive-behavioral treatment 0% (9) $0 $0 n/e n/e Jail diversion for mentally ill offenders 0% (11) $0 $0 n/e n/e Life Skills education programs for adults 0% (4) $0 $0 n/e n/e What Doesn’t?Programs for Youth in the Juvenile Offender System Multidimensional Treatment Foster Care (v. regular group care) -22.0% (3) $51,828 $32,915 $6,945 $77,798 Adolescent Diversion Project (for lower risk offenders) -19.9% (6) $24,328 $18,208 $1,913 $40,623 Family Integrated Transitions -13.0% (1) $30,708 $19,502 $9,665 $40,545 Functional Family Therapy on probation -15.9% (7) $19,529 $14,617 $2,325 $31,821 Multisystemic Therapy -10.5% (10) $12,855 $9,622 $4,264 $18,213 Aggression Replacement Training -7.3% (4) $8,897 $6,659 $897 $14,660 Teen courts Juvenile boot camp to offset institution time Sex offender cognitive-behavioral treatment Restorative justice for low-risk offenders Interagency coordination programs -11.1% 0% -10.2% -8.7% -2.5% (5) (14) (5) (21) (15) $5,907 $0 $32,515 $4,628 $3,084 $4,238 $0 $8,377 $3,320 $2,308 $936 -$8,077 $33,064 $880 $205 $9,208 $8,077 $7,829 $7,067 $5,186 What Can Give Washington Juvenile drug courts -3.5% (15) $4,232 $3,167 $2,777 $4,622 Regular surveillance-oriented parole (v. no parole supervision) Juvenile intensive probation supervision programs Juvenile wilderness challenge Juvenile intensive parole supervision Scared Straight 0% 0% 0% 0% +6.8% (2) (3) (9) (10) (10) $0 $0 $0 $0 -$8,355 $0 $0 $0 $0 -$6,253 $1,201 $1,598 $3,085 $6,460 $58 -$1,201 -$1,598 -$3,085 -$6,460 -$14,667 Taxpayers a Good Return Counseling/psychotherapy for juvenile offenders -18.9% (6) $23,126 $17,309 n/e n/e Juvenile education programs Other family-based therapy programs Team Child Juvenile behavior modification Life skills education programs for juvenile offenders -17.5% -12.2% -10.9% -8.2% -2.7% (3) (12) (2) (4) (3) $41,181 $15,006 $5,759 $19,271 $6,441 $26,153 $11,231 $4,131 $12,238 $4,091 n/e n/e n/e n/e n/e n/e n/e n/e n/e n/e (Better Outcomes) for Their Money? Diversion progs. with services (v. regular juvenile court) -2.7% (20) $1,441 $1,034 n/e n/e Juvenile cognitive-behavioral treatment -2.5% (8) $3,123 $2,337 n/e n/e Court supervision vs. simple release without services 0% (8) $0 $0 n/e n/e Diversion programs with services (v. simple release) 0% (7) $0 $0 n/e n/e Juvenile intensive probation (as alternative to incarceration) 0% (5) $0 $0 n/e n/e Guided Group Interaction 0% (4) $0 $0 n/e n/ePrevention Programs (crime reduction effects only) Nurse Family Partnership-Mothers -56.2% (1) $11,531 $8,161 $5,409 $14,283 Nurse Family Partnership-Children -16.4% (1) $8,632 $4,922 $733 $12,822 Pre-K education for low income 3 & 4 year olds -14.2% (8) $8,145 $4,644 $593 $12,196 Seattle Social Development Project -18.6% (1) $1,605 $4,341 n/e n/e High school graduation -10.4% (1) $1,738 $2,851 n/e n/e Guiding Good Choices -9.1% (1) $570 $2,092 n/e n/e Parent-Child Interaction Therapy -3.7% (1) $268 $784 n/e n/eProgram types in need of additional research & development before we can conclude they do or do not reduce crime outcomes:Programs needing more research for people in the adult offender system Case management in the community for drug offenders COSA (Faith-based supervision of sex offenders) Day fines (compared to standard probation) Domestic violence courts 0% (13) -22.3% (1) 0% (1) 0% (2) Comment Findings are mixed for this broad grouping of programs. Too few evaluations to date. Too few evaluations to date. Too few evaluations to date. Given the Current Level of Credible Research, What Don’t Faith-based programs 0% (5) Too few evaluations to date. Intensive supervision of sex offenders in the community 0% (4) Findings are mixed for this broad grouping of programs. Medical treatment of sex offenders -21.4% (1) Too few evaluations to date. Mixed treatment of sex offenders in the community 0% (2) Too few evaluations to date. Regular parole supervision vs. no parole supervision 0% (1) Too few evaluations to date. Restorative justice programs for lower risk adult offenders 0% (6) Findings are mixed for this broad grouping of programs. We Know? Therapeutic community programs for mentally ill offenders -20.8% (2) Too few evaluations to date. Work release programs (from prison) -4.3% (4) Too few recent evaluations.Programs needing more research for youth in the juvenile offender system Dialectical Behavior Therapy 0% (1) Too few evaluations to date. Increased drug testing (on parole) vs. minimal drug testing 0% (1) Too few evaluations to date. Juvenile curfews 0% (1) Too few evaluations to date. Juvenile day reporting 0% (2) Too few evaluations to date. Juvenile jobs programs 0% (3) Too few recent evaluations. Juvenile therapeutic communities 0% (1) Too few evaluations to date. Mentoring in juvenile justice 0% (1) Too few evaluations to date. Slide 24
Meta-Analysis and CBA: OverviewWhat I’ll cover today• Meta-analytical concepts Research design quality and discount factors• Cost-benefit concepts Resources we monetizeWhat I won’t cover today• Cost-benefit data and calculations• Meta-analytic calculations Slide 25
What is Meta-Analysis?Empirical Summarization of a Set of Literature• Meta-analysis produces an average effect on something.• The unit of measurement is an effect size, which measures the degree to which a program has been shown to change an outcome for program participants relative to a comparison group.• Not all research is of equal quality, and this greatly influences the confidence that can be placed in the results of a study.• A meta-analysis is only as good as the selection and coding criteria used to conduct the study. Slide 27
Meta-Analytic Procedures:Important Criteria to Determine Inclusion of Studies1. Search Criteria (published and unpublished sources)2. Comparison group studies • no single group, pre/post research designs3. Intent-to-treat sampling procedures • Completers only = bias treatment effect4. Crime outcomes • Prefer dichotomous outcomes • Longest follow-up period • Felony convictions Slide 28
Meta-Analytic Procedures:Standards of Rigor and Adjustments to Effect SizesAn adjustment factor is assigned to the results of individualeffect sizes based on our judgment concerning the researchdesign quality. Rating Research Design Quality Discount Multiplier 5 Random assignment None 4 Random assignment with issues .75 3 Studies that attempt to statistically control for .75 unobserved factors (e.g., regression discontinuity or natural experiment) 2 Well done comparison group study with many .625 controls 1 Less well implemented with some covariates .5 Slide 29
Meta-Analytic Procedures:Standards of Rigor and Adjustments to Effect SizesDiscount for:• Not “Real World” evaluations • Lab settings• Evaluation researcher is the program developer• Weak outcome measure (i.e., incarceration) Internally consistent set of procedures. Slide 30
Cost-Benefit ProceduresOnce we have an effect size, how much does it cost to buy that effect size, and what’s it worth to achieve it? CJS resource Effect size response and Cost-benefit Relative to base victimizations population results incurred Slide 33
Results: What Works to Reduce Crime? Change In Benefits Minus Costs, Crime per-person, life cycleAdult Offenders (# of EB Studies) (Probability: you lose $) Cog-Behavioral Treatment -4% (27) $7,100 (<1%) Correctional Education -5% (13) $7,700 (<1%) ISP: surveillance -2% (23) -$2,900 (≈53%) ISP: treatment -18% (11) $6,200 (≈13%)Juvenile Offenders Functional Family Thpy (wf) -14% (8) $23,000 (<1%) Aggression Repl. Trng (wf) -9% (4) $12,900 (<1%)(Draft 2010 Results) Slide 34
Pulling It All TogetherResults, results, and more results.1. Meta-analysis and cost-benefit analysis can inform stakeholders where resources are best utilized2. But results will only tell you the average effect3. Follow up with an outcome evaluation to ensure you are getting the results you expect An example in Washington. Slide 35
Recap of Today’s WebinarYou learned: • Why evaluation is an important element of a cost-benefit analysis (CBA). • Different ways to assess program/policy impacts. • How meta-analysis can be used in a CBA. Slide 38
Follow-upPlease complete the evaluation form as you leave this training.To receive information and notifications about upcoming webinars and other events • Visit the Cost-Benefit Knowledge Bank for Criminal Justice at http://cbkb.org. • Subscribe to receive updates from CBKB. • Follow us on Twitter at http://www.twitter.com/CBKBank.The next webinar will focus on discussing cost-benefit results with the media. Stay tuned for updates. Slide 39
This project is supported by Grant No. 2009-MU-BX K029 awarded by the Bureau ofJustice Assistance. The Bureau of Justice Assistance is a component of the Office ofJustice Programs, which also includes the Bureau of Justice Statistics, the NationalInstitute of Justice, the Office of Juvenile Justice and Delinquency Prevention, andthe Office of Sex Offender Sentencing, Monitoring, Apprehending, Registering, andTracking. Points of view or opinions in this document are those of the author and donot represent the official position or policies of the United States Department ofJustice. Slide 40