Sensitivity Analysis for Cost-Benefit Studies of Justice Policies: A Primer
Upcoming SlideShare
Loading in...5
×
 

Sensitivity Analysis for Cost-Benefit Studies of Justice Policies: A Primer

on

  • 896 views

 

Statistics

Views

Total Views
896
Views on SlideShare
639
Embed Views
257

Actions

Likes
0
Downloads
12
Comments
0

1 Embed 257

http://cbkb.org 257

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Sensitivity Analysis for Cost-Benefit Studies of Justice Policies: A Primer Sensitivity Analysis for Cost-Benefit Studies of Justice Policies: A Primer Presentation Transcript

  • Sensitivity Analysis for Cost-BenefitStudies of Justice Policies: A PrimerSeptember  13,  2012    Ben  Bryant,  Staff  Economist,  Millennium  Challenge  Corpora7on  Carl  Ma5hies,  Senior  Policy  Analyst,  Cost-­‐Benefit  Analysis  Unit,  Vera  Ins7tute  of  Jus7ce     Slide 1
  • Sensitivity Analysis for Cost-Benefit Studies of Justice Policies: A Primer Ben Bryant Carl Matthies Economist Senior policy analystDepartment of Policy and Evaluation Cost-Benefit Analysis Unit Millennium Challenge Corporation Vera Institute of Justice Slide 2 • September 13, 2012
  • AgendaIntroductions and housekeeping 5 minutesSensitivity analysis as part of 10 minutescost-benefit analysis (CBA)Perspectives included in a CBA 5 minutesDeterministic sensitivity analysis 15 minutesProbabilistic sensitivity analysis 15 minutesQ&A and conclusion 10 minutes Slide 3 • September 13, 2012
  • Housekeeping items•  Use the chat feature to send your questions at any time during the webinar.•  If you need webinar support or for troubleshooting: §  Type questions into the chat box. §  Call 800-843-9166. §  Send e-mail to help@readytalk.com.•  This webinar is being recorded.•  The recording and PowerPoint slides will be posted on cbkb.org. Slide 4 • September 13, 2012
  • Webinar previewYou will learn: § Why sensitivity analysis is a critical part of cost-benefit analysis § Different techniques for conducting sensitivity analysis § How to interpret the sensitivity analysis results Slide 5 • September 13, 2012
  • Sensitivity Analysis asPart of Cost-Benefit Analysis(CBA) Slide 6 • September 18, 2012
  • Review: Basic steps of CBA§  Specify the set of alternative projects.§  Decide whose benefits and costs count.§  Catalog the impacts and select measurement indicators.§  Predict the impacts quantitatively over the life of the project.§  Monetize all impacts.§  Discount benefits and costs to obtain present values.§  Compute the net present value (NPV) of each alternative.§  Perform sensitivity analysis.§  Recommend/choose a project. Slide 7 • September 13, 2012
  • Review: Basic steps of CBA  Specify the set of alternative projects. Can be political  Decide whose benefits and costs count.  Catalog the impacts and Requires significant topical select measurement indicators. knowledge  Predict the impacts quantitatively over the life of the project. Requires significant quantitative skills and effort  Monetize all impacts.  Discount benefits and costs to obtain present values. Involves only calculations,  Compute the net present value in theory (NPV) of each alternative.  Perform sensitivity analysis. Varies from simple  Recommend/choose a project. to an entire study Slide 8 • September 13, 2012
  • Strengths and weakness of CBASome strengths: § CBA informs and fosters transparency in policy process by making benefits and costs explicit. § It also uses a common unit of measurement for potentially more efficient allocation of resources.Some weaknesses: § Even with common measurement ($), people may disagree on how to value outcomes. § Results may vary, subject to choices about who counts and what counts. Slide 9 • September 13, 2012
  • State initiatives to legalize marijuana Decisions on Net impact tax rate and on state and regulatory local regime budgets Marijuana consumption Remove (quantity and penalties patterns for sale and of use) possessionSource: Kilmer et al. (2010) Slide 10 • September 13, 2012
  • A web of uncertain variables Decisions on Tax Net impact tax rate and Tax evasion and revenues on state and regulatory tax-induced from legal local regime shift in mix of marijuana budgets marijuana types sales Changes in Marijuana prices production and consumers face distribution costs Other Consumer factors that price influence sensitivity budgets Marijuana (e.g., criminal Non-price consumption justice and Penalties effects on (quantity and treatment removed consumption patterns costs; federal for sale and of use) spending; possession tourism)Source: Kilmer et al. (2010) Slide 11 • September 13, 2012
  • Characterizing uncertainty in a CBAThe way we characterize uncertainty is relatedto the methods we use:•  Best guess (percent change)•  Plausible bounds (deterministic, break-even)•  Probability distribution (Monte Carlo)•  Distinct stakeholder preferences (scenarios) Slide 12 • September 13, 2012
  • Questions? Use the chat feature to send us your questions. Slide 13 • September 13, 2012
  • PerspectivesIncluded in a CBA Slide 14 • September 18, 2012
  • Perspectives•  A program or policy’s net benefits can be highly sensitive to the costs and benefits included (who and what “counts”).•  Think of changing the perspective as a form of sensitivity analysis. Slide 15 • September 13, 2012
  • Which crime costs are included? Total Criminal Justice Crime TangibleOffense Crime Victim Cost System Cost Career Cost CostMurder $744,239 $392,352 $148,555 $1,285,146Rape/Sexual Assault $5,561 $26,479 $9,212 $41,252Aggravated Assault $8,635 $8,641 $2,126 $19,472Robbery $3,274 $13,827 $4,272 $21,373Arson $11,453 $4,392 $584 $16,429Motor Vehicle Theft $6,114 $3,867 $553 $10,534Residential Burglary $1,654 $4,127 $681 $6,169Larceny $479 $2,879 $163 $523Source: McCollister, French, and Fang (2004) Slide 16 • September 13, 2012
  • Which crime costs are included? Offense Total Tangible Cost Total Intangible Cost Murder $1,285,146 $8,442,000 Rape/Sexual Assault $41,252 $199,642 Aggravated Assault $19,472 $95,023 Robbery $21,373 $22,575 Arson $16,429 $5133 Motor Vehicle Theft $10,534 $262 Residential Burglary $6,169 $321 Larceny $3,523 $10 Source: McCollister, French, and Fang (2004) Slide 17 • September 13, 2012
  • Perspectives may affect project choice Annual Annual Net Net Annual Net Annual Benefit Benefit Benefit Program Estimated Impact Program Cost CJS CJS + TV CJS + TV + IV Prevents 100 motor A vehicle thefts per year $300K +$87K +$700K +$730K Prevents 10 sexual B assaults per year $300K -$35K +$20K +$2.0MCJS: Criminal justice systemTV: Tangible victim costsIV: Intangible victim costs Slide 18 • September 13, 2012
  • Questions? Use the chat feature to send us your questions. Slide 19 • September 13, 2012
  • DeterministicSensitivity Analysis Slide 20 • September 18, 2012
  • Deterministic sensitivity analysis This type of analysis examines relationships in specific, carefully chosen ways: •  Partial sensitivity analysis varies model inputs separately to see how sensitive the CBA result is to each input’s value, all else remaining equal. •  Extreme scenario analysis defines worst and best cases. •  Break-even analysis solves for model inputs when the NPV equals zero. Slide 21 • September 13, 2012
  • The base case as starting point•  A preliminary result derived from plausible or best-estimate assumptions•  A reference point or benchmark for sensitivity analysis•  A starting point—not necessarily the most likely outcome! Slide 22 • September 13, 2012
  • Example: Jail work-release program•  Inmates get three days off sentence for every two days worked.•  The daily marginal cost is slightly higher than for jail overall, but the program potentially reduces sentences and recidivism.•  Assumptions: •  A six-year time horizon starts at Year Zero. •  No recidivism costs will occur until Year One. •  Only first-time offenders participate in the program. •  Non-compliers must serve out their entire sentence. •  The program has no effect on non-compliers. Slide 23 • September 13, 2012
  • The work-release CBA model Model Inputs Benefits FormulaHiring two new corrections staff (s) npq(SR)(JMC)(0.6)Number of eligible participants (n) Cost savings from reduced jail timeParticipation rate (p) npq(SR)(BRR)(E)Program compliance rate (q) Cost savings from program effect on recidivismProgram marginal cost/day (PMC) Costs FormulaJail marginal cost/day (JMC) sInitial sentence range for eligible Additional staffindividuals (ISR) npq(SR)(PMC)(0.4)Average program failure point fornon-compliers (f) Program cost for compliersBaseline 5-year recidivism rate (BRR) np(1-q)(SR)(f) (PMC)+Change in recidivism rate among np(1-q)(JMC)(SR)participants (E) Program cost for non-compliersSentence range for recidivists (RSR)Discount rate (d) Slide 24 • September 13, 2012
  • Example: Jail work-release program Model Inputs Base-Case Estimate Hiring two new corrections staff (s) $80,000/year Number of eligible participants (n) 1,000 Participation rate (p) 0.9 Program compliance rate (q) 0.8 Program marginal cost (PMC) $75/day Jail marginal cost (JMC) $55/day Initial sentence range for eligible individuals (ISR) 120 Average program failure point for non-compliers (f) 0.1 Baseline 5-year recidivism rate (BRR) 0.5 Change in recidivism rate among participants (E) 0.2 Sentence range for recidivists (RSR) 180 Discount rate (d) 0.03 NPV for work-release program $6,520,950 The base case looks favorable, but what if there are departures from the mean values? Slide 25 • September 13, 2012
  • Partial sensitivity analysis Slide 26 • September 18, 2012
  • Example: Jail work-release program Model Inputs Base-Case Estimate Lower Bound Upper BoundHiring two new corrections staff (s) $80,000/year $ 70,000/year $90,000/yearNumber of eligible participants (n) 1,000 700 1300Participation rate (p) 0.9 0.8 1.0Program compliance rate (q) 0.8 0.7 1.0Program marginal cost (PMC) $75/day $65/day $80/dayJail marginal cost (JMC) $55/day $50/day $60/dayInitial sentence range for eligible individuals (ISR) 120 15 220Average program failure point fornon-compliers (f) 0.1 0.05 0.4Baseline 5-year recidivism rate (BRR) 0.5 0.3 1.0Change in recidivism rate among participants (E) 0.2 -0.2 0.6Sentence range for recidivists (RSR) 180 120 365Discount rate (d) 0.03 0.03 0.07NPV for work-release program $6,520,950 Slide 27 • September 13, 2012
  • Partial sensitivity analysis Change in recidivism rate among participants (E) Baseline 5-year recidivism rate (BRR) Sentence range for recidivists (RSR) Jail marginal cost (JMC) Average program failure point in non-compliers (p) Program marginal cost (PMC) Program compliance rate (q) Number of eligible participants (n)    1,300   Initial sentence range for eligible individuals (ISR) Participation rate (p) Hiring two new corrections staff (s)    90,000      70,000   Slide 28 • September 18, 2012
  • Worst-case and best-case scenarios Model Inputs Base-Case Estimate Worst Case Best CaseHiring two new corrections staff (s) $80,000/year $90,000/year $70,000/yearNumber of eligible participants (n) 1,000 1300 1,300Participation rate (p) 0.9 1.0 1.0Program compliance rate (q) 0.8 0.7 1.0Program marginal cost (PMC) $75/day $80/day $65/dayJail marginal cost/day (JMC) $55/day $60/day $60/dayInitial sentence range for eligibleindividuals (ISR) 120 15 220Average program failure point fornon-compliers (f) 0.1 0.4 0.05Baseline 5-year recidivism rate (BRR) 0.5 1.0 1.0Change in recidivism rate amongparticipants (E) 0.2 -0.2 0.6Sentence range for recidivists (RSR) 180 120 365Discount rate (d) 0.03 0.07 0.03NPV for work-release program $6.5M -$43.3M $104.0M Slide 29 • September 13, 2012
  • Break-even analysis Model Input Estimate Hiring two new corrections staff (s) $80,000 Number of eligible participants (n) 1,000 Participation rate (p) 0.9 Program compliance rate (q) 0.8 Program marginal cost (PMC) $91/day Jail marginal cost/day (JMC) $55/day Initial sentence range for eligible individuals (SR) 120 Average program failure point for non-compliers (f) 0.1 Baseline 5-year recidivism rate (BRR) 0.5 Change in recidivism rate among participants (E) 0.2 Sentence range for recidivists (SR) 180 Discount rate (d) 0.03 NPV for work-release program $0The program would still break even if its marginal cost were $91/day,all else remaining equal. Slide 30 • September 13, 2012
  • Break-even analysis Model Input Estimate Hiring two new corrections staff (s) $80,000 Number of eligible participants (n) 1,000 Participation rate (p) 0.9 Program compliance rate (q) 0.8 Program marginal cost (PMC) $75/day Jail marginal cost/day (JMC) $55/day Initial sentence range for eligible individuals (SR) 120 Average program failure point for non-compliers (f) 0.1 Baseline 5-year recidivism rate (BRR) 0.5 Change in recidivism rate among participants (E) 0.0 Sentence range for recidivists (SR) 180 Discount rate (d) 0.03 NPV for work-release program $0 The program would break even if it had no effect on recidivism, all else remaining equal. Slide 31 • September 13, 2012
  • Questions? Use the chat feature to send us your questions. Slide 32 • September 13, 2012
  • Probabilistic Sensitivity Analysis:Monte Carlo Simulations Slide 33 • September 18, 2012
  • Probabilistic sensitivity analysisThis type of analysis changes all model inputsat once. •  The other methods tell us little about the likelihood of various program outcomes. •  Monte Carlo simulation repeatedly draws random values for each model input to create a probability distribution of outcomes. Slide 34 • September 13, 2012
  • MS Excel add-onsfor Monte Carlo simulation Slide 35 • September 13, 2012
  • Step 1: Define assumptions. Slide 36 • September 13, 2012
  • Step 1: Define assumptions. Slide 37 • September 13, 2012
  • Define assumptions for all uncertain inputs. Slide 38 • September 13, 2012
  • Step 2: Set the forecast cell. Slide 39 • September 13, 2012
  • MC simulations provide the distributionof outcomes. Slide 49 • September 18, 2012
  • Distribution of outcomes Slide 50 • September 18, 2012
  • Questions? Use the chat feature to send us your questions. Slide 42 • September 13, 2012
  • Wrap-Up Slide 36 • September 18, 2012
  • ReviewWe covered: § Why sensitivity analysis is a critical part of CBA § Different techniques for conducting sensitivity analysis § How to interpret the sensitivity analysis results Slide 44 • September 13, 2012
  • Takeaway points § Base-case CBA is just a starting point. § Sensitivity analysis is a way of managing uncertainty, not eliminating it. § Computation and simulation are relatively easy. § Defining input boundaries and probability distributions is harder. Slide 45 • September 13, 2012
  • Reminders§  Please complete the evaluation form before you leave this webinar.§  To receive information and notifications about upcoming webinars and other events: •  Visit the Cost-Benefit Knowledge Bank for Criminal Justice at cbkb.org. •  Follow us on Twitter at twitter.com/CBKBank. Slide 46 • September 13, 2012
  • Contact information Ben Bryant bryantbp@mcc.gov Carl Matthies cmatthies@vera.org 213-223-2445 E-mail: cbkb@cbkb.org Website: cbkb.org Slide 47 • September 13, 2012
  • The Cost-Benefit Knowledge Bank for Criminal Justice(CBKB) is a project of the Vera Institute of Justicefunded by the U.S. Department of Justice’s Bureau ofJustice Assistance. •  Website (cbkb.org) •  CBA toolkit •  Snapshots of CBA literature •  Podcasts, videocasts, and webinars •  Roundtable discussions •  Community of practice Slide 41
  • This project is supported by Grant No. 2009-MU-BX K029 awarded by theBureau of Justice Assistance. The Bureau of Justice Assistance is acomponent of the Office of Justice Programs, which also includes theBureau of Justice Statistics, the National Institute of Justice, the Office ofJuvenile Justice and Delinquency Prevention, and the Office of SexOffender Sentencing, Monitoring, Apprehending, Registering, andTracking. Points of view or opinions in this presentation are those of theauthors and do not represent the official position or policies of theMillennium Challenge Corporation or the United States Department ofJustice. Slide 42
  • Thank you. Slide 43 • September 18, 2012