Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
$
Manage Successful
Impact Evaluations
FIELD COORDINATOR
WORKSHOP
6-10 JUNE 2016
WASHINGTON, DC
How Much Does It Cost To Get That Impact?
Measuring Cost Effectiveness
8 June 2016
David K. Evans, Senior Economist, Offic...
The Challenge
Increasing Evidence of Impact
Legovinietal.2012
We have all this amazing
evidence of interventions that
will work in your country!
Yes, but how much is this going
to cost...
1. Summarize a complex
program in terms of a
return on investment
2. Permit comparison
across contexts &
times
Combining c...
Intuitive appeal of cost-
effectiveness estimates
JPAL2014
Cost Effectiveness Can Reverse
Ranking of Effective Interventions
What are the most
effective education
programs?
Source: ...
Cost Effectiveness Can Reverse
Ranking of Effective Interventions
What are the most
cost-effective
education programs?
No ...
Cost Effectiveness Can Reverse
Ranking of Effective Interventions
If we want to responsibly use impact evaluation to infor...
Definitions and Debates
Cost benefit
analysis
Cost
effectiveness
analysis
Cost Benefit Analysis
𝑁𝑒𝑑 π‘π‘Ÿπ‘’π‘ π‘’π‘›π‘‘ π‘£π‘Žπ‘™π‘’π‘’ =
𝑑=0
𝑛
𝐡𝑑
(1 + π‘Ÿ) 𝑑
βˆ’
𝑑=0
𝑛
𝐢𝑑
(1 + π‘Ÿ) 𝑑
1. Monetize the
benefits of the
program
...
Cost Effectiveness Analysis
McEwan2012
1. Calculate the
cost
2. Divide it by
incremental
effect
𝐢
𝐸
= π‘π‘œπ‘ π‘‘ π‘π‘’π‘Ÿ π‘Žπ‘‘π‘‘π‘–π‘‘π‘–π‘œπ‘›π‘Žπ‘™ ...
CBA
 Absolute desirability of
intervention
 Allows comparison across
totally different
interventions (health
versus agri...
Question: The Minister of Finance is deciding whether to scale
up an Agricultural Extension program or a Business Training...
Question: The Minister of Education wants to increase student
test scores. What kind of analysis will be most useful?
1. C...
Whose costs?
Take a social perspective
Seed distribution program
Ministry of Finance
DeliveryCommunity leadersHouseholds
M...
Question: The government is giving grants to entrepreneurs.
Where do these enter the costing equation?
1. As cost to the
g...
Ingredients method
β€’ Personnel
β€’ Facilities
β€’ Equipment
β€’ Materials
β€’ Other program inputs
β€’ Client inputs
Recipe for Cost...
Where do we get the info?
Budget
β€’ What it was supposed to cost
β€’ Not actual cost
β€’ Often misses client costs
Interviews &...
Where do we get the info?
Budget
β€’ What it was supposed to cost
β€’ Not actual cost
β€’ Often misses client costs
Interviews &...
Value each ingredient
Adjust for *
– Inflation ---
– Time-value *
– Currency ---
+ Add it all together
Cost of the program...
Question: My agricultural extension program has
community volunteers who spend 10 hours a week and
receive a bag of rice. ...
JPAL2014
What Cost Effectiveness
Looks Like in Practice
β€’ Cost analysis is extra work
β€’ Most studies don’t do it
– Of 77 RCTs, 56% reported
zero data on incremental
costs (McEwan...
Now that you know how to do it
A few caveats to keep in mind
Caveats
Cost-Effectiveness Sensitivity
to Errors in Estimates
Additional s.d. of student learning per $100
Evans&Popova2014
Gray: ...
Cost-Effectiveness Sensitivity to Place
Evans&Popova2014
Actual
effectiveness-
cost ratio for
India program
Gray: Other
pr...
Pilot vs Scaled up
Evans&Popova2014
Cost Sensitivity to Exchange Rates
Dhaliwaletal.2011
Standard exchange rate Purchasing power parity
Assumes relative price...
Ignoring:
1. Categories of ingredients: facilities, equipment, client
inputs
2. Sensitivity (to place, to scale, etc.)
Com...
1. No policymaker adopts a program without
knowing the cost (we hope!)
2. Costing and cost effectiveness take us from
scho...
β€’ Papers!
– Cost-Effectiveness Analysis of Education & Health Interventions in
Developing Countries, by McEwan (2012)
– Co...
β€’ IDB Evaluation Hub: Cost Benefit and Cost-
Effectiveness
– Background materials
– Terms of reference
– Examples
– Templa...
Credits
β€’ All clip art from openclipart.org
β€’ β€œJust Do It” from Adweek.com
β€’ NGO partners from OneAcreFund.org
β€’ Some othe...
How much does it cost to get that impact? Measuring cost effectiveness
Upcoming SlideShare
Loading in …5
×

How much does it cost to get that impact? Measuring cost effectiveness

512 views

Published on

This presentation, on cost effectiveness and cost benefit analysis for impact evaluations, was delivered at the World Bank DIME Field Coordinator workshop on June 8, 2016.

A range of resources for carrying out cost analysis are included in the final slides.

Published in: Economy & Finance
  • Be the first to comment

How much does it cost to get that impact? Measuring cost effectiveness

  1. 1. $ Manage Successful Impact Evaluations FIELD COORDINATOR WORKSHOP 6-10 JUNE 2016 WASHINGTON, DC
  2. 2. How Much Does It Cost To Get That Impact? Measuring Cost Effectiveness 8 June 2016 David K. Evans, Senior Economist, Office of the Chief Economist for Africa
  3. 3. The Challenge Increasing Evidence of Impact Legovinietal.2012
  4. 4. We have all this amazing evidence of interventions that will work in your country! Yes, but how much is this going to cost me? Not just you, man! How much is it going to cost me? Government Minister Entrepreneur IE Expert #1
  5. 5. 1. Summarize a complex program in terms of a return on investment 2. Permit comparison across contexts & times Combining cost & impact data has two goals Dhaliwaletal.2011
  6. 6. Intuitive appeal of cost- effectiveness estimates JPAL2014
  7. 7. Cost Effectiveness Can Reverse Ranking of Effective Interventions What are the most effective education programs? Source: J-PAL, β€œIncreasing test score performance”
  8. 8. Cost Effectiveness Can Reverse Ranking of Effective Interventions What are the most cost-effective education programs? No overlap between those two groups. Source: J-PAL, β€œIncreasing test score performance”
  9. 9. Cost Effectiveness Can Reverse Ranking of Effective Interventions If we want to responsibly use impact evaluation to inform policy, cost analysis is crucial. Source: J-PAL, β€œIncreasing test score performance”
  10. 10. Definitions and Debates Cost benefit analysis Cost effectiveness analysis
  11. 11. Cost Benefit Analysis 𝑁𝑒𝑑 π‘π‘Ÿπ‘’π‘ π‘’π‘›π‘‘ π‘£π‘Žπ‘™π‘’π‘’ = 𝑑=0 𝑛 𝐡𝑑 (1 + π‘Ÿ) 𝑑 βˆ’ 𝑑=0 𝑛 𝐢𝑑 (1 + π‘Ÿ) 𝑑 1. Monetize the benefits of the program 2. Sum those up over time (for n years) 3. Discount future gains 4. Do the same thing for costs 6. VoilΓ ! A summary measure of the lifetime net value of the intervention! 5. Subtract costs from benefits McEwan2012
  12. 12. Cost Effectiveness Analysis McEwan2012 1. Calculate the cost 2. Divide it by incremental effect 𝐢 𝐸 = π‘π‘œπ‘ π‘‘ π‘π‘’π‘Ÿ π‘Žπ‘‘π‘‘π‘–π‘‘π‘–π‘œπ‘›π‘Žπ‘™ 𝑒𝑛𝑖𝑑 π‘œπ‘“ 𝑏𝑒𝑛𝑒𝑓𝑖𝑑 What the h#@! is an additional unit of benefit? 10% increase in business profits 1 standard deviation increase in crop yield 10% increase in farmer knowledge Additional woman receiving prenatal care Additional student in school 1 s.d. increase in student learning
  13. 13. CBA  Absolute desirability of intervention  Allows comparison across totally different interventions (health versus agriculture)  Requires very strong assumptions (lifetime benefits of intervention…) Pros and Cons CEA  Much more transparent: No need to monetize range of benefits  Simpler to explain  Useful for comparing interventions with common outcome (agricultural extension versus targeted savings)  Implicit assumption: Common post-intervention trajectory McEwan2012
  14. 14. Question: The Minister of Finance is deciding whether to scale up an Agricultural Extension program or a Business Training program. What kind of analysis will be most useful? 1. Cost-Benefit Analysis 2. Cost-Effectiveness Analysis Answer: (1)
  15. 15. Question: The Minister of Education wants to increase student test scores. What kind of analysis will be most useful? 1. Cost-Benefit Analysis 2. Cost-Effectiveness Analysis Answer: (2)
  16. 16. Whose costs? Take a social perspective Seed distribution program Ministry of Finance DeliveryCommunity leadersHouseholds Ministry of Agriculture NGO Partner McEwan2012
  17. 17. Question: The government is giving grants to entrepreneurs. Where do these enter the costing equation? 1. As cost to the government. 2. As benefit to the entrepreneur. 3. Both. Answer: It depends. In CBA, the answer is (3), so it cancels out. In CEA, we don’t include other benefits (e.g., increased revenues, better health), so why this? So (1). McEwan 2012 and Dhaliwal et al 2011
  18. 18. Ingredients method β€’ Personnel β€’ Facilities β€’ Equipment β€’ Materials β€’ Other program inputs β€’ Client inputs Recipe for Cost Analysis Metaphor for successful cost analysis Looks delicious! McEwan2012
  19. 19. Where do we get the info? Budget β€’ What it was supposed to cost β€’ Not actual cost β€’ Often misses client costs Interviews & direct observations β€’ Did it go how it was supposed to go? (No) β€’ How many seeds were actually distributed? How much time did it actually require? Data collected in impact evaluation β€’ Remuneration β€’ Time use McEwan2012 Need early: Guides rest of cost analysis Very difficult to gather ex post Must be designed ex ante
  20. 20. Where do we get the info? Budget β€’ What it was supposed to cost β€’ Not actual cost β€’ Often misses client costs Interviews & direct observations β€’ Did it go how it was supposed to go? (No) β€’ How many seeds were actually distributed? How much time did it actually require? Data collected in impact evaluation β€’ Remuneration β€’ Time use McEwan2012 Need early: Guides rest of cost analysis Very difficult to gather ex post Must be designed ex ante
  21. 21. Value each ingredient Adjust for * – Inflation --- – Time-value * – Currency --- + Add it all together Cost of the program What do we do with the info? McEwan2012
  22. 22. Question: My agricultural extension program has community volunteers who spend 10 hours a week and receive a bag of rice. How do I cost that? 1. Market value of a bag of rice 2. Market value of 10 hours of time for people with similar background 3. Both 4. I don’t know; I was checking my email. Answer: (3). Hopefully not (4). Come on team, heads in the game. Really, it depends. If β€œgovernment only” perspective, could leave out, but volunteers in pilot may not be available at scale. McEwan 2012 and Dhaliwal et al 2011
  23. 23. JPAL2014 What Cost Effectiveness Looks Like in Practice
  24. 24. β€’ Cost analysis is extra work β€’ Most studies don’t do it – Of 77 RCTs, 56% reported zero data on incremental costs (McEwan 2014) β€’ Not doing it significantly limits policy relevance Well, that was easy. Not! That was exhausting!
  25. 25. Now that you know how to do it A few caveats to keep in mind Caveats
  26. 26. Cost-Effectiveness Sensitivity to Errors in Estimates Additional s.d. of student learning per $100 Evans&Popova2014 Gray: Measured cost effectiveness Black: 90% confidence interval on impact estimate Lesson: Be humble about cross program ordering.
  27. 27. Cost-Effectiveness Sensitivity to Place Evans&Popova2014 Actual effectiveness- cost ratio for India program Gray: Other programs Additional s.d. of student learning per $100 Red: Use other country costs of community teachers Lesson: Adapting cost-effectiveness across countries takes work.
  28. 28. Pilot vs Scaled up Evans&Popova2014
  29. 29. Cost Sensitivity to Exchange Rates Dhaliwaletal.2011 Standard exchange rate Purchasing power parity Assumes relative prices in country Adjusts for price levels across countries but messes up relative prices
  30. 30. Ignoring: 1. Categories of ingredients: facilities, equipment, client inputs 2. Sensitivity (to place, to scale, etc.) Common Pitfalls AdaptedfromDeLaBriere 3. Exchange rates 4. Reality (costing ideal vs. actual program). Not providing enough details to verify 1 to 4
  31. 31. 1. No policymaker adopts a program without knowing the cost (we hope!) 2. Costing and cost effectiveness take us from scholarship to practical policy implementation 3. It is just as important to be prospective with costs as with impact evaluation 4. Field coordinators are particularly well placed to gather cost data Remember!
  32. 32. β€’ Papers! – Cost-Effectiveness Analysis of Education & Health Interventions in Developing Countries, by McEwan (2012) – Comparative Cost-Effectiveness Analysis to Inform Policy in Developing Countries, by Dhaliwal et al (2011) – Cost-Effectiveness Measurement in Development: Accounting for Local Costs & Noisy Impacts, by Evans & Popova (2014) β€’ A book! – Cost-Effectiveness Analysis: Methods & Applications, by McEwan & Levin (2000) Additional Resources - 1
  33. 33. β€’ IDB Evaluation Hub: Cost Benefit and Cost- Effectiveness – Background materials – Terms of reference – Examples – Templates β€’ Poverty Action Lab – Examples for student participation and student learning – What does this look like in Excel? Additional Resources - 2
  34. 34. Credits β€’ All clip art from openclipart.org β€’ β€œJust Do It” from Adweek.com β€’ NGO partners from OneAcreFund.org β€’ Some other photos from World Bank Flickr stream Thank you!

Γ—