Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Mixed Methods in Impact Evaluation May 2011


Published on

Howard White's presentation at 3ie-LIDC symposium on impact evaluation methods and policy influence "Thinking out of the black box" in London on May 23.

Published in: Technology, Business
  • Be the first to comment

  • Be the first to like this

Mixed Methods in Impact Evaluation May 2011

  1. 1. Mixed Methods in Impact Evaluation Howard White International Initiative for Impact Evaluation
  2. 2. <ul><li>Measurement is not evaluation </li></ul>
  3. 3. <ul><li>Why did the Bangladesh Integrated Nutrition Program (BINP) fail? </li></ul><ul><li>Why did the Bangladesh Integrated Nutrition Project (BINP) fail? </li></ul>
  4. 4. Comparison of impact estimates
  5. 5. Summary of theory
  6. 6. The theory of change Right target group for nutritional counselling
  7. 7. The theory of change Knowledge acquired and used
  8. 8. The theory of change The right children are enrolled in the programme
  9. 9. The theory of change Supplementary feeding is supplementary
  10. 10. Participation rates
  11. 11. Data used in BINP study <ul><li>Project evaluation data (three rounds) </li></ul><ul><li>Save the Children evaluation </li></ul><ul><li>Helen Keller Nutritional Surveillance Survey </li></ul><ul><li>DHS (one round) </li></ul><ul><li>Project reports </li></ul><ul><li>Anthropological studies of village life </li></ul><ul><li>Action research (focus groups, CNP survey) </li></ul>
  12. 12. Examples of ‘atheoretical’ IEs <ul><li>School capitation grant studies that don’t ask how the money was used </li></ul><ul><li>BCC intervention studies that don’t ask if behaviour has changed (indeed, almost any study that does not capture behavior change) </li></ul><ul><li>Microfinance studies that don’t look at use of funds and cash flow </li></ul><ul><li>Studies of capacity development that don’t ask if knowledge acquired and used </li></ul>
  13. 13. Different parts of causal chain require different analysis <ul><li>Factual versus counterfactual </li></ul><ul><li>Examples of factual </li></ul><ul><ul><li>Use of funds </li></ul></ul><ul><ul><li>Targeting </li></ul></ul><ul><ul><li>Participatory processes </li></ul></ul><ul><li>Quantitative and qualitative and the combination of the two </li></ul>
  14. 14. Examples from AP SHGs Number of SHGs and % penetration Drop outs & corrupt practices The angry man Returns to cows and goats: quantitative ethnography
  15. 15. Why the angry man was angry Loan allocation is to households not individuals
  16. 16. More examples <ul><li>The disconnected in connected villages pretty much everywhere </li></ul><ul><li>The role of the community in social funds in Malawi and Zambia </li></ul>
  17. 17. Poorest don’t connect
  18. 18. People participate in making bricks , n ot decisions
  19. 19. I have to go now. I have a community in my office
  20. 20. <ul><li>Also mix methods for identification strategy </li></ul>
  21. 21. AP village fund allocation Fixed funds per community: more households per SHG Lower membership rates in larger villages
  22. 22. <ul><li>The puzzle of the disconnected households </li></ul>
  23. 23. You can’t carry electricity on boats
  24. 24. Mixing methods <ul><li>Understanding context to </li></ul><ul><ul><li>Shape evaluation questions </li></ul></ul><ul><ul><li>Design data collection </li></ul></ul><ul><li>Mapping out theory(ies) of change </li></ul><ul><li>Addressing factual questions, leading to… </li></ul><ul><li>… interpretation of counterfactual findings </li></ul><ul><li>One example on data collection </li></ul>
  25. 25. <ul><li>General principle: the quality of data deteriorates the more formal the process of data collection </li></ul>
  26. 26. What do questionnaires miss? <ul><li>Consumption </li></ul><ul><ul><li>Festivals </li></ul></ul><ul><ul><li>Labour exchange </li></ul></ul><ul><ul><li>Wildfoods </li></ul></ul><ul><li>Net income from household enterprises </li></ul><ul><li>Abuse of nearly all kinds </li></ul><ul><li>Who is a household member? </li></ul>
  27. 27. Protein in Northern Zambia
  28. 28. Who is in the household roster? This is my sister, Hana. Her mummy is my mummy’s sister
  29. 29. What is to be done? <ul><ul><li>Know what questions to ask and how </li></ul></ul><ul><ul><li>Proxy measures </li></ul></ul><ul><ul><li>Enumerator training </li></ul></ul><ul><ul><li>Contrived informality </li></ul></ul>
  30. 30. The challenge of integration <ul><li>Parallel studies not integrated studies (multi-disciplinary not inter-disciplinary) </li></ul><ul><li>Why? </li></ul><ul><ul><li>At best silo mentality, at worst arrogance (“trust me, I’m an economist”) </li></ul></ul><ul><ul><li>Academic incentives </li></ul></ul><ul><ul><li>People just don’t know how to do it </li></ul></ul><ul><li>What to do? </li></ul><ul><ul><li>Start with theory of change </li></ul></ul><ul><ul><li>Team members who bridge studies </li></ul></ul><ul><ul><li>Detailed team discussions around causal chain </li></ul></ul><ul><ul><li>Quality of external peer review </li></ul></ul>
  31. 31. <ul><li>Thank you </li></ul><ul><li>Visit </li></ul>