Evaluation: Lessons Learned for the Global Health Initiative Siân Curtis, PhD Project Director, MEASURE Evaluation Carolin...
Overview <ul><li>Background on evaluation debates and challenges </li></ul><ul><li>Examples of different designs in practi...
Growing Emphasis on Evaluation  <ul><li>Obama presidential policy directive on development </li></ul><ul><li>Evaluation is...
Scientific Debate <ul><li>Lancet evaluation series  : Bryce et al. 2010; Victora et al. 2010. </li></ul><ul><li>Econometri...
Ideal Impact Assessment
 
Methodological Constraints to Rigorous Impact Evaluation <ul><li>Pure comparison areas may not exist – other programs in c...
<ul><li>Need to think about evaluation at the beginning not at the end, but it is hard to attract attention at that point ...
Example 1: Evaluation of PEPFAR OVC Programs in Kenya and Tanzania
Background <ul><li>Evaluation of five OVC programs in Kenya and Tanzania that used different program models </li></ul><ul>...
Children’s Psychosocial Well-being Three programs offered Home Visiting & two offered Guardian Support Groups Child   Outc...
Prevalence of Home Visiting
Limitations <ul><li>No baseline so cannot assess change since the program began </li></ul><ul><li>Essentially assume inter...
Advantages <ul><li>Yielded some immediate data to inform current programs </li></ul><ul><ul><li>Process information on pro...
Example 2: COMPASS Project  Evaluation, Nigeria
Background <ul><li>Integrated RH/child survival/primary education project targeted to 51 LGAs in five states </li></ul><ul...
Selected FP Indicators
Strengths and Limitations <ul><li>Don’t know what the trend would have been without program – assume trend is associated w...
Example 3: NSDP Evaluation, Bangladesh
Background <ul><li>Urban and rural programs to increase use of FP and MCH services in the poorest communities in Banglades...
Antenatal Care from a Trained Provider,  Births in last year, Rural Areas Rural NSDP Rural Non-NSDP 56 48
Iron Supplementation Rural  NSDP and non-NSDP Remember: ANC (last 1 year) was 56% in 2005 so, Iron<ANC 6.9 Rural NSDP Rura...
Strengths and Limitations <ul><li>Adds another piece of the puzzle – have change over time and change compared to a non-pr...
Concluding Comments
The Evaluation Tension <ul><li>Scientific Purity </li></ul><ul><li>Reductionist </li></ul><ul><li>Refined </li></ul><ul><l...
Some Lessons <ul><li>Need to be clear about what the program is you are evaluating </li></ul><ul><ul><li>More attention to...
M&E Staircase: HIV/AIDS Source: Adapted from Rugg et al. 2004
Source: Bryce, 2010. Step Wise Approaches to Evaluation: Child Survival
Data Use <ul><li>Critical to think about data use throughout the evaluation process, not just at the end </li></ul><ul><li...
Concluding Messages <ul><li>Methods are not a substitute for thinking </li></ul><ul><li>Put pieces together </li></ul><ul>...
Closing Thoughts <ul><li>“ It is the combination of mechanism and context that generates outcomes and without understandin...
MEASURE Evaluation is funded by the U.S. Agency for  International Development through Cooperative Agreement  GHA-A-00-08-...
Upcoming SlideShare
Loading in …5
×

Evaluation: Lessons Learned for the Global Health Initiative

974 views

Published on

Presented at a Duke Population Research Institute seminar series lecture

Published in: Health & Medicine, Technology
1 Comment
0 Likes
Statistics
Notes
  • This is good . How can a get a soft copy for reference?
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Be the first to like this

No Downloads
Views
Total views
974
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
0
Comments
1
Likes
0
Embeds 0
No embeds

No notes for slide
  • .
  • Household surveys targeted children age 8-14 or 7-15. Multiple questionnaires; 1) household 2) caregiver 3) caregiver questionnaire on child 4) child questionnaire. Sample sizes range from 488 caregivers in the TSA program in Tanzania to 3,423 children in the IAP program in Kenya. Details of samples vary across programs but generally intervention group consists of children (and their households and caregivers) enrolled in the program of interest and the comparison group consists of children from the same (or adjacent) community, often newly identified as OVC/MVC.
  • Effects not consistent across programs – only one outcome affected by two programs No more than two outcomes per program Check details of models here: are they multivariate models? Do they only include children who reported received the service as opposed to all children identified as intervention children in the analysis? Our friend Home visiting evaluated across 3 programs Allamano, CRS &amp; TSA PSS Outcomes: Only Allamano’s home visits associated with higher levels of child-reported adult support, lower isolation among children Only TSA’s home visits associated with higher levels child self-esteem CRS’ home visits fewer behavioral problems among beneficiaries CRS and Allamano - Lower isolation among children Overall, no more than two per program and one only one…. Anyone explain this-- Different training curriculums &amp; few psychosocial skills Only one outcome– social isolation– in two programs; encourage children to play; a special visitor for them SG– only 1, CRS Support groups– not targeting children, but trickling down to help them promote better behaviors among their children. Also, these were guardians reported measures, so could also be affecting their own perceptions and acceptance of child behaviors
  • Only 49%, 29%, and 57% of the intervention group of CRS, TSA, and Allamano, respectively, reported having a home visitor. These findings were unexpected as programmers assumed that most, if not all, would report receiving this core service.
  • Would you rather drive completely blind or in a heavy fog? What do you need to make decisions?
  • Peter: Is this among women who got ANC?
  • Evaluation: Lessons Learned for the Global Health Initiative

    1. 1. Evaluation: Lessons Learned for the Global Health Initiative Siân Curtis, PhD Project Director, MEASURE Evaluation Carolina Population Center September 27th, 2010, Washington, DC
    2. 2. Overview <ul><li>Background on evaluation debates and challenges </li></ul><ul><li>Examples of different designs in practice </li></ul><ul><ul><li>Post-test only design – evaluation of OVC programs in Kenya and Tanzania </li></ul></ul><ul><ul><li>Pre-post design – COMPASS evaluation in Nigeria </li></ul></ul><ul><ul><li>Pre-post with comparison group design – NSDP evaluation in Bangladesh </li></ul></ul><ul><li>Concluding comments </li></ul>
    3. 3. Growing Emphasis on Evaluation <ul><li>Obama presidential policy directive on development </li></ul><ul><li>Evaluation is in the Global Health Initiative (GHI) core principles </li></ul><ul><li>USAID evaluation revitalization efforts </li></ul><ul><li>IOM PEPFAR evaluations </li></ul><ul><li>PMI evaluation </li></ul><ul><li>Global Fund 5 year impact evaluation and OR initiative </li></ul><ul><li>IHP M&E Working Group – common evaluation framework initiative </li></ul><ul><li>3IE Initiative </li></ul><ul><li>CFGD “When will we ever learn” report </li></ul>
    4. 4. Scientific Debate <ul><li>Lancet evaluation series : Bryce et al. 2010; Victora et al. 2010. </li></ul><ul><li>Econometrics literature : Deaton, 2009; Imbens, 2009; Heckman & Smith, 1995. </li></ul><ul><li>HIV Prevention : UNAIDS Prevention evaluation think tank, 2008; UNAIDS Strategic guidance for evaluating HIV prevention programs, 2010; Padian et al. 2010. </li></ul><ul><li>Malaria : Rowe et al. 2007 </li></ul><ul><li>FP/RH : Angeles, Guilkey and Mroz, 1998; Chen and Guilkey, 2002; Angeles, Guilkey and Mroz, 2005 </li></ul>
    5. 5. Ideal Impact Assessment
    6. 7. Methodological Constraints to Rigorous Impact Evaluation <ul><li>Pure comparison areas may not exist – other programs in comparison areas or cross-over of interventions to comparison areas </li></ul><ul><li>Non-random placement of programs – intervention areas and control areas often not comparable </li></ul><ul><li>Need/ability to control for other factors beyond the program that might affect outcomes </li></ul><ul><li>Different statistical estimates affected by validity of the assumptions made </li></ul><ul><li>Source: Map taken from Victora et al. 2010. </li></ul>
    7. 8. <ul><li>Need to think about evaluation at the beginning not at the end, but it is hard to attract attention at that point </li></ul><ul><li>Timing – projects are already underway and it is hard to incorporate a strong evaluation design </li></ul><ul><li>Scale/intensity – many projects are too small to expect to be able to demonstrate population impact </li></ul><ul><li>Pressure for rapid results to inform programs now </li></ul><ul><li>Expectations of multiple stakeholders – scope, competing objectives, multiple/unclear research questions </li></ul><ul><li>Political will – need someone in a position of authority to buy in and advocate for evaluation </li></ul>Practical Constraints to Rigorous Impact Evaluation
    8. 9. Example 1: Evaluation of PEPFAR OVC Programs in Kenya and Tanzania
    9. 10. Background <ul><li>Evaluation of five OVC programs in Kenya and Tanzania that used different program models </li></ul><ul><li>Multiple data collection approaches </li></ul><ul><ul><li>Case studies to document programs </li></ul></ul><ul><ul><li>Program expenditures to document different costs </li></ul></ul><ul><ul><li>Outcome evaluation – household survey </li></ul></ul><ul><ul><ul><li>Post-test only design with intervention and comparison groups </li></ul></ul></ul>
    10. 11. Children’s Psychosocial Well-being Three programs offered Home Visiting & two offered Guardian Support Groups Child Outcomes Pro-social behavior Total difficulties Adult support Global self-esteem Social isolation Support Group Effect Effect No Effect No Effect No Effect Home Visiting No Effect Effect Effect Effect Effect
    11. 12. Prevalence of Home Visiting
    12. 13. Limitations <ul><li>No baseline so cannot assess change since the program began </li></ul><ul><li>Essentially assume intervention and comparison groups similar before the program </li></ul><ul><ul><li>Self-selection into intervention (OVC) or recall of intervention; selective targeting of programs. </li></ul></ul><ul><li>Assume differences are due to program effects; no programs in comparison groups </li></ul><ul><ul><li>Comparison groups report exposure to interventions </li></ul></ul>
    13. 14. Advantages <ul><li>Yielded some immediate data to inform current programs </li></ul><ul><ul><li>Process information on program implementation </li></ul></ul><ul><li>Potentially provide a baseline for future evaluation studies (OVC) </li></ul><ul><li>Alternative was no evaluation – is the information obtained better than no information? Is it worth the cost? </li></ul><ul><ul><li>Is it better to drive blind than to drive in a dense fog? </li></ul></ul>
    14. 15. Example 2: COMPASS Project Evaluation, Nigeria
    15. 16. Background <ul><li>Integrated RH/child survival/primary education project targeted to 51 LGAs in five states </li></ul><ul><li>Evaluation Design: Repeated cross-sectional surveys in selected intervention LGAs (no comparison areas) </li></ul><ul><ul><li>Household, facility and school surveys </li></ul></ul><ul><ul><li>2005, 2007, 2009 </li></ul></ul><ul><li>Facility and school surveys are linked to the household survey – facilities serving households interviewed </li></ul>
    16. 17. Selected FP Indicators
    17. 18. Strengths and Limitations <ul><li>Don’t know what the trend would have been without program – assume trend is associated with the program </li></ul><ul><li>Can say whether targets for selected outcome and intermediate outcome indicators were met (outcome monitoring) </li></ul><ul><li>Can start to put together some different pieces of the puzzle – trends in outcomes and intermediate outcomes </li></ul><ul><li>More sophisticated analyses potentially possible linking household and facility data. </li></ul>
    18. 19. Example 3: NSDP Evaluation, Bangladesh
    19. 20. Background <ul><li>Urban and rural programs to increase use of FP and MCH services in the poorest communities in Bangladesh </li></ul><ul><li>Evaluation Design – repeated household surveys in project and non-project areas </li></ul><ul><ul><li>Surveys in 2001, 2003, 2005 </li></ul></ul><ul><ul><li>Aimed to use the same sample clusters 2001-2005 </li></ul></ul><ul><ul><li>Comparison areas adjacent to project areas </li></ul></ul><ul><ul><li>Short community and facility questionnaires included </li></ul></ul><ul><ul><li>Total HH samples approx. 10,000-12,000 </li></ul></ul>
    20. 21. Antenatal Care from a Trained Provider, Births in last year, Rural Areas Rural NSDP Rural Non-NSDP 56 48
    21. 22. Iron Supplementation Rural NSDP and non-NSDP Remember: ANC (last 1 year) was 56% in 2005 so, Iron<ANC 6.9 Rural NSDP Rural Non-NSDP 2.6 2.8 -1.4
    22. 23. Strengths and Limitations <ul><li>Adds another piece of the puzzle – have change over time and change compared to a non-program area </li></ul><ul><li>Parallel trend assumption – trend in comparison area represents what trend would have been in the program area without the program </li></ul><ul><li>Assumes no additional programs in comparison area or other changes that affect outcomes </li></ul><ul><li>Limited information on program pathway </li></ul><ul><li>Difference-in-difference models did not behave well. </li></ul>
    23. 24. Concluding Comments
    24. 25. The Evaluation Tension <ul><li>Scientific Purity </li></ul><ul><li>Reductionist </li></ul><ul><li>Refined </li></ul><ul><li>Practice </li></ul><ul><li>Holistic </li></ul><ul><li>Messy </li></ul>
    25. 26. Some Lessons <ul><li>Need to be clear about what the program is you are evaluating </li></ul><ul><ul><li>More attention to Program Impact Pathway / Framework </li></ul></ul><ul><li>Need to determine whether the program was actually implemented </li></ul><ul><ul><li>More attention to good process evaluation – Not the same as reporting lots of indicators </li></ul></ul>
    26. 27. M&E Staircase: HIV/AIDS Source: Adapted from Rugg et al. 2004
    27. 28. Source: Bryce, 2010. Step Wise Approaches to Evaluation: Child Survival
    28. 29. Data Use <ul><li>Critical to think about data use throughout the evaluation process, not just at the end </li></ul><ul><li>Engagement of stakeholders critical to understanding the evaluation questions from different perspectives and creating ownership and demand </li></ul><ul><li>Proactive and explicit data use activities will help stakeholders understand and apply findings – recommendations from them better than from research team </li></ul>
    29. 30. Concluding Messages <ul><li>Methods are not a substitute for thinking </li></ul><ul><li>Put pieces together </li></ul><ul><li>Keep the end use of the information in mind </li></ul>
    30. 31. Closing Thoughts <ul><li>“ It is the combination of mechanism and context that generates outcomes and without understanding that combination, scientific progress is not possible” </li></ul><ul><li>“ In the end there is no substitute for careful evaluation of a chain of evidence by people who have experience and expertise in the field ” </li></ul><ul><li>From Deaton, 2009 </li></ul>
    31. 32. MEASURE Evaluation is funded by the U.S. Agency for International Development through Cooperative Agreement GHA-A-00-08-00003-00 and is implemented by the Carolina Population Center at the University of North Carolina at Chapel Hill, in partnership with Futures Group, ICF Macro, John Snow, Inc., Management Sciences for Health, and Tulane University. The views expressed in this presentation do not necessarily reflect the views of USAID or the United States government. Visit us online at http://www.cpc.unc.edu/measure.

    ×