• Save
Evaluation: Lessons Learned for the Global Health Initiative
Upcoming SlideShare
Loading in...5

Evaluation: Lessons Learned for the Global Health Initiative



Presented at a Duke Population Research Institute seminar series lecture

Presented at a Duke Population Research Institute seminar series lecture



Total Views
Views on SlideShare
Embed Views



0 Embeds 0

No embeds


Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
  • This is good . How can a get a soft copy for reference?
    Are you sure you want to
    Your message goes here
Post Comment
Edit your comment
  • .
  • Household surveys targeted children age 8-14 or 7-15. Multiple questionnaires; 1) household 2) caregiver 3) caregiver questionnaire on child 4) child questionnaire. Sample sizes range from 488 caregivers in the TSA program in Tanzania to 3,423 children in the IAP program in Kenya. Details of samples vary across programs but generally intervention group consists of children (and their households and caregivers) enrolled in the program of interest and the comparison group consists of children from the same (or adjacent) community, often newly identified as OVC/MVC.
  • Effects not consistent across programs – only one outcome affected by two programs No more than two outcomes per program Check details of models here: are they multivariate models? Do they only include children who reported received the service as opposed to all children identified as intervention children in the analysis? Our friend Home visiting evaluated across 3 programs Allamano, CRS & TSA PSS Outcomes: Only Allamano’s home visits associated with higher levels of child-reported adult support, lower isolation among children Only TSA’s home visits associated with higher levels child self-esteem CRS’ home visits fewer behavioral problems among beneficiaries CRS and Allamano - Lower isolation among children Overall, no more than two per program and one only one…. Anyone explain this-- Different training curriculums & few psychosocial skills Only one outcome– social isolation– in two programs; encourage children to play; a special visitor for them SG– only 1, CRS Support groups– not targeting children, but trickling down to help them promote better behaviors among their children. Also, these were guardians reported measures, so could also be affecting their own perceptions and acceptance of child behaviors
  • Only 49%, 29%, and 57% of the intervention group of CRS, TSA, and Allamano, respectively, reported having a home visitor. These findings were unexpected as programmers assumed that most, if not all, would report receiving this core service.
  • Would you rather drive completely blind or in a heavy fog? What do you need to make decisions?
  • Peter: Is this among women who got ANC?

Evaluation: Lessons Learned for the Global Health Initiative Evaluation: Lessons Learned for the Global Health Initiative Presentation Transcript

  • Evaluation: Lessons Learned for the Global Health Initiative Siân Curtis, PhD Project Director, MEASURE Evaluation Carolina Population Center September 27th, 2010, Washington, DC
  • Overview
    • Background on evaluation debates and challenges
    • Examples of different designs in practice
      • Post-test only design – evaluation of OVC programs in Kenya and Tanzania
      • Pre-post design – COMPASS evaluation in Nigeria
      • Pre-post with comparison group design – NSDP evaluation in Bangladesh
    • Concluding comments
  • Growing Emphasis on Evaluation
    • Obama presidential policy directive on development
    • Evaluation is in the Global Health Initiative (GHI) core principles
    • USAID evaluation revitalization efforts
    • IOM PEPFAR evaluations
    • PMI evaluation
    • Global Fund 5 year impact evaluation and OR initiative
    • IHP M&E Working Group – common evaluation framework initiative
    • 3IE Initiative
    • CFGD “When will we ever learn” report
  • Scientific Debate
    • Lancet evaluation series : Bryce et al. 2010; Victora et al. 2010.
    • Econometrics literature : Deaton, 2009; Imbens, 2009; Heckman & Smith, 1995.
    • HIV Prevention : UNAIDS Prevention evaluation think tank, 2008; UNAIDS Strategic guidance for evaluating HIV prevention programs, 2010; Padian et al. 2010.
    • Malaria : Rowe et al. 2007
    • FP/RH : Angeles, Guilkey and Mroz, 1998; Chen and Guilkey, 2002; Angeles, Guilkey and Mroz, 2005
  • Ideal Impact Assessment
  • Methodological Constraints to Rigorous Impact Evaluation
    • Pure comparison areas may not exist – other programs in comparison areas or cross-over of interventions to comparison areas
    • Non-random placement of programs – intervention areas and control areas often not comparable
    • Need/ability to control for other factors beyond the program that might affect outcomes
    • Different statistical estimates affected by validity of the assumptions made
    • Source: Map taken from Victora et al. 2010.
    • Need to think about evaluation at the beginning not at the end, but it is hard to attract attention at that point
    • Timing – projects are already underway and it is hard to incorporate a strong evaluation design
    • Scale/intensity – many projects are too small to expect to be able to demonstrate population impact
    • Pressure for rapid results to inform programs now
    • Expectations of multiple stakeholders – scope, competing objectives, multiple/unclear research questions
    • Political will – need someone in a position of authority to buy in and advocate for evaluation
    Practical Constraints to Rigorous Impact Evaluation
  • Example 1: Evaluation of PEPFAR OVC Programs in Kenya and Tanzania
  • Background
    • Evaluation of five OVC programs in Kenya and Tanzania that used different program models
    • Multiple data collection approaches
      • Case studies to document programs
      • Program expenditures to document different costs
      • Outcome evaluation – household survey
        • Post-test only design with intervention and comparison groups
  • Children’s Psychosocial Well-being Three programs offered Home Visiting & two offered Guardian Support Groups Child Outcomes Pro-social behavior Total difficulties Adult support Global self-esteem Social isolation Support Group Effect Effect No Effect No Effect No Effect Home Visiting No Effect Effect Effect Effect Effect
  • Prevalence of Home Visiting
  • Limitations
    • No baseline so cannot assess change since the program began
    • Essentially assume intervention and comparison groups similar before the program
      • Self-selection into intervention (OVC) or recall of intervention; selective targeting of programs.
    • Assume differences are due to program effects; no programs in comparison groups
      • Comparison groups report exposure to interventions
  • Advantages
    • Yielded some immediate data to inform current programs
      • Process information on program implementation
    • Potentially provide a baseline for future evaluation studies (OVC)
    • Alternative was no evaluation – is the information obtained better than no information? Is it worth the cost?
      • Is it better to drive blind than to drive in a dense fog?
  • Example 2: COMPASS Project Evaluation, Nigeria
  • Background
    • Integrated RH/child survival/primary education project targeted to 51 LGAs in five states
    • Evaluation Design: Repeated cross-sectional surveys in selected intervention LGAs (no comparison areas)
      • Household, facility and school surveys
      • 2005, 2007, 2009
    • Facility and school surveys are linked to the household survey – facilities serving households interviewed
  • Selected FP Indicators
  • Strengths and Limitations
    • Don’t know what the trend would have been without program – assume trend is associated with the program
    • Can say whether targets for selected outcome and intermediate outcome indicators were met (outcome monitoring)
    • Can start to put together some different pieces of the puzzle – trends in outcomes and intermediate outcomes
    • More sophisticated analyses potentially possible linking household and facility data.
  • Example 3: NSDP Evaluation, Bangladesh
  • Background
    • Urban and rural programs to increase use of FP and MCH services in the poorest communities in Bangladesh
    • Evaluation Design – repeated household surveys in project and non-project areas
      • Surveys in 2001, 2003, 2005
      • Aimed to use the same sample clusters 2001-2005
      • Comparison areas adjacent to project areas
      • Short community and facility questionnaires included
      • Total HH samples approx. 10,000-12,000
  • Antenatal Care from a Trained Provider, Births in last year, Rural Areas Rural NSDP Rural Non-NSDP 56 48
  • Iron Supplementation Rural NSDP and non-NSDP Remember: ANC (last 1 year) was 56% in 2005 so, Iron<ANC 6.9 Rural NSDP Rural Non-NSDP 2.6 2.8 -1.4
  • Strengths and Limitations
    • Adds another piece of the puzzle – have change over time and change compared to a non-program area
    • Parallel trend assumption – trend in comparison area represents what trend would have been in the program area without the program
    • Assumes no additional programs in comparison area or other changes that affect outcomes
    • Limited information on program pathway
    • Difference-in-difference models did not behave well.
  • Concluding Comments
  • The Evaluation Tension
    • Scientific Purity
    • Reductionist
    • Refined
    • Practice
    • Holistic
    • Messy
  • Some Lessons
    • Need to be clear about what the program is you are evaluating
      • More attention to Program Impact Pathway / Framework
    • Need to determine whether the program was actually implemented
      • More attention to good process evaluation – Not the same as reporting lots of indicators
  • M&E Staircase: HIV/AIDS Source: Adapted from Rugg et al. 2004
  • Source: Bryce, 2010. Step Wise Approaches to Evaluation: Child Survival
  • Data Use
    • Critical to think about data use throughout the evaluation process, not just at the end
    • Engagement of stakeholders critical to understanding the evaluation questions from different perspectives and creating ownership and demand
    • Proactive and explicit data use activities will help stakeholders understand and apply findings – recommendations from them better than from research team
  • Concluding Messages
    • Methods are not a substitute for thinking
    • Put pieces together
    • Keep the end use of the information in mind
  • Closing Thoughts
    • “ It is the combination of mechanism and context that generates outcomes and without understanding that combination, scientific progress is not possible”
    • “ In the end there is no substitute for careful evaluation of a chain of evidence by people who have experience and expertise in the field ”
    • From Deaton, 2009
  • MEASURE Evaluation is funded by the U.S. Agency for International Development through Cooperative Agreement GHA-A-00-08-00003-00 and is implemented by the Carolina Population Center at the University of North Carolina at Chapel Hill, in partnership with Futures Group, ICF Macro, John Snow, Inc., Management Sciences for Health, and Tulane University. The views expressed in this presentation do not necessarily reflect the views of USAID or the United States government. Visit us online at http://www.cpc.unc.edu/measure.