Lessons Learned from OVC Evaluations for Future Public Health Evaluations


Published on

Published in: Education, Technology
1 Like
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • .
  • Lessons Learned from OVC Evaluations for Future Public Health Evaluations

    1. 1. Lessons Learned from OVC Evaluations for Future Public Health Evaluations Siân Curtis, PhD OVC Evaluation Dissemination Meeting, September 3 rd , 2009, Washington, DC
    2. 2. Growing Emphasis on Evaluation <ul><li>IOM PEPFAR evaluation report and reauthorization legislation </li></ul><ul><li>Global Fund 5 year impact evaluation and OR initiative </li></ul><ul><li>CFGD “When will we ever learn” report </li></ul><ul><li>3IE Initiative </li></ul><ul><li>IHP M&E Working Group – common evaluation framework initiative </li></ul><ul><li>USAID evaluation revitalization efforts </li></ul>
    3. 3. Ideal Impact Assessment
    4. 4. <ul><li>Need to think about evaluation at the beginning not at the end, but it is hard to attract attention at that point </li></ul><ul><li>Timing – projects are already underway and it is hard to incorporate a strong evaluation design </li></ul><ul><li>Scale – many projects are too small to expect to be able to demonstrate impact </li></ul><ul><li>Pressure for rapid results to inform programs now </li></ul><ul><li>Expectations of multiple stakeholders – scope, competing objectives, multiple/unclear research questions </li></ul><ul><li>Political will – need someone in a position of authority to buy in and advocate for evaluation </li></ul>Challenges to Implementing Rigorous Impact Evaluation
    5. 5. Methodological Constraints to Rigorous Impact Evaluation <ul><li>Non-random placement of programs - intervention areas and control areas often not comparable </li></ul><ul><li>Suitable control areas may not exist – other programs in control areas or cross-over of interventions to control areas </li></ul><ul><li>Need/ability to control for other factors beyond the program that might affect outcomes </li></ul><ul><li>(Victora, Black, and Bryce 2009) </li></ul>
    6. 6. OVC Evaluation Experience <ul><li>Timing </li></ul><ul><ul><li>programs already underway – no baseline; post-test only design; </li></ul></ul><ul><ul><li>Length and intensity of exposure - short duration of exposure (i.e. less 2 years) but impact likely to be longer term </li></ul></ul><ul><li>Scale </li></ul><ul><ul><li>coverage low in intervention areas – quality of beneficiary lists </li></ul></ul><ul><ul><li>some programs small </li></ul></ul>
    7. 7. OVC Evaluation Experience <ul><li>Pressure for rapid results </li></ul><ul><ul><li>post-test only design; short program exposure. </li></ul></ul><ul><li>Multiple stakeholder </li></ul><ul><ul><li>Supports data use; </li></ul></ul><ul><ul><li>Managing expectations regarding scope and coverage of study </li></ul></ul><ul><li>Political will/leadership – needs to be strong to facilitate buy-in from all stakeholders </li></ul>
    8. 8. OVC Evaluation Experience <ul><li>Program participation was non-random </li></ul><ul><ul><li>purposive selection of intervention areas </li></ul></ul><ul><ul><li>self-selection into (some) programs </li></ul></ul><ul><ul><li>controls different from beneficiaries. </li></ul></ul><ul><li>Control areas – contamination between program and control areas i.e. some children in control areas reported receiving interventions </li></ul>
    9. 9. Additional Issues for OVC Evaluation <ul><li>Multiple outcome domains – what to focus on? </li></ul><ul><li>Measurement tools for outcome domains vary in how widely tested they are and how well they work </li></ul><ul><li>Measurement and comparison of cost-effectiveness across multiple domains new </li></ul><ul><li>Lack of standardized interventions/variable intensity and quality </li></ul><ul><ul><li>Wide variation in combination of interventions offered and way the programs are implemented </li></ul></ul>
    10. 10. Data Use <ul><li>Critical to think about data use throughout the evaluation process, not just at the end </li></ul><ul><li>Engagement of stakeholders critical to understanding the evaluation questions from different perspectives and creating ownership and demand </li></ul><ul><li>Proactive and explicit data use activities will help stakeholders understand and apply findings - recommendations from them better than from research team </li></ul>
    11. 11. Conclusions <ul><li>Continuing challenge to develop pragmatic evaluation designs that meet rigorous scientific standards within field realities – ongoing area of research </li></ul><ul><li>Recognize the long term benefits of evaluations for future programs – “public good” </li></ul><ul><li>Takes time for programs to scale-up and to have an effect – evaluations need to be ongoing. </li></ul><ul><li>More work needed to test measures of OVC outcomes – often multi-dimensional </li></ul><ul><li>Attention to data use (both short and long term) throughout process needed </li></ul>
    12. 12. MEASURE Evaluation is funded by the U.S. Agency for International Development through Cooperative Agreement GHA-A-00-08-00003-00 and is implemented by the Carolina Population Center at the University of North Carolina at Chapel Hill, in partnership with Futures Group International, ICF Macro, John Snow, Inc., Management Sciences for Health, and Tulane University. The views expressed in this presentation do not necessarily reflect the views of USAID or the United States government. Visit us online at http://www.cpc.unc.edu/measure.