Lessons Learned from OVC Evaluations for Future Public Health Evaluations

Lessons Learned from OVC Evaluations for Future Public Health Evaluations Siân Curtis, PhD OVC Evaluation Dissemination Meeting, September 3 rd , 2009, Washington, DC

Growing Emphasis on Evaluation
IOM PEPFAR evaluation report and reauthorization legislation
Global Fund 5 year impact evaluation and OR initiative
CFGD “When will we ever learn” report
3IE Initiative
IHP M&E Working Group – common evaluation framework initiative
USAID evaluation revitalization efforts

Ideal Impact Assessment

Need to think about evaluation at the beginning not at the end, but it is hard to attract attention at that point
Timing – projects are already underway and it is hard to incorporate a strong evaluation design
Scale – many projects are too small to expect to be able to demonstrate impact
Pressure for rapid results to inform programs now
Expectations of multiple stakeholders – scope, competing objectives, multiple/unclear research questions
Political will – need someone in a position of authority to buy in and advocate for evaluation
Challenges to Implementing Rigorous Impact Evaluation

Methodological Constraints to Rigorous Impact Evaluation
Non-random placement of programs - intervention areas and control areas often not comparable
Suitable control areas may not exist – other programs in control areas or cross-over of interventions to control areas
Need/ability to control for other factors beyond the program that might affect outcomes
(Victora, Black, and Bryce 2009)

OVC Evaluation Experience
Timing
programs already underway – no baseline; post-test only design;
Length and intensity of exposure - short duration of exposure (i.e. less 2 years) but impact likely to be longer term
Scale
coverage low in intervention areas – quality of beneficiary lists
some programs small

Pressure for rapid results
post-test only design; short program exposure.
Multiple stakeholder
Supports data use;
Managing expectations regarding scope and coverage of study
Political will/leadership – needs to be strong to facilitate buy-in from all stakeholders

Program participation was non-random
purposive selection of intervention areas
self-selection into (some) programs
controls different from beneficiaries.
Control areas – contamination between program and control areas i.e. some children in control areas reported receiving interventions

Additional Issues for OVC Evaluation
Multiple outcome domains – what to focus on?
Measurement tools for outcome domains vary in how widely tested they are and how well they work
Measurement and comparison of cost-effectiveness across multiple domains new
Lack of standardized interventions/variable intensity and quality
Wide variation in combination of interventions offered and way the programs are implemented

Data Use
Critical to think about data use throughout the evaluation process, not just at the end
Engagement of stakeholders critical to understanding the evaluation questions from different perspectives and creating ownership and demand
Proactive and explicit data use activities will help stakeholders understand and apply findings - recommendations from them better than from research team

Conclusions
Continuing challenge to develop pragmatic evaluation designs that meet rigorous scientific standards within field realities – ongoing area of research
Recognize the long term benefits of evaluations for future programs – “public good”
Takes time for programs to scale-up and to have an effect – evaluations need to be ongoing.
More work needed to test measures of OVC outcomes – often multi-dimensional
Attention to data use (both short and long term) throughout process needed

MEASURE Evaluation is funded by the U.S. Agency for International Development through Cooperative Agreement GHA-A-00-08-00003-00 and is implemented by the Carolina Population Center at the University of North Carolina at Chapel Hill, in partnership with Futures Group International, ICF Macro, John Snow, Inc., Management Sciences for Health, and Tulane University. The views expressed in this presentation do not necessarily reflect the views of USAID or the United States government. Visit us online at http://www.cpc.unc.edu/measure.

Lessons Learned from OVC Evaluations for Future Public Health Evaluations

by MEASURE Evaluation on Oct 13, 2009

Accessibility

Categories

Tags

Upload Details

Usage Rights

Statistics

Lessons Learned from OVC Evaluations for Future Public Health Evaluations — Presentation Transcript