Evaluating Impact of OVC Programs: Standardizing our methods


Published on

Jen Chapman presents on the Orphans and Vulnerable Children Program Evaluation Tool Kit, which supports PEPFAR-funded programs and helps fulfill the aims presented in the USAID Evaluation Policy.

Published in: Health & Medicine
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Evaluating Impact of OVC Programs: Standardizing our methods

  1. 1. Evaluating the impact of OVC programs: standardizing our methods Jenifer Chapman, PhD Senior OVC Advisor MEASURE Evaluation
  2. 2. Overview  Background  Purpose of the OVC Survey Tools  Process of development  Guiding principles  Structure and content  When to use the Tools  When not to use the Tools  Using the data
  3. 3. Tools in a Toolbox  There is no single data collection tool that can meet all OVC program targeting, case management and M&E requirements.  This set of survey tools responds to distinct information needs related to program planning and evaluation, and fills a tools gap.
  4. 4. The problem  High investment in OVC programs BUT impact is unclear & questions regarding “what works” in improving household well-being  Part of the challenge: lack of standardized measures and tools for child and household outcomes (well-being) 4
  5. 5. A proposed solution Standardized questionnaires for use in a survey of children ages 0-17 years and their adult caregivers
  6. 6. The purpose  Standardize population-level child and caregiver well- being data beyond what is available from routine surveys  Produce actionable data to inform programs and enable mid-course corrections  Enable comparative assessments of child and caregiver well-being and household economic status across a diverse set of interventions and regions
  7. 7. Who are these tools for?  Local and international research institutions and other implementing organizations with evaluation agenda
  8. 8. Our Process  Two step, participatory process:  Build consensus around core impact indicators for PEPFAR-funded OVC programs  Develop OVC program evaluation (survey) tools
  9. 9. Distilling the core indicators Step 1: Extensive literature search Step 2: Gaps (HES, PSS) filled through targeted research  Result: >600 child/HH wellbeing questions/indicators Step 3: Analysis against 8 criteria  Result: shorter list of questions for discussion 9
  10. 10. Inclusion criteria 1. Measures impact/outcomes 2. Amenable to change from program interventions 3. Relevant across a wide range of interventions 4. Contributes to a holistic vision of child wellbeing 5. Verifiable through another source 6. Easy to implement 7. Relevant across different regions / countries 8. Relevant or easily adapted across age and sex 10
  11. 11. Finalizing the core indicators  External working group: solicited review from 49 stakeholders  Received feedback from > 25 individuals/groups  Finalized core set of 12 child and 3 household measures 11
  12. 12. From indicators to tools: Guiding principles  Questionnaires measure program outcomes  Program outcome data should be collected by trained data collectors  A documented protocol is required  Protocol with tools needs to undergo ethical approval both in the country of data collection and in the US  Tools require pilot testing in new settings before use
  13. 13. Developing the tools  Tools drafted with strong stakeholder input  Draft tools piloted in Zambia (and Nigeria)  Cognitive interviews to test key concepts (e.g. social support)  Household pre-test of full tools, procedures
  14. 14. Structure and content 1. Caregiver questionnaire (including questions on household) 2. Child questionnaire (ages 0-9 years), administered to caregiver 3. Child questionnaire (ages 10-17), administered to child with parental consent & child assent
  15. 15. Sections Core questions Optional modules Section 1: Household schedule • Household schedule* (10) • Changes in household composition (4) Section 2: Background Information on Caregiver and Household • Demographic information* (7) • Work* (3) • Access to money (3) • Shelter (1) • Household Economic Status (forthcoming) • Progress out of Poverty Index or similar (country specific) Section 3: Food Security • Household food security (6) • Dietary Diversity (1) Section 4: Caregiver Well- being and Attitudes • General health (2) • Caregiver support (4) • Parental self-efficacy (1) • Perceptions and experience of child discipline, violent discipline (forthcoming) • Gender roles and decisionmaking power* (9) Section 5: HIV/AIDS Testing, Knowledge, Attitudes • Basic HIV/AIDS knowledge* (7) • HIV testing* (3) • Attitudes to condom educ (1) • HIV/AIDS attitudes* (4) Section 6: Access to HIV Prevention, Care & Support • Household access to services (1) *DHS, bold=core indicator Caregiver questionnaire
  16. 16. Sections Core questions Optional modules Section 1: Child Health and Welfare • Confirm demographics (5) • General health & disability (4) • Birth certificate (2) • Vaccinations (11) • Fever (<5 years)* (1) • Diarrhea (<5 years)* (1) • Experience of neglect (2) • Slept under mosquito net* (1) • HIV testing experience* (2) • Fever: extended* (4) • Diarrhea: extended* (3) • Health for children living with HIV/AIDS (forthcoming) Section 2: Education and Work • School attendance*, progression/repeats, drop-outs, missed school days (5+ years) (9) • Work for wages (2) • Early childhood stimulation (2) Section 3: Food Consumption • Food consumption (2+ years) (8) • Dietary diversity (1) Section 4: Access to HIV Prevention, Care & Support • Child access to services (1) Section 5: Anthropometric Measures (of Children) • Weight*, Height*, MUAC Child questionnaire (ages 0-9) *DHS, bold=core indicator
  17. 17. Child questionnaire (ages 10-17) Sections Core questions Optional modules Section 1: Background Information on Child • Confirm demographics* (5) • Identity of caregiver (1) Section 2: Diary • Daily log (6) Section 3: Education • School attendance*, progression/repeats, drop- outs (9) Section 4: Chores & Work • Chores (3) • Work (7) Section 5: Food & Alcohol Consumption • Food consumption (8) • Alcohol consumption (3) • Dietary diversity (1) Section 6: Health, Support & Protection • Birth certificate (2) • General health & disability (3) • General support (4) • Health for children living with HIV/AIDS (forthcoming) • Perceptions/experience of violence (forthcoming) Section 7: HIV Testing, Knowledge, and Attitudes • Basic HIV/AIDS knowledge* (7) • HIV testing * (3) • Child development knowledge (6) • HIV/AIDS attitudes and beliefs (4) Section 8: Sexual Experience • Sexual behavior (13-17 yrs) (5) Section 9: Access to HIV Prevention, Care & Support • Child access to services (1) Section 10: Anthropometric Measures: Weight and Height • Weight, Height, MUAC
  18. 18. When are these the right tools? Tools are useful if your question is: 1.Is my program having, or did my program have an impact on the children and households it reached? 2.What are the characteristics of children and their caregivers in my country, state/province or district/area, in terms of education, health, protection, and psychosocial support? 3.Where do the children most in need of program support live? 4.Approximately how many children need services or support? 5.What are the needs of my program’s registered beneficiaries, in terms of education, health, protection, and psychosocial support?
  19. 19. These are not the right tools for you if… You want to know: Which children in selected communities to target How a particular child/household is faring Which households, children or caregivers are worst off What services to provide or refer for a particular child / household The number of children/households that are receiving program support, and the types of support received Whether staff are carrying out their responsibilities Whether interventions are being implemented as planned
  20. 20. And, why a special OVC survey?  DHS and MICS take a general population sample  difficult to discern the program’s contribution  DHS and MICS include some, but not all of the OVC core indicators
  21. 21. Using the Data  Representative sample of program beneficiaries  Data collected at one point in time  Data collected at two points in time  Representative sample of the general population
  22. 22. Beneficiary sample: 1 point in time  Often called: Baseline, Midline or Endline  If baseline or midline: Use data for program planning or design, or mid-course corrections  Example: high food insecurity found at baseline  Result: Change in workplan, PMP agreed between partner and USG, emphasizing food security  If endline: Use data to inform follow-on activities
  23. 23. Beneficiary sample: 2 (or more) points in time  Commonly referred to as an “evaluation”  Baseline data should be used immediately  Evaluation results inform future programming, policy  But, change in wellbeing from time 1 to time 2, does not mean program is 100% responsible  Much stronger result if:  Comparison group is added (counterfactual)  Panel study / cohort
  24. 24. General population sample  Commonly called a Situation Analysis  Use data for needs-based resource allocation at national or sub-national level (not individual or community level)  Example:  Nigeria OVC Situation Analysis
  25. 25. Triangulate  Analyze data alongside DHS and MICS data  Check for differences and similarities  Determine whether differences or similarities make sense, or point to a data quality issue
  26. 26. Implementation so far  Zambia: Impact evaluation of savings and internal lending communities on OVC wellbeing  Baseline data available early Fall  Nigeria: Baseline survey of OVC umbrella grant mechanism beneficiaries in 10 states (planning phase)
  27. 27. You said a toolkit?  Tools & Manual  Data analysis guide  Template protocol with consent/assent forms  Data collector training materials  And then what?  Revising as we learn  Supporting countries to implement
  28. 28. Where can I find out more? Go to our website: http://www.cpc.unc.edu/measure/our-work/ovc Keep in touch on Child Status Net: http://childstatus.net/ Email: Jenifer Chapman: jchapman@futuresgroup.com and Janet Shriberg: jshriberg@usaid.gov
  29. 29. The research presented here has been supported by the President’s Emergency Plan for AIDS Relief (PEPFAR) through the United States Agency for International Development (USAID) under the terms of MEASURE Evaluation cooperative agreement GHA-A-00-08-00003- 00. Views expressed are not necessarily those of PEPFAR, USAID or the United States government. MEASURE Evaluation is implemented by the Carolina Population Center at the University of North Carolina at Chapel Hill in partnership with Futures Group, ICF International, John Snow, Inc., Management Sciences for Health, and Tulane University.
  30. 30. Questions?