Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Collecting the PEPFAR OVC MER Essential Survey Indicators: Frequently Asked Questions

749 views

Published on

Gretchen Bachman and Christine Fu (USAID); Lisa Parker, Jenifer Chapman, Lisa Marie Albert, Walter Obiero, and Susan Settergren from MEASURE Evaluation. January 2017 Webinar.

Published in: Health & Medicine
  • Be the first to comment

Collecting the PEPFAR OVC MER Essential Survey Indicators: Frequently Asked Questions

  1. 1. Monitoring, Evaluation, and Reporting (MER) Orphan and Vulnerable Children Essential Survey Indicators Gretchen Bachman and Christine Fu USAID Lisa Parker, Jenifer Chapman, Lisa Marie Albert, Walter Obiero, and Susan Settergren MEASURE Evaluation/Palladium January 31, 2017 FREQUENTLYASKEDQUESTIONS
  2. 2. Outcome indicators measuring household and child well-being • Designated as “Essential Survey Indicators” (ESI) Purpose: • To obtain snapshot of program outcomes at a point in time AND • To assess changes in outcomes among orphans and vulnerable children (OVC) program beneficiaries over time The purpose
  3. 3. 9 MER Essential Survey Indicators 1. Percent of children whose primary caregiver knows child’s HIV status 2. Percent of children <5 years of age who are undernourished 3. Percent of children too sick to participate in daily activities 4. Percent of children who have a birth certificate 5. Percent of children regularly attending school 6. Percent of children who progressed in school during the last year 7. Percent of children <5 years of age who recently engaged in stimulating activities with any household member over 15 years of age 8. Percent of caregivers who agree that harsh physical punishment is an appropriate means of discipline or control in the home or school 9. Percent of households able to access money to pay for unexpected household expenses
  4. 4. MER implementation to date Data reported into DATIM: • Uganda • Zimbabwe • Swaziland Data collection completed: • Namibia • Nigeria • Kenya • Cote d’Ivoire Upcoming: • Lesotho • Rwanda • Mozambique • Botswana • Tanzania • Haiti • Malawi • DRC
  5. 5. Frequently Asked Questions
  6. 6. How will the data be used? Data will support: • Strategic portfolio development • Targeting • Programming • Implementation • Resource allocations • Provide evidence to the U.S. Congress • Be triangulated with other relevant OVC data
  7. 7. What types of decisions will be made based on MER OVC Essential Survey Indicator data? Example: Education indicators • USG Mission decisions • Implementing partner decisions
  8. 8. Is a MER OVC ESI survey required for all OVC projects? • Requirement for OVC projects with an annual budget of USD $1 million • Countries with total HKID funding of less than USD$1 million per year are not required
  9. 9. How should countries select PEPFAR-funded OVC projects for an OVC Essential Indicator Survey? Considerations: • Agency representation • Project size, scope, and funding level • Availability of accurate data project registers • Timeline of the project • Budget available for the survey Not necessary to collect MER OVC ESIs from all OVC projects in a country
  10. 10. Who is responsible for designing, collecting, and reporting the MER Essential Survey Indicators? USG Mission • Responsible for collection of MER ESIs • Procures data collection services directly—or through the OVC implementing partner(s) Data Collection Organization • Must be external to project service delivery • Writes survey report and prepares indicators for reporting and submission OVC Implementing Partner • Enters the indicator data into DATIM
  11. 11. Do study protocols need to be reviewed by USG? Yes, protocols should be submitted to the USG Mission and the PEPFAR OVC Technical Working Group - for review prior to IRB submission Protocols will be reviewed to ensure that guidance is being adhered to
  12. 12. Is ethical approval required for the collection of the MER Essential Survey Indicators? U.S. regulations: Not considered research and does not require ethical review However, 1) surveys involve vulnerable populations, and 2) ethics review may be required locally • If country requires ethics review—recommended to submit to U.S. ethics board
  13. 13. Do all nine Essential Survey Indicators need to be collected? • Yes—even if project not implementing interventions directly linked to outcome indicator • Allows for holistic picture of well-being of project beneficiaries across projects and countries
  14. 14. Have any changes been made to the survey tool? Two questions added: 1) Which specific OVC interventions the household and/or its members have received 2) When beneficiary was registered with project and/or how long beneficiary has been receiving services
  15. 15. Is it possible to adapt the survey questions to better reflect the local context? The indicators should be administered exactly as written Translated questions should be adjusted to align with local discourse Recall periods should not be changed
  16. 16. Is it possible to add questions? Questions can be added to obtain additional information Balance need for additional information with extra time and costs
  17. 17. When should the MER survey be conducted? • Should be collected at:  two points in time  two years apart  during the same time of the year  beginning of project • Need to balance:  sufficient number of beneficiaries enrolled and  beneficiaries having not received more than six months of services Do not recommend conducting MER survey for projects ending in less than two-and-a-half years.
  18. 18. Who should be surveyed at Round 1? • Households representative of all registered, active project beneficiaries • Survey not designed to assess outcomes among children in general population Photo credit: Lisa Marie Albert
  19. 19. How should geographic areas, households, and children be chosen for the Round 2 survey? Beneficiary cohorts maintained for at least two years: • Cross-section of all active beneficiaries at the time of Round 2 Graduate beneficiaries in less than two years: • Cross-section of all active beneficiaries at the time of Round 1(includes those that have transitioned, graduated, or exited the project)
  20. 20. Should we conduct MER surveys in PEPFAR transition areas or for projects that are follow-ons? We do not recommend collecting the ESIs in PEPFAR transition areas If MER survey conducted on previous project within last two years not necessary to conduct baseline MER survey for new project • If follow-on project is similar to previous project
  21. 21. Is it possible to compare results across projects? Advise against comparing indicator results across projects even within the same country Comparisons between projects may be misleading Projects can differ significantly from one another: • How long project implemented • Target populations/project eligibility requirements • Different geographies
  22. 22. Is it acceptable to aggregate data from all projects operating within a geographic region? We advise against aggregating due to variation in: • project implementation • design • duration • contextual factors, and • beneficiary vulnerability criteria Data must be project-specific and reflect project as a whole
  23. 23. If improvements over time in the indicators can we attribute changes to the project? No comparison group—we cannot attribute changes in indicators to project Triangulating the MER survey data with data from: • routine monitoring • household vulnerability assessments • evaluations, and • case management Use conceptual framework to help interpret
  24. 24. Can the MER OVC ESIs be accurately captured through an evaluation? 1. Does the evaluation study include a large enough sample of project beneficiaries? 2. Does the evaluation study include all geographic areas where the program operates or is it restricted to a sub-area? 3. Does the evaluation focus on beneficiaries receiving a specific subset of interventions?
  25. 25. What type of information should be included in the MER Essential Survey Indicator report? • Study design • Sampling procedures • Study population • Indicator values presented in context  Data interpretation  Conceptual framework
  26. 26. Who needs to have a copy of the final MER Essential Survey Indicator data and reports? Organization designing and collection the MER data • Provides study dataset/s in an excel file or csv format to the USG Mission • Submits survey report to PEPFAR OVC Technical Working Group via Christine Fu, USAID’s senior research and evaluation advisor, at chfu@usaid.gov Data storage • At USG mission and on USAID/Washington’s Datahub repository Datasets must be de-identified
  27. 27. How should the data be entered into DATIM? Numerators and denominators should be entered by age-group and sex Narrative section should include: • OVC project interventions • Beneficiary population • Study design • Sampling strategy and sample size • Etc.
  28. 28. How many households should be sampled for the survey?
  29. 29. How should we select our sample? Cluster Sampling: For medium-sized and large projects 30 cluster minimum Probability proportionate to size recommended Example: 33 cluster x 15 household design Simple Random Sampling For smaller projects
  30. 30. Which children in the household should be included in the survey? Interview caregiver about all children ages 0–17 years within the beneficiary household Photo credit: Lisa Marie Albert
  31. 31. How can we ensure that the sampling frame is representative of program beneficiaries? Conduct data quality assessments Address any deficiencies in record-keeping prior to implementing survey and developing sampling frame
  32. 32. Future resources in development • Protocol template • Community Trace and Verify guidance • Enumerator data collection manual • Supervisor data collection manual
  33. 33. Where can I find out more?
  34. 34. Q&A USAID • Gretchen Bachman • Christine Fu MEASURE Evaluation/Palladium • Lisa Parker • Lisa Marie Albert (OVCimpact.org) • Jenifer Chapman (Mozambique, Tanzania) • Walter Obiero (Nigeria, Namibia) • Susan Settergren (Kenya, Lesotho) Resource persons:
  35. 35. Further questions? Please contact Christine Fu, USAID senior research and evaluation advisor, at: chfu@usaid.gov To reach MEASURE Evaluation send e-mails to ovcimpact@thepalladiumgroup.com OVC Impact Website: https://ovcimpact.org/
  36. 36. This presentation was produced with the support of the United States Agency for International Development (USAID) under the terms of MEASURE Evaluation cooperative agreement AID-OAA-L-14-00004. MEASURE Evaluation is implemented by the Carolina Population Center, University of North Carolina at Chapel Hill in partnership with ICF International; John Snow, Inc.; Management Sciences for Health; Palladium; and Tulane University. Views expressed are not necessarily those of USAID or the United States government. www.measureevaluation.org

×