Gretchen Bachman and Christine Fu (USAID); Lisa Parker, Jenifer Chapman, Lisa Marie Albert, Walter Obiero, and Susan Settergren from MEASURE Evaluation. January 2017 Webinar.

1. Monitoring, Evaluation, and Reporting (MER) Orphan and
Vulnerable Children Essential Survey Indicators
Gretchen Bachman and
Christine Fu
USAID
Lisa Parker, Jenifer Chapman,
Lisa Marie Albert, Walter
Obiero, and Susan Settergren
MEASURE
Evaluation/Palladium
January 31, 2017
FREQUENTLYASKEDQUESTIONS

2. Outcome indicators measuring household and child
well-being
• Designated as “Essential Survey Indicators” (ESI)
Purpose:
• To obtain snapshot of program outcomes at a point in time
AND
• To assess changes in outcomes among orphans and
vulnerable children (OVC) program beneficiaries over time
The purpose

3. 9 MER Essential Survey Indicators
1. Percent of children whose primary caregiver knows child’s HIV status
2. Percent of children <5 years of age who are undernourished
3. Percent of children too sick to participate in daily activities
4. Percent of children who have a birth certificate
5. Percent of children regularly attending school
6. Percent of children who progressed in school during the last year
7. Percent of children <5 years of age who recently engaged in
stimulating activities with any household member over 15 years of age
8. Percent of caregivers who agree that harsh physical punishment is an
appropriate means of discipline or control in the home or school
9. Percent of households able to access money to pay for unexpected
household expenses

4. MER implementation to date
Data reported into DATIM:
• Uganda
• Zimbabwe
• Swaziland
Data collection completed:
• Namibia
• Nigeria
• Kenya
• Cote d’Ivoire
Upcoming:
• Lesotho
• Rwanda
• Mozambique
• Botswana
• Tanzania
• Haiti
• Malawi
• DRC

5. Frequently Asked Questions

6. How will the data be used?
Data will support:
• Strategic portfolio development
• Targeting
• Programming
• Implementation
• Resource allocations
• Provide evidence to the U.S. Congress
• Be triangulated with other relevant OVC
data

7. What types of decisions will be made based on
MER OVC Essential Survey Indicator data?
Example: Education
indicators
• USG Mission
decisions
• Implementing
partner decisions

8. Is a MER OVC ESI survey required for all
OVC projects?
• Requirement for OVC projects with an
annual budget of USD $1 million
• Countries with total HKID funding of less
than USD$1 million per year are not
required

9. How should countries select PEPFAR-funded OVC projects
for an OVC Essential Indicator Survey?
Considerations:
• Agency representation
• Project size, scope, and funding level
• Availability of accurate data project registers
• Timeline of the project
• Budget available for the survey
Not necessary to collect MER OVC ESIs from all OVC
projects in a country

10. Who is responsible for designing, collecting, and reporting
the MER Essential Survey Indicators?
USG Mission
• Responsible for collection of MER ESIs
• Procures data collection services directly—or
through the OVC implementing partner(s)
Data Collection Organization
• Must be external to project service delivery
• Writes survey report and prepares indicators for
reporting and submission
OVC Implementing Partner
• Enters the indicator data into DATIM

11. Do study protocols need to be reviewed
by USG?
Yes, protocols should be submitted to the
USG Mission and the PEPFAR OVC Technical
Working Group - for review prior to IRB
submission
Protocols will be reviewed to ensure that
guidance is being adhered to

12. Is ethical approval required for the collection of the
MER Essential Survey Indicators?
U.S. regulations: Not considered research and
does not require ethical review
However,
1) surveys involve vulnerable populations, and
2) ethics review may be required locally
• If country requires ethics review—recommended to
submit to U.S. ethics board

13. Do all nine Essential Survey Indicators need to
be collected?
• Yes—even if project not implementing
interventions directly linked to outcome
indicator
• Allows for holistic picture of well-being of
project beneficiaries across projects and
countries

14. Have any changes been made to the survey
tool?
Two questions added:
1) Which specific OVC interventions the
household and/or its members have
received
2) When beneficiary was registered with
project and/or how long beneficiary has
been receiving services

15. Is it possible to adapt the survey questions to
better reflect the local context?
The indicators should be administered
exactly as written
Translated questions should be adjusted to
align with local discourse
Recall periods should not be changed

16. Is it possible to add questions?
Questions can be added to obtain
additional information
Balance need for additional information
with extra time and costs

17. When should the MER survey be conducted?
• Should be collected at:
 two points in time
 two years apart
 during the same time of the year
 beginning of project
• Need to balance:
 sufficient number of beneficiaries enrolled and
 beneficiaries having not received more than six
months of services
Do not recommend conducting MER survey for projects
ending in less than two-and-a-half years.

18. Who should be surveyed at Round 1?
• Households
representative of
all registered,
active project
beneficiaries
• Survey not
designed to assess
outcomes among
children in general
population
Photo credit: Lisa Marie Albert

19. How should geographic areas, households, and
children be chosen for the Round 2 survey?
Beneficiary cohorts maintained for at least two
years:
• Cross-section of all active beneficiaries at the time of
Round 2
Graduate beneficiaries in less than two years:
• Cross-section of all active beneficiaries at the time of
Round 1(includes those that have transitioned,
graduated, or exited the project)

20. Should we conduct MER surveys in PEPFAR transition
areas or for projects that are follow-ons?
We do not recommend collecting the ESIs in
PEPFAR transition areas
If MER survey conducted on previous project
within last two years not necessary to conduct
baseline MER survey for new project
• If follow-on project is similar to previous project

21. Is it possible to compare results across
projects?
Advise against comparing indicator results across projects
even within the same country
Comparisons between projects may be misleading
Projects can differ significantly from one another:
• How long project implemented
• Target populations/project eligibility requirements
• Different geographies

22. Is it acceptable to aggregate data from all projects
operating within a geographic region?
We advise against aggregating due to variation
in:
• project implementation
• design
• duration
• contextual factors, and
• beneficiary vulnerability criteria
Data must be project-specific and reflect project
as a whole

23. If improvements over time in the indicators can
we attribute changes to the project?
No comparison group—we cannot attribute
changes in indicators to project
Triangulating the MER survey data with data from:
• routine monitoring
• household vulnerability assessments
• evaluations, and
• case management
Use conceptual framework to help interpret

24. Can the MER OVC ESIs be accurately
captured through an evaluation?
1. Does the evaluation study include a large
enough sample of project beneficiaries?
2. Does the evaluation study include all
geographic areas where the program operates
or is it restricted to a sub-area?
3. Does the evaluation focus on beneficiaries
receiving a specific subset of interventions?

25. What type of information should be included in the
MER Essential Survey Indicator report?
• Study design
• Sampling procedures
• Study population
• Indicator values presented in context
 Data interpretation
 Conceptual framework

26. Who needs to have a copy of the final MER
Essential Survey Indicator data and reports?
Organization designing and collection the MER data
• Provides study dataset/s in an excel file or csv format
to the USG Mission
• Submits survey report to PEPFAR OVC Technical
Working Group via Christine Fu, USAID’s senior
research and evaluation advisor, at chfu@usaid.gov
Data storage
• At USG mission and on USAID/Washington’s Datahub
repository
Datasets must be de-identified

27. How should the data be entered into
DATIM?
Numerators and denominators should be
entered by age-group and sex
Narrative section should include:
• OVC project interventions
• Beneficiary population
• Study design
• Sampling strategy and sample size
• Etc.

28. How many households should be sampled for
the survey?

29. How should we select our sample?
Cluster Sampling:
For medium-sized and large projects
30 cluster minimum
Probability proportionate to size recommended
Example: 33 cluster x 15 household design
Simple Random Sampling
For smaller projects

30. Which children in the household should be
included in the survey?
Interview caregiver
about all children
ages 0–17 years
within the
beneficiary
household
Photo credit: Lisa Marie Albert

31. How can we ensure that the sampling frame is
representative of program beneficiaries?
Conduct data quality
assessments
Address any
deficiencies in
record-keeping prior
to implementing
survey and
developing sampling
frame

32. Future resources in development
• Protocol template
• Community Trace and Verify guidance
• Enumerator data collection manual
• Supervisor data collection manual

33. Where can I find out more?

34. Q&A
USAID
• Gretchen Bachman
• Christine Fu
MEASURE Evaluation/Palladium
• Lisa Parker
• Lisa Marie Albert (OVCimpact.org)
• Jenifer Chapman (Mozambique, Tanzania)
• Walter Obiero (Nigeria, Namibia)
• Susan Settergren (Kenya, Lesotho)
Resource persons:

35. Further questions?
Please contact Christine Fu, USAID senior research
and evaluation advisor, at: chfu@usaid.gov
To reach MEASURE Evaluation send e-mails to
ovcimpact@thepalladiumgroup.com
OVC Impact Website: https://ovcimpact.org/

36. This presentation was produced with the support of the United States Agency for
International Development (USAID) under the terms of MEASURE Evaluation
cooperative agreement AID-OAA-L-14-00004. MEASURE Evaluation is
implemented by the Carolina Population Center, University of North Carolina at
Chapel Hill in partnership with ICF International; John Snow, Inc.; Management
Sciences for Health; Palladium; and Tulane University. Views expressed are not
necessarily those of USAID or the United States government.
www.measureevaluation.org