Anbrasi Edward, PhD, MPH, MBA, MSc, Associate Scientist, Johns Hopkins University Bloomberg School of Public Health and Jennifer Winestock Luna, MPH, Director of M&E Services for Realizing Global Health describe Program Evaluation Models and use a case study of a program in Yemen to lead participants through an example of monitoring and evaluation practices.

1. Johns Hopkins Bloomberg School of Public Health
Construct and Conduct of Monitoring and Evaluation for
Health Programs
Anbrasi Edward and Jennifer Winestock Luna
Christina Bowles
Christian Connections for International Health
Pre-conference Workshop July 13, 2017

2. 2
Modules 4 & 5 combined:
Using the Yemen case study to
illustrate strengths and
weaknesses of models of
program evaluation

3. © 2014, Johns Hopkins University. All rights reserved.©2016, Johns Hopkins University. All rights reserved.
Aspects of Program Evaluation Models
• Identifying technical areas of the project
• Determining target populations
• Identifying appropriate sources of information
• Choosing indicators
3

4. © 2014, Johns Hopkins University. All rights reserved.©2016, Johns Hopkins University. All rights reserved.
Yemen Example
• Planned as a 5-year project
• Initiated, but suspended due to security situation
• During years 1&2 scheduled to be in 3 governorates
• Sana’a Governorate, Dhamar, and Aden
• Additional Governorates to be added later
• Community and facility level work
• Rural and urban areas.

5. © 2014, Johns Hopkins University. All rights reserved.©2016, Johns Hopkins University. All rights reserved.
Technical areas?

6. © 2014, Johns Hopkins University. All rights reserved.©2016, Johns Hopkins University. All rights reserved.
Technical Areas
Sick Child Pneumonia (ARI) x
Diarrhea (CDD) x
Childhood Malaria
Community case management
(CCM)
Malaria (childhood and adult)
Immunization x
Maternal and Newborn care Maternal x
Newborn care x
Pregnancy Spacing and Family Planning x
Breastfeeding and IYCF (nutrition) x
Water and Sanitation (WASH) x
Gender x

7. © 2014, Johns Hopkins University. All rights reserved.©2016, Johns Hopkins University. All rights reserved.
What do we mean by target population?

8. © 2014, Johns Hopkins University. All rights reserved.©2016, Johns Hopkins University. All rights reserved.
Target population:
• Refers to People
• Who we expect to change because of our intervention
• Who we expect to benefit from our intervention
i.e.
• Newborns because we expect them to be breastfed within one
hour of birth
• Mothers of newborns because they will breast feed their
newborns
• Community midwives because we expect them to know how to
facilitate immediate breastfeeding
• District health managers because we expect them to know how
supervise midwives who help mothers

9. © 2014, Johns Hopkins University. All rights reserved.©2016, Johns Hopkins University. All rights reserved.
Yemen example
What target populations did you find?

10. © 2014, Johns Hopkins University. All rights reserved.©2016, Johns Hopkins University. All rights reserved.
Target populations
Pregnant women Couples
Women giving birth; Newborns
Women with PE/E Low birth weight babies
Women in post-natal period Infants under 6 months
Women 15-49 years who gave birth Children under 2 years
Women with children <24 months Children <5 years of age
Married women

11. © 2014, Johns Hopkins University. All rights reserved.©2016, Johns Hopkins University. All rights reserved.
Target populations
Governorate- level staff Communities; Community and
religious leaders
District-level staff CHVs,
Immunization managers Volunteers
Health care providers district level Non-health personnel
Health care providers primary health centers Trainers
Immunization service providers
Rural informal drug sellers

12. © 2014, Johns Hopkins University. All rights reserved.©2016, Johns Hopkins University. All rights reserved.
Target populations
For measurement purposes good to combine groups:
Survey mothers of children < 24 months to obtain information about the
experience of those mothers:
• During their most recent pregnancy
• When they delivered their youngest child
• During the post-natal period after the birth of their youngest child
Ask same mothers about their child <24 months
• Nutrition
• Immunization status

13. Sources of Information
13

14. © 2014, Johns Hopkins University. All rights reserved.©2016, Johns Hopkins University. All rights reserved.
Sources of Information
Quantitative
• Provides information that can be statistically significant
• Sampling can be used to estimate levels for population
Qualitative
• Answers questions about why
• Focus groups
• Key informant interviews
• Observation about behaviors (i.e. service provider/client
interactions)

15. © 2014, Johns Hopkins University. All rights reserved.©2016, Johns Hopkins University. All rights reserved.
Sources of Information
• Routine
• Periodic

16. © 2014, Johns Hopkins University. All rights reserved.©2016, Johns Hopkins University. All rights reserved.
Sources of information
• Examples of routine sources?

17. © 2014, Johns Hopkins University. All rights reserved.©2016, Johns Hopkins University. All rights reserved.
Sources of information
Routine - examples
• Health information system (HMIS)
• DHIS2
• Health facility records
• Patient registers
• CHW activity reports
• Child health cards
• Project monitoring information
17

18. © 2014, Johns Hopkins University. All rights reserved.©2016, Johns Hopkins University. All rights reserved.
Routine Information – DHIS2
• Who is familiar with DHIS2
• What is it?

19. © 2014, Johns Hopkins University. All rights reserved.©2016, Johns Hopkins University. All rights reserved.
Routine Information – DHIS2
Open source software platform
• For reporting, analysis, and dissemination for health programs
• Developed by Health Information Systems Programme (HISP, University of Oslo)
• Used by more than 40 countries around the world as part of routine health
information systems
• Health facilities input data periodically, not necessarily at same time; important to record
date when information is extracted
• Used by NGOs and Project
• I.e. CARE, USAID’s Maternal and Child Survival Program (MCSP)

20. Header/Full Bleed Image
Name/Subject Subhead
Title/caption
Title/caption
©2015, Johns Hopkins University. All rights reserved.©2015, Johns Hopkins University. All rights reserved.©2016, Johns Hopkins University. All rights reserved.

21. © 2014, Johns Hopkins University. All rights reserved.©2016, Johns Hopkins University. All rights reserved.
Routine Information is useful
HMIS information:
• Is available on a regular basis
• Does not require the same effort to collect as a survey
• Provides information about services and activities that won’t obtain
from a survey
•But..
• Leaves out details of knowledge, practice, and coverage of
households
• Does not provide information about people who do not use health
services

22. © 2014, Johns Hopkins University. All rights reserved.©2016, Johns Hopkins University. All rights reserved.
Routine Information is useful
Project monitoring information:
• Can be tailored to exact needs
• Provides information on activities implemented
• May provide information about context
•But…
• Normally does not contain information about specific households
• Cannot be sampled to obtain population estimates of coverage
• May not be sustained after project ends

23. © 2014, Johns Hopkins University. All rights reserved.©2016, Johns Hopkins University. All rights reserved.
Sources of information
• Examples of periodic sources?

24. © 2014, Johns Hopkins University. All rights reserved.©2016, Johns Hopkins University. All rights reserved.
Sources of information – Periodic examples
• Census
• National or local, CBIO
• Health facility surveys
• Service Provision Assessment (SPA)
• Quality of Care (QoC) surveys
• Includes observation
• Household surveys
• DHS, MICS
• Knowledge, Practice, and Coverage (KPC)
24

25. © 2014, Johns Hopkins University. All rights reserved.©2016, Johns Hopkins University. All rights reserved.
Sources of information – Periodic
Census
• Usually more expensive than a survey
• Provides exact information on population level coverage
• No need to calculate confidence intervals, since this represents the
universe in a given geographical area.
• Useful if available for population you are interested in for the time
period that are interested in.
• May be part of a project or may be national
• If included as part of routine project work, can be valuable and practical

26. © 2014, Johns Hopkins University. All rights reserved.©2016, Johns Hopkins University. All rights reserved.
Periodic Information - Surveys
Health facility surveys
• Service Provision Assessment (SPA)
• Quality of Care (QoC) surveys
• Includes observation
Household surveys
• DHS, MICS
• Large national level, limited geographical disaggregation
• May not coincide with intervention are you are studying
• Long time between surveys, may not coincide with intervention
you are studying

27. © 2014, Johns Hopkins University. All rights reserved.©2016, Johns Hopkins University. All rights reserved.
Periodic Information – KPC Surveys
Rapid Small Sample Household Survey that:
• Useful for designing surveys for integrated health programs linked to
communities
• Developed to collect standardized information from a variety of technical
areas
• Contains Modules that have questionnaires, indicators, tab plans, instructions
Implementers select modules relevant to technical areas
• Within each module select questions & indicators relevant to interventions

28. KPC Modules
 Sick Child - (ARI, CDD, Malaria,
CCM)
 Malaria (stand-alone)
 Immunization
 Maternal Newborn Care
 Pregnancy Spacing & Family
Planning
 Water & Sanitation
 Nutrition module (IYCF, child
anth. Maternal Nut.)
 Gender

29. © 2014, Johns Hopkins University. All rights reserved.©2016, Johns Hopkins University. All rights reserved.
KPC Tool Resources
MCSP website
• http://www.mcsprogram.org/resource/knowledge-practice-coverage-
tool/?_sfm_resource_topic=monitoring-evaluation
CORE Group website
• http://www.coregroup.org/our-technical-work/working-groups/monitoring-
and-evaluation
Realizing Global Health (RGH)
• Rapid Survey Handbook
• https://www.realizingglobalhealth.com/recommended-resources/

30. Indicators
30

31. © 2014, Johns Hopkins University. All rights reserved.©2016, Johns Hopkins University. All rights reserved.
Indicator
What is an indicator?
31

32. © 2014, Johns Hopkins University. All rights reserved.©2016, Johns Hopkins University. All rights reserved.
Indicator
A variable…
...that measures…
…one aspect of a project or program…
…for a specific population
32

33. Characteristics of a Good Indicator
5. Comparable
6. Measurable
7. Timely
8. Important for the project
1. Valid
2. Reliable
3. Precise
4. Non-directional definition
33

34. © 2014, Johns Hopkins University. All rights reserved.©2016, Johns Hopkins University. All rights reserved.
Characteristics of a Good Indictor
Valid means that…
• The indicator measures what it is intended to measure
• The change in the indicator matches the change in the element that we
want to influence
Reliable means that…
• Repeated measurements at different times and by different people will
yield the same result

35. © 2014, Johns Hopkins University. All rights reserved.©2016, Johns Hopkins University. All rights reserved.
Characteristics of a Good Indicator
Precise means that…
• The indicator measures what it is supposed to measure
Non-directional definition means that…
• The indicator does not state how the measurement will change
Correct definition Directional definition
Children <6 months exclusively
breastfed
Increased % of children<6
months exclusively breastfed
Prevalence of stunting in
Children <24 months
Decreased prevalence of
stunting in Children <24 months

36. © 2014, Johns Hopkins University. All rights reserved.©2016, Johns Hopkins University. All rights reserved.
Characteristics of a Good Indicator
Comparable means that…
• The same information can be measured in different contexts
• Different countries, regions within a country
• The same information can be compared at different points in time.
• It is important to measure the same indictor at the beginning and at
other points in time (i.e. final evaluation)

37. © 2014, Johns Hopkins University. All rights reserved.©2016, Johns Hopkins University. All rights reserved.
Characteristics of a Good Indicator
Measurable means that…
• Methodologies exist to measure it
• Sources of information are available to collect it
• Cost of collecting information is not prohibitive
• It is practical to collect

38. © 2014, Johns Hopkins University. All rights reserved.©2016, Johns Hopkins University. All rights reserved.
Characteristics of a Good Indicator
Timely means that…
• Collection produces information that can be used by
• Project implementers when they need the information
• Stakeholders when they need the information
• Collection responds to reporting requirements of donors
• Repeated measurements allow sufficient time for change to occur

39. © 2014, Johns Hopkins University. All rights reserved.©2016, Johns Hopkins University. All rights reserved.
Characteristics of a Good Indicator
Important for the project means that…
• The information is useful for:
• Decision making
• Determining results or objectives that are part of the project design
• Sharing results of project with others

40. © 2014, Johns Hopkins University. All rights reserved.©2016, Johns Hopkins University. All rights reserved.
Small Group Work – (3 people, 20 min)
Create a basic M&E System to measure the following (Table 1 Yemen example)
• The project will work with community and religious leaders
• To generate demand for spacing out intervals between the last birth and the next
pregnancy
• So that the mother can breastfeed her baby for the first two years to promote
good health.
Define
• Target populations
• Sources of information
• Select 2-3 illustrative indicators

41. © 2014, Johns Hopkins University. All rights reserved.©2016, Johns Hopkins University. All rights reserved.
Plenary Discussion

42. © 2014, Johns Hopkins University. All rights reserved.©2016, Johns Hopkins University. All rights reserved.
Thank you