This document discusses how monitoring and evaluation (M&E) systems can support program evaluation through three case studies. It finds that while M&E systems may not replace randomized controlled trials, they can provide adequate evidence when integrated data from routine collection and surveys are analyzed together. Specifically, the document examines how M&E data has been used to evaluate the impact of a community health insurance scheme in Burkina Faso, a malaria partnership in Tanzania, and an HIV prevention program in India. In each case, the M&E system data provided plausible evidence of impact when analyzed longitudinally and through dose-response relationships, though the studies fell short of experimental evidence.

1. M&E Systems for Evaluation:
Where M meets E
Elizabeth G. Sutherland
Jim Thomas
September 14, 2012

2. What do we mean by M&E
system?

3. Strengths of M&E systems

4. Can M&E systems data support
evaluation?
 Increasing emphasis on need for evaluation to
drive evidence-based programming
 Recognition of limitations of RCTs for large-scale
public health program evaluation (e.g. Victora et
al. 2012)
 Rise of innovative approaches integrated with
routine M&E systems in countries

5. Examples from the literature

6. Making the most of M&E Data
 Community Health Insurance (CHI) Scheme in
Burkina Faso
 RBM partnership Impact Evaluation in Tanzania
 Avahan Assessment in India

7. CHI: Objective
 Determine impact of CHI on health service
utilization in Nouna District, Burkina Faso

8. Design of CHI evaluation
 Step Wedge design
 Households with insurance vs. no insurance
 All households eventually received access to the
intervention
 Study duration was 4 years

9. CHI: Results
 40% increase in number of outpatient visits
 2% increase in number of inpatient visits
 Comparison between insured households in
intervention area to matched controls in non-
intervention areas

10. CHI: Data sources
 Demographic Surveillance System (DSS)
 Sampling frame
 Cluster characteristics
 Household surveys in study clusters
 CHI program data

11. CHI: Advantages
 Uses mix of existing data collection sampling
frames and data, as well as collecting new data
 Step wedge a relatively rigorous approach,
provides good evidence

12. CHI: Disadvantages
 Feasibility of randomized roll out
 Can lead to extended time frame for intervention
implementation and study duration
 Possibility of contamination across clusters

13. RBM partnership: Objectives
 To describe trends in intervention coverage
 To describe trends in morbidity and mortality
 To attempt to link these trends

14. RBM partnership: Design
 Plausibility analysis
 Temporal analysis
 Dose response analysis
 Deaths averted models

15. RBM partnership: Trends

16. RBM partnership: Estimating
impact

17. RBM partnership: Data sources
 Population based surveys (DHS, MIS, MICS)
 HMIS
 HDSS case study
 Special studies and reports
 IGME, IHME estimates

18. RBM partnership: Advantages
 Makes full use of available data through M&E
system and other sources to answer “E”
questions
 Efficient and feasible
 Recognizes that effectiveness of individual
program interventions is already proven

19. RBM partnership: Disadvantages
 Retrospective
 Incomplete coverage of data sources
 Builds a plausible case cannot attribute probable
causation

20. Avahan: Objectives
 Avahan is a large scale HIV prevention program
seeking to:
 Target high-risk groups
 Estimate impact on HIV prevalence in the general
population

21. Avahan: Design
 Launched in six states simultaneously
 Plausibility design
 Dose response analysis

22. Avahan: Results

23. Avahan: Data sources
 Avahan program data
 Antenatal HIV surveillance data
 National Family Health Survey data

24. Avahan: Advantages
 Makes efficient use of existing data
 Takes advantage of heterogeneity of program
implementation
 Provides some statistical evidence

25. Summary
 M&E systems are capable of producing high-quality
data
 These data can be used for evaluation purposes
 These evaluations fall short of “gold standard” of RCT
 Feasible and efficient
 Adequate or Plausible case, not necessarily Probable

26. Avahan: Disadvantages
 Not the gold standard evidence provided by
experimental data
 Modeling (like all statistical models) relies on
some assumptions when making estimates,
which may or may not be valid

27. References
12 Components Monitoring and Evaluation System Strengthening Tool. Geneva: UNAIDS, 2009a. Available at:
http://www.unaids.org/en/media/unaids/contentassets/documents/document/2010/2_MERG_Strengthening_Tool_12_Components_ME_System.p
Victora CG, Black RE, Boerma JT, Bryce J. Measuring impact in the Millennium Development Goal era and beyond: a new approach to
large-scale effectiveness evaluations. The Lancet 2011; 377: 85-95.
Habich, J., Victora, C., and Vaughan, J. Evaluation designs for adequacy, plausibility and probability of public health programme
performance and impact. International Journal of Epidemiology,1999 Feb;28(1):10-8.
De Allegrini, M., Pokhrel S, Becher, H. et al. Step-wedge cluster-randomised community based trials: An application to the study of the
impact of community health insurance. Health Research Policy and Systems 2008, 6:10.
Roll Back Malaria Partnership. Focus on Mainland Tanzania. Progress and Impact Series number 3. January 2012. Available at:
http://www.rbm.who.int/ProgressImpactSeries/docs/report10-en.pdf
Ng, M. , Gakidou, E. , Levin-Recotr, A., et al. Assessment of population-level effect of Avahan, an HIV-prevention initiative in India. The
Lancet 10.1016/S0140-6736(11)61390-1. Published online October 11, 2011.

28. Take homes and your examples

29. Pearl
Building strong M&E systems has the potential to
yield high quality data for “E” as well as “M” data
needs

30. MEASURE Evaluation is funded by the U.S. Agency for
International Development (USAID) and implemented by the
Carolina Population Center at the University of North Carolina
at Chapel Hill in partnership with Futures Group International,
ICF International, John Snow, Inc., Management Sciences for
Health, and Tulane University. Views expressed in this
presentation do not necessarily reflect the views of USAID or the
U.S. government.
MEASURE Evaluation is the USAID Global Health Bureau's
primary vehicle for supporting improvements in monitoring and
evaluation in population, health and nutrition worldwide.