Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Beyond Indicators and Reporting: M&E as a Systems Strengthening Intervention


Published on

Sian Curtis, MEASURE Evaluation Project Director
Washington, DC
January 24, 2012

Published in: Health & Medicine
  • Be the first to comment

Beyond Indicators and Reporting: M&E as a Systems Strengthening Intervention

  1. 1. Beyond Indicators and Reporting: M&E as a Systems Strengthening Intervention Siân Curtis, PhD Project Director, MEASURE Evaluation Carolina Population Center January 24, 2012, Washington, DC
  2. 2. Overview <ul><li>Background: </li></ul><ul><ul><li>Demand for M&E </li></ul></ul><ul><ul><li>Tensions </li></ul></ul><ul><li>Progress </li></ul><ul><li>Lessons and Opportunities </li></ul><ul><li>Conclusions </li></ul>
  3. 3. Why do we do M&E? <ul><li>Accountability and Funding </li></ul><ul><ul><li>What gets measured gets done </li></ul></ul><ul><ul><li>If you’re not measured you don’t count </li></ul></ul><ul><li>Program Management, Learning and Evaluation </li></ul><ul><ul><li>Are programs operating as expected? </li></ul></ul><ul><ul><li>Are programs achieving the intended benefits? </li></ul></ul><ul><ul><li>Where do I best invest scarce resources? </li></ul></ul><ul><ul><li>How can we improve programs in future? </li></ul></ul>
  4. 4. Demand for M&E in Global Health <ul><li>1990s: Results Frameworks in USAID </li></ul><ul><li>2000: MDG targets </li></ul><ul><li>2000s: PEPFAR & GFATM </li></ul><ul><li>2009: WHO M&E Framework </li></ul><ul><li>2009: GHI – metrics, monitoring and evaluation </li></ul><ul><li>2010: USAID Evaluation Policy </li></ul>
  5. 5. Perceived Weak M&E Systems <ul><li>Evaluation of World Bank MAP program highlighted weak M&E systems (Ainsworth et al. 2005) </li></ul><ul><li>Impact evaluation of GFATM highlighted major data gaps in countries that seriously limit the ability to evaluate programs (Macro International, 2009) </li></ul><ul><li>M&E concerns listed for remedial action in 56% of GFATM grants in sub-Saharan Africa (Peersman et al. 2009) </li></ul>
  6. 6. Why are M&E Systems Weak?
  7. 7. 12 Components
  8. 9. M&E and HIS <ul><li>Information Sources </li></ul><ul><li>RHIS </li></ul><ul><li>Surveys & Surveillance </li></ul><ul><li>Rapid Assessments </li></ul><ul><li>Program Records etc. </li></ul><ul><li>Functions </li></ul><ul><li>Facility and health system </li></ul><ul><li>Management </li></ul><ul><li>Planning </li></ul><ul><li>Budgeting </li></ul><ul><li>M&E </li></ul>
  9. 10. Tensions and Challenges <ul><li>Proliferation of indicators in M&E overwhelms existing HIS and potentially undermines data quality and other HIS functions (Boerma and Stansfield, 2007). </li></ul><ul><li>Ripple effect: changes in indicators have potentially large impact on data collection </li></ul><ul><li>Challenge of generating accurate and complete data from large numbers of funding recipients within a single country for a large number of interventions (Macro International, 2009) </li></ul><ul><li>Balance between global, national, and sub-national health information needs and between accountability and learning functions </li></ul>
  10. 11. The burden of data
  11. 12. What progress has been made?
  12. 13. Harmonization at Global Level <ul><li>Indicator Standards </li></ul><ul><ul><li>UNAIDS National AIDS programmes: A Guide to Monitoring and Evaluation, 2000 </li></ul></ul><ul><ul><li>UNAIDS Indicator registry (launched 2009) </li></ul></ul><ul><ul><li>Guidelines for Core Population Coverage Indicators for Roll Back Malaria (2004, 2009) </li></ul></ul><ul><li>Data Collection Tools </li></ul><ul><ul><li>WHO Three interlinked patient monitoring system (2009) </li></ul></ul><ul><ul><li>Malaria Indicator Survey Toolkit (2005) </li></ul></ul>
  13. 14. Percentage of Countries With Key HIV M&E System Elements, NCPI, 2005 and 2007. Source: Peersman et al. 2009. N=85
  14. 15. Availability of Selected UNGASS Indicators, 2008 Among countries that considered the indicator relevant. N <=147 Source: Peersman et al. 2009.
  15. 16. Example: Jamaica
  16. 17. M&E System Development <ul><li>Pre-2004: Multiple, uncoordinated data collection systems, databases and donor reports </li></ul><ul><li>2004: HIV/AIDS M&E unit formed. Baseline assessment of national M&E system. </li></ul><ul><li>2006: Stakeholder capacity assessment. </li></ul><ul><ul><li>Roles and responsibilities defined </li></ul></ul><ul><ul><li>M&E system performance gaps prioritized </li></ul></ul>
  17. 18. M&E System Development <ul><li>2007: M&E Reference Group (MERG) established </li></ul><ul><ul><li>Consolidation of 130+ indicators to 30 core national indicators </li></ul></ul><ul><li>2008-2010: M&E operations manual developed </li></ul><ul><ul><li>46 data collection forms consolidated into 25 required and recommended national forms mapped to indicators </li></ul></ul><ul><ul><li>Parish-specific protocols consolidated into national standard data collection protocols </li></ul></ul>
  18. 19. M&E System Development <ul><li>2010: M&E Manual Roll out </li></ul><ul><ul><li>1000+ people trained 2007-2011 </li></ul></ul><ul><ul><li>Over 80% or people with HIV M&E system responsibilities using at least one of the new forms </li></ul></ul><ul><li>2011: Impact Evaluation Training </li></ul><ul><li>2012: Leadership/Data Use and sustainability measurement </li></ul>
  19. 20. PMTCT in Jamaica 2005 -2010 Source: Jarrett, 2011. 2005 2008 2009 2010 # ANC Attendees Tested 28,651 (96%) 28,659 (>95%) 30,076 (>95%) 20,259 (>95%) # HIV +ve women delivered 401 616 440 404 % of women getting ARVs 74% 84% 83% 87% # of HIV exposed infants 407 612 439 392 # Infants getting PMTCT 353 (87%) 605 (98%) 430 (98%) 383 (98%) Transmission Rate 10% <5% 4.3% Pending
  20. 22. Plans for 2012-2017 <ul><li>Status of the HIV M&E System in Jamaica – 12 components </li></ul><ul><li>Prioritized Key Actions </li></ul><ul><ul><ul><li>Update OPMAN </li></ul></ul></ul><ul><ul><ul><li>Data quality SOPs </li></ul></ul></ul><ul><ul><ul><li>Data Use/Dissemination Plan </li></ul></ul></ul><ul><ul><ul><li>Human Capacity* </li></ul></ul></ul><ul><ul><ul><li>Linking of HIV MIS & National HIS </li></ul></ul></ul><ul><li>Improved MARP surveillance and size estimation </li></ul>
  21. 23. Example: Côte d’Ivoire
  22. 24. M&E/HIS Strengthening <ul><li>2003: Assessment – joint HMIS strengthening plan </li></ul><ul><li>2004-2005: Existing HMIS forms revised to collect new HIV/AIDS data </li></ul><ul><li>March 2006: New HMIS tools finalized </li></ul><ul><li>2005-2006: Development of HMIS computer system </li></ul><ul><li>2004-2007: Training of MOH core HMIS trainers </li></ul><ul><li>2008: HMIS supervision guidelines </li></ul>
  23. 27. How do we get data in the meantime? <ul><li>2005: Active data collection for PMTCT, VCT, Care and Treatment. </li></ul><ul><ul><li>183 facilities visited </li></ul></ul><ul><ul><li>First annual national HIV/AIDS report (2005) </li></ul></ul><ul><li>Repeated 2006, 2008, 2009 </li></ul>
  24. 28. Recent Developments <ul><li>2007-2008: Comparison of PEPFAR IP data with RHIS data </li></ul><ul><ul><li>40% coverage in RHIS </li></ul></ul><ul><li>2009: ART patient monitoring electronic records (SIGDEP) </li></ul><ul><li>2011: SIGDEP used in 222 of 404 HIV care and treatment sites </li></ul><ul><li>2010: SIGDEP replaced active data collection </li></ul><ul><li>2011 SCMS assessment recommended supporting SIGDEP to manage drug stocks and dispensing </li></ul><ul><li>Repeated RDQA showed completeness of reporting increased from 75% to 81% and timeliness of reports increased from 14% to 32% in intervention districts. </li></ul>
  25. 29. <ul><li>Common Themes </li></ul>
  26. 31. Behavioral Determinants Knowledge/ skills, attitudes , values, motivation PRISM Framework for Understanding Routine Health Information System (RHIS) Performance Improved Health System Performance Improved Health Outcomes Technical Determinants Data generation architecture Information/communication technology Desired Outputs = RHIS performance • good quality information • appropriate use of information Inputs RHIS assessment, RHIS strategies RHIS interventions Organizational Determinants Information culture, health system structure, roles & responsibilities, resources
  27. 32. GHI Principles Apply <ul><li>Country ownership and sustainability </li></ul><ul><li>Health systems strengthening </li></ul><ul><li>Integration, coordination and multilateral engagement </li></ul><ul><li>Women-centered information </li></ul>
  28. 33. Common Challenges <ul><li>M&E often “last in, first out” when budgets are tight </li></ul><ul><li>Single M&E unit has to balance immediate reporting responsibilities with longer term M&E system strengthening activities </li></ul><ul><li>Indicator harmonization is key BUT is a long, intensive process involving multiple stakeholders </li></ul><ul><li>Changing indicator definitions without full consideration of data collection implications and costs </li></ul><ul><li>Human capacity and turn-over </li></ul>
  29. 34. Common Opportunities <ul><li>Significant shift in awareness and culture around M&E </li></ul><ul><li>Key system elements increasingly in place </li></ul><ul><li>Creative collaborative solutions at country level to address inherent tensions and build country systems </li></ul><ul><li>Increasing flexibility at global level and emphasis on country ownership and systems </li></ul><ul><li>Needs sustained efforts to implement now </li></ul>
  30. 35. Building M&E Systems is an Intervention <ul><li>M&E is more than indicators and reporting </li></ul><ul><ul><li>Attention to data collection, data analysis and data use </li></ul></ul><ul><li>Lack of information is not the same as lack of indicators </li></ul><ul><ul><li>Diagnose carefully </li></ul></ul><ul><li>Takes time and money to build M&E systems </li></ul><ul><ul><li>Needs systematic and sustained investment </li></ul></ul><ul><ul><li>Set strategic priorities </li></ul></ul>
  31. 37. References <ul><li>Ainsworth, M, Vaillancourt, D. and J.H. Gaubatz. 2005. Committing to Results: Improving the Effectiveness of HIV/AIDS Assistance. The World Bank, Washington DC. </li></ul><ul><li>Boerma, J.T. and S.K. Stansfield. 2007. Health statistics now: are we making the right investments? Lancet , 369:779-86. </li></ul><ul><li>Jarret, 2011. The Jamaica National HIV/STI M&E System: Past, Present & Future. MEASURE Evaluation Seminar, Chapel Hill, NC. Nov 2, 2011. </li></ul><ul><li>Macro International Inc. 2009. Final report. Global Fund Five-Year Evaluation: Study Area 3. The Impact of Collective Efforts on the Reduction of the Disease Burden of AIDS, TB, and Malaria. Macro International Inc., Calverton, MD. </li></ul><ul><li>Peersman et al. 2004. Are Investments in National HIV Monitoring and Evaluation Systems Paying Off. JAIDS 52:S87-S96. </li></ul>
  32. 38. Acknowledgements <ul><li>Stephanie Watson-Grant, Country Portfolio Manager and Jamaica Focal Person, MEASURE Evaluation. </li></ul><ul><li>Leontine Gnassou, Country Resident Advisor, Côte d’Ivoire, MEASURE Evaluation. </li></ul><ul><li>Alimou Barry, Country Focal Person, Côte d’Ivoire, MEASURE Evaluation. </li></ul><ul><li>Tariq Azim, Country Resident Advisor, Ethiopia, MEASURE Evaluation (several photos) </li></ul>
  33. 39. MEASURE Evaluation is funded by the U.S. Agency for International Development through Cooperative Agreement GHA-A-00-08-00003-00 and is implemented by the Carolina Population Center at the University of North Carolina at Chapel Hill, in partnership with Futures Group, ICF Macro, John Snow, Inc., Management Sciences for Health, and Tulane University. The views expressed in this presentation do not necessarily reflect the views of USAID or the United States government. Visit us online at