Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Lessons Learned In Using the Most Significant Change Technique in Evaluation


Published on

This presentation was part of a Data for Impact (D4I) webinar held in January 2020.

Published in: Health & Medicine
  • Be the first to comment

  • Be the first to like this

Lessons Learned In Using the Most Significant Change Technique in Evaluation

  1. 1. Lessons Learned In Using the Most Significant Change Technique in Evaluation Mary Freyder, MPH, LMSW Bridgit Adamou, MPH January 8, 2020
  2. 2. • Data for Impact (D4I) overview • Sharing lessons from the field – Most Significant Change (MSC) in five countries under MEASURE Evaluation Project • Deep dive - evaluating the Local Capacity Initiative (LCI) in Uganda • Questions and answers Presentation Overview
  3. 3. • Generate evidence • Strengthen capacity • Ensure data quality • Integrate gender • Promote data use • Learn D4I works to:
  4. 4. • Participatory approach • Participants give personal stories of significant change, directly or indirectly related to intervention • Stories typically analyzed and filtered through various organization levels Introduction to MSC
  5. 5. MSC process in three steps 1. Decide the types of stories to be collected 2. Gather stories from participants 3. Share stories with all stakeholders to learn
  6. 6. • A program adapts to different or changing contexts, leading to differences in implementation and outcomes • Cause and effect are poorly understood MSC: Useful when… • Multiple intervention components exist • Evaluation is focused on learning, not just accountability
  7. 7. • Flexible, captures broad range of results • Addresses limitations of other, more traditional methods • Simple process MSC: Strengths • Goal-free, no predetermined outcome
  8. 8. • Bias toward success stories • Subjectivity in story selection • Bias toward popular views • Bias toward views of those who are good storytellers MSC: Limitations
  9. 9. MEASURE Evaluation’s applications of MSC • Feed the Future’s Integrating Nutrition in Value Chains (INVC) program in Malawi • M&E systems strengthening in Côte d’Ivoire and Nigeria • USAID’s Gender, Policy, and Measurement Program, in partnership with the Suaahara Project and the Government of Nepal • USAID’s Sustainable Comprehensive Responses (SCORE) Project in Uganda • PEPFAR’s Local Capacity Initiative (LCI) in Uganda and Ghana
  10. 10. • Part of the evaluation of Feed the Future’s INVC program • Nutrition promoters trained on MSC in two districts • 26 volunteer nutrition promoters collected 277 stories from project beneficiary households RESULTS: • Stories were short, lacked details, and were repetitive • Many of the stories covered multiple domains • MSC was useful for understanding participants’ perceptions of the key benefits of INVC Application of MSC in Malawi
  11. 11. • In both countries, researchers sought to determine the most significant changes experienced by stakeholders in the improvement of each country’s national HIV M&E system. • Using a facilitator well-trained in MSC, a stakeholder workshop was held in each of the countries to identify stories. • Groups used an adapted self-assessment tool to guide discussion, agree on a response, and then provide sources of evidence for the responses. • Researchers validated the stories and conducted key informant interviews to obtain additional details. Application of MSC in Côte d’Ivoire and Nigeria
  12. 12. RESULTS: • Engaging stakeholders in the MSC method helped them identify important changes resulting from interventions to strengthen M&E systems. Application of MSC in Côte d’Ivoire and Nigeria • Researchers clearly heard stakeholders’ voices and could comprehensively and systematically quantify improvements in each of the 12 components of M&E systems strengthening.
  13. 13. • Impact evaluation was conducted to understand the contributions of two approaches to strengthening the capacity of Health Facility Operation & Management Committees (HFOMCs). • MSC method was used for focus group discussions with HFOMC members and community members to see if significant changes had occurred in their lives. • A local research organization collected stories from community members and presented them to the HFOMC members for group discussion. Application of MSC in Nepal
  14. 14. Application of MSC in Nepal RESULTS: • Using the MSC approach with community members was challenging because the intervention was implemented by the same organization that implemented another program and community members confused the two. • HFOMC members shared useful stories about how the project helped them obtain information.
  15. 15. • The SCORE project aimed to benefit orphans and vulnerable children and their caregivers. • The program’s aims were to build economic resilience, enhance food security, improve child protection, and increase access to education and critical services. • Forty randomly selected beneficiaries and project staff were asked about positive and negative changes resulting from the program. Application of MSC in Uganda
  16. 16. RESULTS: • Because of time and resource constraints, the MSC framework was used to develop open-ended questions for the participants. • The MSC method generated rich and interesting responses. Application of MSC in Uganda • Analyzing the answers revealed that the layered, multifaceted components of the intervention led to positive change, particularly at the family level.
  17. 17. • The MSC approach is time-consuming and requires careful facilitation. • People collecting stories should be well trained and supervised. • The MSC approach is easy for participants to understand. • It is worthwhile to allow ample time for the interviews—even if this means gathering fewer stories. Lessons learned across MSC studies
  18. 18. • Follow-up interviews with beneficiaries, program staff, or donors can further strengthen the learning aspect of this approach and complement MSC data. Lessons learned across MSC studies • The MSC technique is useful when evaluators must narrow down the components of participant observations of what changed. • This method tends to have a bias for positive responses.
  19. 19. • Research brief: Experiences and Lessons Learned: Implementing the Most Significant Change Method • Report: Integrating Nutrition in Value Chains in Malawi: Using Most Significant Change stories to understand community experiences • Case study: A case study to measure national HIV M&E systems strengthening: Nigeria • Case study: A case study to measure national HIV M&E systems strengthening: Côte d’Ivoire • Report: Uganda’s SCORE program for vulnerable children and their families: Mixed-methods performance evaluation D4I resources on the MSC technique
  20. 20. This presentation was produced with the support of the United States Agency for International Development (USAID) under the terms of the Data for Impact (D4I) associate award 7200AA18LA00008, which is implemented by the Carolina Population Center at the University of North Carolina at Chapel Hill, in partnership with Palladium International, LLC; ICF Macro, Inc.; John Snow, Inc.; and Tulane University. The views expressed in this publication do not necessarily reflect the views of USAID or the United States government.