More Related Content
Similar to Lessons Learned In Using the Most Significant Change Technique in Evaluation
More from removed_62798267384a091db5c693ad7f1cc5ac
Lessons Learned In Using the Most Significant Change Technique in Evaluation
- 1. Lessons Learned InUsing the Most Significant Change Technique in Evaluation Mary Freyder, MPH, LMSW Bridgit Adamou, MPH January 8, 2020
- 2. • Data forImpact (D4I) overview • Sharing lessons from the field – Most Significant Change (MSC) in five countries under MEASURE Evaluation Project • Deep dive - evaluating the Local Capacity Initiative (LCI) in Uganda • Questions and answers Presentation Overview
- 3. • Generate evidence •Strengthen capacity • Ensure data quality • Integrate gender • Promote data use • Learn D4I works to:
- 4. • Participatory approach •Participants give personal stories of significant change, directly or indirectly related to intervention • Stories typically analyzed and filtered through various organization levels Introduction to MSC
- 5. MSC process inthree steps 1. Decide the types of stories to be collected 2. Gather stories from participants 3. Share stories with all stakeholders to learn
- 6. • A programadapts to different or changing contexts, leading to differences in implementation and outcomes • Cause and effect are poorly understood MSC: Useful when… • Multiple intervention components exist • Evaluation is focused on learning, not just accountability
- 7. • Flexible, capturesbroad range of results • Addresses limitations of other, more traditional methods • Simple process MSC: Strengths • Goal-free, no predetermined outcome
- 8. • Bias towardsuccess stories • Subjectivity in story selection • Bias toward popular views • Bias toward views of those who are good storytellers MSC: Limitations
- 9. MEASURE Evaluation’s applicationsof MSC • Feed the Future’s Integrating Nutrition in Value Chains (INVC) program in Malawi • M&E systems strengthening in Côte d’Ivoire and Nigeria • USAID’s Gender, Policy, and Measurement Program, in partnership with the Suaahara Project and the Government of Nepal • USAID’s Sustainable Comprehensive Responses (SCORE) Project in Uganda • PEPFAR’s Local Capacity Initiative (LCI) in Uganda and Ghana
- 10. • Part ofthe evaluation of Feed the Future’s INVC program • Nutrition promoters trained on MSC in two districts • 26 volunteer nutrition promoters collected 277 stories from project beneficiary households RESULTS: • Stories were short, lacked details, and were repetitive • Many of the stories covered multiple domains • MSC was useful for understanding participants’ perceptions of the key benefits of INVC Application of MSC in Malawi
- 11. • In bothcountries, researchers sought to determine the most significant changes experienced by stakeholders in the improvement of each country’s national HIV M&E system. • Using a facilitator well-trained in MSC, a stakeholder workshop was held in each of the countries to identify stories. • Groups used an adapted self-assessment tool to guide discussion, agree on a response, and then provide sources of evidence for the responses. • Researchers validated the stories and conducted key informant interviews to obtain additional details. Application of MSC in Côte d’Ivoire and Nigeria
- 12. RESULTS: • Engaging stakeholdersin the MSC method helped them identify important changes resulting from interventions to strengthen M&E systems. Application of MSC in Côte d’Ivoire and Nigeria • Researchers clearly heard stakeholders’ voices and could comprehensively and systematically quantify improvements in each of the 12 components of M&E systems strengthening.
- 13. • Impact evaluationwas conducted to understand the contributions of two approaches to strengthening the capacity of Health Facility Operation & Management Committees (HFOMCs). • MSC method was used for focus group discussions with HFOMC members and community members to see if significant changes had occurred in their lives. • A local research organization collected stories from community members and presented them to the HFOMC members for group discussion. Application of MSC in Nepal
- 14. Application of MSCin Nepal RESULTS: • Using the MSC approach with community members was challenging because the intervention was implemented by the same organization that implemented another program and community members confused the two. • HFOMC members shared useful stories about how the project helped them obtain information.
- 15. • The SCOREproject aimed to benefit orphans and vulnerable children and their caregivers. • The program’s aims were to build economic resilience, enhance food security, improve child protection, and increase access to education and critical services. • Forty randomly selected beneficiaries and project staff were asked about positive and negative changes resulting from the program. Application of MSC in Uganda
- 16. RESULTS: • Because oftime and resource constraints, the MSC framework was used to develop open-ended questions for the participants. • The MSC method generated rich and interesting responses. Application of MSC in Uganda • Analyzing the answers revealed that the layered, multifaceted components of the intervention led to positive change, particularly at the family level.
- 17. • The MSCapproach is time-consuming and requires careful facilitation. • People collecting stories should be well trained and supervised. • The MSC approach is easy for participants to understand. • It is worthwhile to allow ample time for the interviews—even if this means gathering fewer stories. Lessons learned across MSC studies
- 18. • Follow-up interviewswith beneficiaries, program staff, or donors can further strengthen the learning aspect of this approach and complement MSC data. Lessons learned across MSC studies • The MSC technique is useful when evaluators must narrow down the components of participant observations of what changed. • This method tends to have a bias for positive responses.
- 19. • Research brief:Experiences and Lessons Learned: Implementing the Most Significant Change Method • Report: Integrating Nutrition in Value Chains in Malawi: Using Most Significant Change stories to understand community experiences • Case study: A case study to measure national HIV M&E systems strengthening: Nigeria • Case study: A case study to measure national HIV M&E systems strengthening: Côte d’Ivoire • Report: Uganda’s SCORE program for vulnerable children and their families: Mixed-methods performance evaluation D4I resources on the MSC technique
- 20. This presentation wasproduced with the support of the United States Agency for International Development (USAID) under the terms of the Data for Impact (D4I) associate award 7200AA18LA00008, which is implemented by the Carolina Population Center at the University of North Carolina at Chapel Hill, in partnership with Palladium International, LLC; ICF Macro, Inc.; John Snow, Inc.; and Tulane University. The views expressed in this publication do not necessarily reflect the views of USAID or the United States government. www.data4impactproject.org
Editor's Notes
- #4 D4I, which is a MEASURE Evaluation associate award and is funded by United States Agency for International Development, works with countries to: Generate strong evidence for program and policy decision making Build individual and organizational capacity Enhance the use of data for global health programs and policies
- #5 MEASURE Evaluation has implemented the MSC technique in evaluations of health programs. This presentation summarizes our experience and shares lessons learned in the application of MSC. But first, what is MSC? It is a participatory qualitative approach to monitoring and evaluation that involves assessing the changes and impacts as the result of a program from the perspective of program participants or beneficiaries. It is participatory because program participants themselves and stakeholders are usually involved in deciding types of change to explore, and in analyzing the data, or stories, collected. Project staff collect stories from beneficiaries, partners, or participants. Story collectors ask questions such as the following: “During the last period, in your opinion, what was the most significant change that took place for participants in the project?” Respondents describe both the change and the reasons they consider it significant. The stories are then analyzed and filtered through different levels. “Typically” is in italics here because MSC has been implemented in many different ways, ranging from the original version of it, with multiple levels of story review/analysis and voting, to just asking individuals for their stories and researchers analysing those. You will see this illustrated through the MEASURE Evaluation examples coming up.
- #6 The process entails three basic steps: 1. Deciding the types of stories that should be collected from beneficiaries, partners, or participants 2. Gathering stories by asking respondents to describe both the change they experienced and the reasons they consider it significant, then collectively determining which stories are the most significant 3. Sharing the stories with stakeholders and contributors to learn about what is valued
- #7 MSC is useful when you are evaluating a program that adapts to different or evolving contexts, leading to differences in implementation and outcomes. When cause and effect is not understood well. When there are multiple program components And when your evaluation is learning focused rather than solely focused on accountability.
- #8 MSC is flexible: open to broader range of results, including those that may not have originally been intended by the program. Complementary to more traditional methods: can help address limitations of other methods. For example, if you are using a structured survey only, you may miss unanticipated changes, negative or positive. Simple, straightforward process that anybody can understand and use. Compared to other monitoring approaches, it is easy to communicate across cultures. There aren’t any indicators, so there is no need to explain what an indicator is. Everyone can tell stories about events they think were important. Goal-free: no assumption of what success should look like. Because it is not a linear model, there are no predetermined outcomes.
- #9 This method is biased towards successes – particularly to make the storyteller look good. Evaluators can overcome this by having a domain of change for negative stories. There is subjectivity in story selection. Recording the reasons for selecting stories helps strengthen the process There are biases toward popular views. And there are also biases toward views of those who are good storytellers. This is why don’t use as stand-alone tool.
- #10 MEASURE Evaluation applied most significant change in five different settings to monitor and evaluate various community based or complex programs. One was a nutrition program in Malawi. Another looked at how national government HIV M&E systems in Cote d’Ivoire and Nigeria were strengthened. In Nepal, MSC was used as part of an impact evaluation involving health facility operation and management committees. The Sustainable Comprehensive Responses, or SCORE, program in Uganda aimed to benefit orphans and vulnerable children and their caregivers. And then lastly there was the PEPFAR-supported global initiative to support local partners in HIV policy advocacy for highly vulnerable populations such as men who have sex with men, transgender women and sex workers. My colleague, Mary, will discuss this application more in-depth.
- #11 MSC was intended to be one aspect of qualitative data collection for a large impact evaluation of Feed the Future’s Integrating Nutrition in Value Chains program. Volunteer nutrition promoters were trained on MSC in Malawi’s Mchinji and Lilongwe districts to obtain information from participants about their experiences with the INVC program. Over the course of a year, 26 nutrition promoters, who were also program implementers, collected 277 MSC stories from project beneficiary households. Outcomes: Because the nutrition promoters wrote the stories in their notebooks, summarizing them rather than writing them verbatim, the stories tended to be very short, had few details, and were repetitive. The similarities among stories made it difficult for the promoters to discuss, rank, and select the most significant stories. Many of the stories covered more than one domain of change, which also made it challenging for promoters to assign the stories to the domains. The MSC method was useful for understanding participants’ perceptions of the key benefits of and challenges to the INVC program. (Due to the cancellation of the impact evaluation, no other qualitative data was collected to supplement the MSC stories.)
- #12 In two case studies, researchers sought to produce evidence of how Côte d’Ivoire and Nigeria’s national HIV M&E systems were strengthened. The MSC technique was applied in both countries to determine the MSCs experienced by stakeholders in the improvement of each country’s national HIV M&E system. With the support of a facilitator well-trained in MSC, a stakeholder workshop was held in each of the countries to identify stories. Participants started by self-selecting into groups. Next, each group used an adapted self-assessment tool to guide discussion, agree on a response, and then provide sources of evidence for the responses. Groups reported to the plenary to verify each other’s findings. Afterwards the researchers validated the stories and conducted key informant interviews to obtain additional details and perspectives on the MSCs that were identified in the workshop.
- #13 Researchers found that engaging stakeholders in the MSC method helped them identify the important changes resulting from interventions to strengthen M&E systems, which also helped to raise interest in these interventions and the defined domains of change. Using the MSC method also meant that researchers clearly heard stakeholders’ voices and could comprehensively and systematically quantify improvements or successes in each of the 12 components of M&E systems strengthening.
- #14 MEASURE Evaluation conducted an impact evaluation to understand the contributions of two approaches to strengthening the capacity of the Health Facility Operation and Management Committees (HFOMCs) in Nepal. At the end line of the evaluation, we used the MSC method to conduct eight focus group discussions with HFOMC members and community members to see if significant changes had occurred in their lives as a direct result of their participation in the program or as a result of the program’s interventions overall. The plan was to apply the MSC method in two ways: (1) HFOMC members were to engage in focus group discussions about changes they experienced through program participation and (2) HFOMC members were to collect stories of change from the community for the HFOMC to discuss and vote on. The latter story collection did not happen as planned, however, because of HFOMC members’ limited time and limited skills in facilitating discussions, among other reasons. Instead, a local research organization collected the stories from community members and presented them to the HFOMC members for group discussion.
- #15 Using the MSC approach with community members proved challenging because the intervention was implemented by the same organization that implemented another program and community members confused the two. However, HFOMC members shared useful stories about how the project helped them obtain information.
- #16 USAID’s Sustainable, Comprehensive Responses (SCORE) project operated in Uganda to benefit orphans and vulnerable children and their caregivers. The program’s aims were to build economic resilience, enhance food security, improve child protection, and increase access to education and critical services. MEASURE Evaluation used the MSC method to assess the project’s effects on beneficiaries and the strengths and challenges associated with implementation. Using the MSC techniquie, 40 randomly selected beneficiaries and project staff about positive and negative changes resulting from the program.
- #17 Time and resource constraints prevented respondents from gathering to discuss and vote on the changes that had the most impact. Instead, we used the MSC framework to develop open-ended questions for the participants, which generated rich and interesting responses. Analysis of these answers revealed that the layered, multifaceted components of the intervention led to positive change, particularly at the family level.
- #18 Before we dive deep into the LCI Uganda evaluation, I’ll share a few lessons learned across all four studies. Mary will highlight a few of these in her presentation. The MSC approach is time-consuming and requires careful facilitation. If people capturing stories lack sufficient time and skills, it is better to select others better suited to use it correctly, or plan to spend significant time training data collectors to record stories—or avoid the method altogether. People collecting stories should be well trained and supervised so the stories are thoroughly captured with detailed notes. The MSC approach is easy for participants to understand. It is worthwhile to allow ample time for the interviews—even if this means gathering fewer stories. It’s important that data collectors can fully explain the questions and that participants have enough time to think about their experiences and articulate their responses.
- #19 Follow-up interviews with beneficiaries, program staff, or donors can further strengthen the learning aspect of this approach and complement the MSC data. The MSC technique is useful when evaluators must systematically narrow down the components of participant observations of what changed. As mentioned earlier, this method tends to have a bias for positive responses. Specifically asking a question about most significant negative change can address this potential drawback.