SlideShare a Scribd company logo

Lessons Learned In Using the Most Significant Change Technique in Evaluation

This presentation was part of a Data for Impact (D4I) webinar held in January 2020.

1 of 20
Lessons Learned In Using the Most
Significant Change Technique in Evaluation
Mary Freyder, MPH, LMSW
Bridgit Adamou, MPH
January 8, 2020
• Data for Impact (D4I) overview
• Sharing lessons from the field – Most Significant
Change (MSC) in five countries under MEASURE
Evaluation Project
• Deep dive - evaluating the Local Capacity Initiative
(LCI) in Uganda
• Questions and answers
Presentation Overview
• Generate evidence
• Strengthen capacity
• Ensure data quality
• Integrate gender
• Promote data use
• Learn
D4I works to:
• Participatory approach
• Participants give personal stories of significant
change, directly or indirectly related to intervention
• Stories typically analyzed and filtered through
various organization levels
Introduction to MSC
MSC process in three steps
1. Decide the types of stories to be collected
2. Gather stories from participants
3. Share stories with all stakeholders to learn
• A program adapts to different or changing
contexts, leading to differences in implementation
and outcomes
• Cause and effect are poorly
understood
MSC: Useful when…
• Multiple intervention
components exist
• Evaluation is focused on
learning, not just
accountability
Ad

Recommended

Using Most Significant Change in a Mixed-Methods Evaluation in Uganda
Using Most Significant Change in a Mixed-Methods Evaluation in UgandaUsing Most Significant Change in a Mixed-Methods Evaluation in Uganda
Using Most Significant Change in a Mixed-Methods Evaluation in UgandaMEASURE Evaluation
 
Managing missing values in routinely reported data: One approach from the Dem...
Managing missing values in routinely reported data: One approach from the Dem...Managing missing values in routinely reported data: One approach from the Dem...
Managing missing values in routinely reported data: One approach from the Dem...MEASURE Evaluation
 
How to improve the capabilities of health information systems to address emer...
How to improve the capabilities of health information systems to address emer...How to improve the capabilities of health information systems to address emer...
How to improve the capabilities of health information systems to address emer...MEASURE Evaluation
 
Routine data use in evaluation: practical guidance
Routine data use in evaluation: practical guidanceRoutine data use in evaluation: practical guidance
Routine data use in evaluation: practical guidanceMEASURE Evaluation
 
Monitoring evaluation
Monitoring evaluationMonitoring evaluation
Monitoring evaluationCarlo Magno
 
Capacity Development For Monitoring And Evaluation
Capacity Development For Monitoring And EvaluationCapacity Development For Monitoring And Evaluation
Capacity Development For Monitoring And EvaluationKnowledge Management Center
 
Data for Impact: Lessons Learned in Using the Ripple Effects Mapping Method
Data for Impact: Lessons Learned in Using the Ripple Effects Mapping MethodData for Impact: Lessons Learned in Using the Ripple Effects Mapping Method
Data for Impact: Lessons Learned in Using the Ripple Effects Mapping MethodMEASURE Evaluation
 

More Related Content

What's hot

Lessons learned in using process tracing for evaluation
Lessons learned in using process tracing for evaluationLessons learned in using process tracing for evaluation
Lessons learned in using process tracing for evaluationMEASURE Evaluation
 
Monitoring and evaluation
Monitoring and evaluationMonitoring and evaluation
Monitoring and evaluationMd Rifat Anam
 
Data management tools for monitoring, evaluation and learning—Options and ideas
Data management tools for monitoring, evaluation and learning—Options and ideasData management tools for monitoring, evaluation and learning—Options and ideas
Data management tools for monitoring, evaluation and learning—Options and ideasILRI
 
Data Demand and Use : Facilitating the Use of Data to Inform Programs and Pla...
Data Demand and Use: Facilitating the Use of Data to Inform Programs and Pla...Data Demand and Use: Facilitating the Use of Data to Inform Programs and Pla...
Data Demand and Use : Facilitating the Use of Data to Inform Programs and Pla...MEASURE Evaluation
 
Data Demand and Use Training Materials
Data Demand and Use Training MaterialsData Demand and Use Training Materials
Data Demand and Use Training MaterialsMEASURE Evaluation
 
Data Visualization for Decision Making in HIV Programs
Data Visualization for Decision Making in HIV ProgramsData Visualization for Decision Making in HIV Programs
Data Visualization for Decision Making in HIV ProgramsMEASURE Evaluation
 
Monitoring and Evaluation Framework
Monitoring and Evaluation FrameworkMonitoring and Evaluation Framework
Monitoring and Evaluation FrameworkMichelle Joja
 
Basics of Monitoring & Evaluation
Basics of Monitoring & EvaluationBasics of Monitoring & Evaluation
Basics of Monitoring & EvaluationFida Karim 🇵🇰
 
Monitoring And Evaluation Presentation
Monitoring And Evaluation PresentationMonitoring And Evaluation Presentation
Monitoring And Evaluation PresentationEquiGov Institute
 
Framework for Linking Data with Action
Framework for Linking Data with ActionFramework for Linking Data with Action
Framework for Linking Data with ActionMEASURE Evaluation
 
Assessment of Constraints to Data Use
Assessment of Constraints to Data UseAssessment of Constraints to Data Use
Assessment of Constraints to Data UseMEASURE Evaluation
 
PCM - Project Cycle Management, Training on Evaluation
PCM - Project Cycle Management, Training on EvaluationPCM - Project Cycle Management, Training on Evaluation
PCM - Project Cycle Management, Training on Evaluationrexcris
 
Monitoring and evaluation (Part 1)
Monitoring and evaluation (Part 1)Monitoring and evaluation (Part 1)
Monitoring and evaluation (Part 1)Vilimaka Foliaki
 
Monitoring & evaluation presentation[1]
Monitoring & evaluation presentation[1]Monitoring & evaluation presentation[1]
Monitoring & evaluation presentation[1]skzarif
 
On Medicaid claims data analysis
On Medicaid claims data analysisOn Medicaid claims data analysis
On Medicaid claims data analysisElspeth Slayter
 
Importance of M&E
Importance of M&EImportance of M&E
Importance of M&Eclearsateam
 

What's hot (20)

Lessons learned in using process tracing for evaluation
Lessons learned in using process tracing for evaluationLessons learned in using process tracing for evaluation
Lessons learned in using process tracing for evaluation
 
Monitoring and evaluation
Monitoring and evaluationMonitoring and evaluation
Monitoring and evaluation
 
Data management tools for monitoring, evaluation and learning—Options and ideas
Data management tools for monitoring, evaluation and learning—Options and ideasData management tools for monitoring, evaluation and learning—Options and ideas
Data management tools for monitoring, evaluation and learning—Options and ideas
 
Data Demand and Use : Facilitating the Use of Data to Inform Programs and Pla...
Data Demand and Use: Facilitating the Use of Data to Inform Programs and Pla...Data Demand and Use: Facilitating the Use of Data to Inform Programs and Pla...
Data Demand and Use : Facilitating the Use of Data to Inform Programs and Pla...
 
Data Demand and Use Training Materials
Data Demand and Use Training MaterialsData Demand and Use Training Materials
Data Demand and Use Training Materials
 
Data Visualization for Decision Making in HIV Programs
Data Visualization for Decision Making in HIV ProgramsData Visualization for Decision Making in HIV Programs
Data Visualization for Decision Making in HIV Programs
 
Monitoring and Evaluation Framework
Monitoring and Evaluation FrameworkMonitoring and Evaluation Framework
Monitoring and Evaluation Framework
 
Basics of Monitoring & Evaluation
Basics of Monitoring & EvaluationBasics of Monitoring & Evaluation
Basics of Monitoring & Evaluation
 
Monitoring And Evaluation Presentation
Monitoring And Evaluation PresentationMonitoring And Evaluation Presentation
Monitoring And Evaluation Presentation
 
Monitoring and Evaluation
Monitoring and EvaluationMonitoring and Evaluation
Monitoring and Evaluation
 
M&E Plan
M&E PlanM&E Plan
M&E Plan
 
Framework for Linking Data with Action
Framework for Linking Data with ActionFramework for Linking Data with Action
Framework for Linking Data with Action
 
Assessment of Constraints to Data Use
Assessment of Constraints to Data UseAssessment of Constraints to Data Use
Assessment of Constraints to Data Use
 
M & E Fundamentals.
M & E Fundamentals.M & E Fundamentals.
M & E Fundamentals.
 
PCM - Project Cycle Management, Training on Evaluation
PCM - Project Cycle Management, Training on EvaluationPCM - Project Cycle Management, Training on Evaluation
PCM - Project Cycle Management, Training on Evaluation
 
Monitoring and evaluation (Part 1)
Monitoring and evaluation (Part 1)Monitoring and evaluation (Part 1)
Monitoring and evaluation (Part 1)
 
Monitoring & evaluation presentation[1]
Monitoring & evaluation presentation[1]Monitoring & evaluation presentation[1]
Monitoring & evaluation presentation[1]
 
On Medicaid claims data analysis
On Medicaid claims data analysisOn Medicaid claims data analysis
On Medicaid claims data analysis
 
Data Demand and Use Workshop
Data Demand and Use WorkshopData Demand and Use Workshop
Data Demand and Use Workshop
 
Importance of M&E
Importance of M&EImportance of M&E
Importance of M&E
 

Similar to Lessons Learned In Using the Most Significant Change Technique in Evaluation

Evaluating a community media approach to promote nutrition in Niger
Evaluating a community media approach to promote nutrition in NigerEvaluating a community media approach to promote nutrition in Niger
Evaluating a community media approach to promote nutrition in NigerJSI
 
Sustaining quality approaches for locally embedded community health services ...
Sustaining quality approaches for locally embedded community health services ...Sustaining quality approaches for locally embedded community health services ...
Sustaining quality approaches for locally embedded community health services ...REACHOUTCONSORTIUMSLIDES
 
Quality improvement and Community Health Worker performance: A mixed method r...
Quality improvement and Community Health Worker performance: A mixed method r...Quality improvement and Community Health Worker performance: A mixed method r...
Quality improvement and Community Health Worker performance: A mixed method r...REACHOUTCONSORTIUMSLIDES
 
Improving Agricultural Productivity through Training; a Case of Heifer Projec...
Improving Agricultural Productivity through Training; a Case of Heifer Projec...Improving Agricultural Productivity through Training; a Case of Heifer Projec...
Improving Agricultural Productivity through Training; a Case of Heifer Projec...Waswa Balunywa
 
Day 2 panel 4 quality improvement for mnh tz 108020
Day 2 panel 4 quality improvement for mnh tz 108020Day 2 panel 4 quality improvement for mnh tz 108020
Day 2 panel 4 quality improvement for mnh tz 108020ea-imcha
 
ICCM_Sara Riese_10.16.13
ICCM_Sara Riese_10.16.13ICCM_Sara Riese_10.16.13
ICCM_Sara Riese_10.16.13CORE Group
 
Transitioning from reach every district to reach every community
Transitioning from reach every district to reach every communityTransitioning from reach every district to reach every community
Transitioning from reach every district to reach every communityJSI
 
Two Examples of Program Planning, Monitoring and Evaluation
Two Examples of Program Planning, Monitoring and EvaluationTwo Examples of Program Planning, Monitoring and Evaluation
Two Examples of Program Planning, Monitoring and EvaluationMEASURE Evaluation
 
Implementation of the British framework and tools for assessing the needs of ...
Implementation of the British framework and tools for assessing the needs of ...Implementation of the British framework and tools for assessing the needs of ...
Implementation of the British framework and tools for assessing the needs of ...BASPCAN
 
Cies 2015 global literacy & going to scale sc presentation 11march2015
Cies 2015 global literacy & going to scale sc presentation 11march2015Cies 2015 global literacy & going to scale sc presentation 11march2015
Cies 2015 global literacy & going to scale sc presentation 11march2015Cecilia Ochoa
 
Big lottery fund evidence briefing
Big lottery fund evidence briefing Big lottery fund evidence briefing
Big lottery fund evidence briefing Ecorys
 
Day 2 panel 4 replicating mama toto tz 108024
Day 2 panel 4 replicating mama toto tz 108024Day 2 panel 4 replicating mama toto tz 108024
Day 2 panel 4 replicating mama toto tz 108024ea-imcha
 
Using case-based methods to assess scalability and sustainability: Lessons fr...
Using case-based methods to assess scalability and sustainability: Lessons fr...Using case-based methods to assess scalability and sustainability: Lessons fr...
Using case-based methods to assess scalability and sustainability: Lessons fr...Barb Knittel
 
Measuring National M&E System Strengthening in Nigeria: Application of the Mo...
Measuring National M&E System Strengthening in Nigeria: Application of the Mo...Measuring National M&E System Strengthening in Nigeria: Application of the Mo...
Measuring National M&E System Strengthening in Nigeria: Application of the Mo...MEASURE Evaluation
 
Pathway to Practice: Incorporating Evidence into Military Family Services
Pathway to Practice: Incorporating Evidence into Military Family ServicesPathway to Practice: Incorporating Evidence into Military Family Services
Pathway to Practice: Incorporating Evidence into Military Family ServicesMFLNFamilyDevelopmnt
 
Day 2 panel 4 improving standards based management mw 108031
Day 2 panel 4 improving standards based management mw 108031Day 2 panel 4 improving standards based management mw 108031
Day 2 panel 4 improving standards based management mw 108031ea-imcha
 
The Current State of Play of Community Health Workers Training Programs in Su...
The Current State of Play of Community Health Workers Training Programs in Su...The Current State of Play of Community Health Workers Training Programs in Su...
The Current State of Play of Community Health Workers Training Programs in Su...germainsky
 
How to Select an Evidence-Based Teen Pregnancy Prevention Program that Works ...
How to Select an Evidence-Based Teen Pregnancy Prevention Program that Works ...How to Select an Evidence-Based Teen Pregnancy Prevention Program that Works ...
How to Select an Evidence-Based Teen Pregnancy Prevention Program that Works ...Office of Adolescent Health
 
Developing the India smallholder dairy value chain impact pathway(s)
Developing the India smallholder dairy value chain impact pathway(s) Developing the India smallholder dairy value chain impact pathway(s)
Developing the India smallholder dairy value chain impact pathway(s) ILRI
 

Similar to Lessons Learned In Using the Most Significant Change Technique in Evaluation (20)

Evaluating a community media approach to promote nutrition in Niger
Evaluating a community media approach to promote nutrition in NigerEvaluating a community media approach to promote nutrition in Niger
Evaluating a community media approach to promote nutrition in Niger
 
Sustaining quality approaches for locally embedded community health services ...
Sustaining quality approaches for locally embedded community health services ...Sustaining quality approaches for locally embedded community health services ...
Sustaining quality approaches for locally embedded community health services ...
 
Quality improvement and Community Health Worker performance: A mixed method r...
Quality improvement and Community Health Worker performance: A mixed method r...Quality improvement and Community Health Worker performance: A mixed method r...
Quality improvement and Community Health Worker performance: A mixed method r...
 
Improving Agricultural Productivity through Training; a Case of Heifer Projec...
Improving Agricultural Productivity through Training; a Case of Heifer Projec...Improving Agricultural Productivity through Training; a Case of Heifer Projec...
Improving Agricultural Productivity through Training; a Case of Heifer Projec...
 
Day 2 panel 4 quality improvement for mnh tz 108020
Day 2 panel 4 quality improvement for mnh tz 108020Day 2 panel 4 quality improvement for mnh tz 108020
Day 2 panel 4 quality improvement for mnh tz 108020
 
ICCM_Sara Riese_10.16.13
ICCM_Sara Riese_10.16.13ICCM_Sara Riese_10.16.13
ICCM_Sara Riese_10.16.13
 
Transitioning from reach every district to reach every community
Transitioning from reach every district to reach every communityTransitioning from reach every district to reach every community
Transitioning from reach every district to reach every community
 
Two Examples of Program Planning, Monitoring and Evaluation
Two Examples of Program Planning, Monitoring and EvaluationTwo Examples of Program Planning, Monitoring and Evaluation
Two Examples of Program Planning, Monitoring and Evaluation
 
Implementation of the British framework and tools for assessing the needs of ...
Implementation of the British framework and tools for assessing the needs of ...Implementation of the British framework and tools for assessing the needs of ...
Implementation of the British framework and tools for assessing the needs of ...
 
Cies 2015 global literacy & going to scale sc presentation 11march2015
Cies 2015 global literacy & going to scale sc presentation 11march2015Cies 2015 global literacy & going to scale sc presentation 11march2015
Cies 2015 global literacy & going to scale sc presentation 11march2015
 
Big lottery fund evidence briefing
Big lottery fund evidence briefing Big lottery fund evidence briefing
Big lottery fund evidence briefing
 
Day 2 panel 4 replicating mama toto tz 108024
Day 2 panel 4 replicating mama toto tz 108024Day 2 panel 4 replicating mama toto tz 108024
Day 2 panel 4 replicating mama toto tz 108024
 
Using case-based methods to assess scalability and sustainability: Lessons fr...
Using case-based methods to assess scalability and sustainability: Lessons fr...Using case-based methods to assess scalability and sustainability: Lessons fr...
Using case-based methods to assess scalability and sustainability: Lessons fr...
 
Measuring National M&E System Strengthening in Nigeria: Application of the Mo...
Measuring National M&E System Strengthening in Nigeria: Application of the Mo...Measuring National M&E System Strengthening in Nigeria: Application of the Mo...
Measuring National M&E System Strengthening in Nigeria: Application of the Mo...
 
Pathway to Practice: Incorporating Evidence into Military Family Services
Pathway to Practice: Incorporating Evidence into Military Family ServicesPathway to Practice: Incorporating Evidence into Military Family Services
Pathway to Practice: Incorporating Evidence into Military Family Services
 
Day 2 panel 4 improving standards based management mw 108031
Day 2 panel 4 improving standards based management mw 108031Day 2 panel 4 improving standards based management mw 108031
Day 2 panel 4 improving standards based management mw 108031
 
The Current State of Play of Community Health Workers Training Programs in Su...
The Current State of Play of Community Health Workers Training Programs in Su...The Current State of Play of Community Health Workers Training Programs in Su...
The Current State of Play of Community Health Workers Training Programs in Su...
 
How to Select an Evidence-Based Teen Pregnancy Prevention Program that Works ...
How to Select an Evidence-Based Teen Pregnancy Prevention Program that Works ...How to Select an Evidence-Based Teen Pregnancy Prevention Program that Works ...
How to Select an Evidence-Based Teen Pregnancy Prevention Program that Works ...
 
Developing the India smallholder dairy value chain impact pathway(s)
Developing the India smallholder dairy value chain impact pathway(s) Developing the India smallholder dairy value chain impact pathway(s)
Developing the India smallholder dairy value chain impact pathway(s)
 
Role of Evaluation in Decision Making
Role of Evaluation in Decision MakingRole of Evaluation in Decision Making
Role of Evaluation in Decision Making
 

More from MEASURE Evaluation

Use of Routine Data for Economic Evaluations
Use of Routine Data for Economic EvaluationsUse of Routine Data for Economic Evaluations
Use of Routine Data for Economic EvaluationsMEASURE Evaluation
 
Tuberculosis/HIV Mobility Study: Objectives and Background
Tuberculosis/HIV Mobility Study: Objectives and BackgroundTuberculosis/HIV Mobility Study: Objectives and Background
Tuberculosis/HIV Mobility Study: Objectives and BackgroundMEASURE Evaluation
 
LCI Evaluation Uganda Organizational Network Analysis
LCI Evaluation Uganda Organizational Network AnalysisLCI Evaluation Uganda Organizational Network Analysis
LCI Evaluation Uganda Organizational Network AnalysisMEASURE Evaluation
 
Using Organizational Network Analysis to Plan and Evaluate Global Health Prog...
Using Organizational Network Analysis to Plan and Evaluate Global Health Prog...Using Organizational Network Analysis to Plan and Evaluate Global Health Prog...
Using Organizational Network Analysis to Plan and Evaluate Global Health Prog...MEASURE Evaluation
 
Understanding Referral Networks for Adolescent Girls and Young Women
Understanding Referral Networks for Adolescent Girls and Young WomenUnderstanding Referral Networks for Adolescent Girls and Young Women
Understanding Referral Networks for Adolescent Girls and Young WomenMEASURE Evaluation
 
Local Capacity Initiative (LCI) Evaluation
Local Capacity Initiative (LCI) EvaluationLocal Capacity Initiative (LCI) Evaluation
Local Capacity Initiative (LCI) EvaluationMEASURE Evaluation
 
Development and Validation of a Reproductive Empowerment Scale
Development and Validation of a Reproductive Empowerment ScaleDevelopment and Validation of a Reproductive Empowerment Scale
Development and Validation of a Reproductive Empowerment ScaleMEASURE Evaluation
 
Sustaining the Impact: MEASURE Evaluation Conversation on Maternal and Child ...
Sustaining the Impact: MEASURE Evaluation Conversation on Maternal and Child ...Sustaining the Impact: MEASURE Evaluation Conversation on Maternal and Child ...
Sustaining the Impact: MEASURE Evaluation Conversation on Maternal and Child ...MEASURE Evaluation
 
Malaria Data Quality and Use in Selected Centers of Excellence in Madagascar:...
Malaria Data Quality and Use in Selected Centers of Excellence in Madagascar:...Malaria Data Quality and Use in Selected Centers of Excellence in Madagascar:...
Malaria Data Quality and Use in Selected Centers of Excellence in Madagascar:...MEASURE Evaluation
 
Evaluating National Malaria Programs’ Impact in Moderate- and Low-Transmissio...
Evaluating National Malaria Programs’ Impact in Moderate- and Low-Transmissio...Evaluating National Malaria Programs’ Impact in Moderate- and Low-Transmissio...
Evaluating National Malaria Programs’ Impact in Moderate- and Low-Transmissio...MEASURE Evaluation
 
Improved Performance of the Malaria Surveillance, Monitoring, and Evaluation ...
Improved Performance of the Malaria Surveillance, Monitoring, and Evaluation ...Improved Performance of the Malaria Surveillance, Monitoring, and Evaluation ...
Improved Performance of the Malaria Surveillance, Monitoring, and Evaluation ...MEASURE Evaluation
 
Use of Qualitative Comparative Analysis in the Assessment of the Actionable D...
Use of Qualitative Comparative Analysis in the Assessment of the Actionable D...Use of Qualitative Comparative Analysis in the Assessment of the Actionable D...
Use of Qualitative Comparative Analysis in the Assessment of the Actionable D...MEASURE Evaluation
 
Sustaining the Impact: MEASURE Evaluation Conversation on Health Informatics
Sustaining the Impact: MEASURE Evaluation Conversation on Health InformaticsSustaining the Impact: MEASURE Evaluation Conversation on Health Informatics
Sustaining the Impact: MEASURE Evaluation Conversation on Health InformaticsMEASURE Evaluation
 
7 Steps to EnGendering Evaluations of HIV programs with Adolescent Girls and ...
7 Steps to EnGendering Evaluations of HIV programs with Adolescent Girls and ...7 Steps to EnGendering Evaluations of HIV programs with Adolescent Girls and ...
7 Steps to EnGendering Evaluations of HIV programs with Adolescent Girls and ...MEASURE Evaluation
 
Sexual Orientation and Gender Identity Measures for Global Survey Research
Sexual Orientation and Gender Identity Measures for Global Survey ResearchSexual Orientation and Gender Identity Measures for Global Survey Research
Sexual Orientation and Gender Identity Measures for Global Survey ResearchMEASURE Evaluation
 
What’s Next? Practical Implementation Lessons from the Partnership for HIV-Fr...
What’s Next?Practical Implementation Lessons from the Partnership for HIV-Fr...What’s Next?Practical Implementation Lessons from the Partnership for HIV-Fr...
What’s Next? Practical Implementation Lessons from the Partnership for HIV-Fr...MEASURE Evaluation
 
Measuring Outcomes for Vulnerable Children: A Global Snapshot
Measuring Outcomes for Vulnerable Children: A Global SnapshotMeasuring Outcomes for Vulnerable Children: A Global Snapshot
Measuring Outcomes for Vulnerable Children: A Global SnapshotMEASURE Evaluation
 
Sustaining the Impact: MEASURE Evaluation Conversation on Health Systems Stre...
Sustaining the Impact: MEASURE Evaluation Conversation on Health Systems Stre...Sustaining the Impact: MEASURE Evaluation Conversation on Health Systems Stre...
Sustaining the Impact: MEASURE Evaluation Conversation on Health Systems Stre...MEASURE Evaluation
 
Les dialogues communautaires pour diffuser des résultats de recherche Example...
Les dialogues communautaires pour diffuser des résultats de recherche Example...Les dialogues communautaires pour diffuser des résultats de recherche Example...
Les dialogues communautaires pour diffuser des résultats de recherche Example...MEASURE Evaluation
 
Seven Steps to EnGendering Evaluations of Public Health Programs
 Seven Steps to EnGendering Evaluations of Public Health Programs Seven Steps to EnGendering Evaluations of Public Health Programs
Seven Steps to EnGendering Evaluations of Public Health ProgramsMEASURE Evaluation
 

More from MEASURE Evaluation (20)

Use of Routine Data for Economic Evaluations
Use of Routine Data for Economic EvaluationsUse of Routine Data for Economic Evaluations
Use of Routine Data for Economic Evaluations
 
Tuberculosis/HIV Mobility Study: Objectives and Background
Tuberculosis/HIV Mobility Study: Objectives and BackgroundTuberculosis/HIV Mobility Study: Objectives and Background
Tuberculosis/HIV Mobility Study: Objectives and Background
 
LCI Evaluation Uganda Organizational Network Analysis
LCI Evaluation Uganda Organizational Network AnalysisLCI Evaluation Uganda Organizational Network Analysis
LCI Evaluation Uganda Organizational Network Analysis
 
Using Organizational Network Analysis to Plan and Evaluate Global Health Prog...
Using Organizational Network Analysis to Plan and Evaluate Global Health Prog...Using Organizational Network Analysis to Plan and Evaluate Global Health Prog...
Using Organizational Network Analysis to Plan and Evaluate Global Health Prog...
 
Understanding Referral Networks for Adolescent Girls and Young Women
Understanding Referral Networks for Adolescent Girls and Young WomenUnderstanding Referral Networks for Adolescent Girls and Young Women
Understanding Referral Networks for Adolescent Girls and Young Women
 
Local Capacity Initiative (LCI) Evaluation
Local Capacity Initiative (LCI) EvaluationLocal Capacity Initiative (LCI) Evaluation
Local Capacity Initiative (LCI) Evaluation
 
Development and Validation of a Reproductive Empowerment Scale
Development and Validation of a Reproductive Empowerment ScaleDevelopment and Validation of a Reproductive Empowerment Scale
Development and Validation of a Reproductive Empowerment Scale
 
Sustaining the Impact: MEASURE Evaluation Conversation on Maternal and Child ...
Sustaining the Impact: MEASURE Evaluation Conversation on Maternal and Child ...Sustaining the Impact: MEASURE Evaluation Conversation on Maternal and Child ...
Sustaining the Impact: MEASURE Evaluation Conversation on Maternal and Child ...
 
Malaria Data Quality and Use in Selected Centers of Excellence in Madagascar:...
Malaria Data Quality and Use in Selected Centers of Excellence in Madagascar:...Malaria Data Quality and Use in Selected Centers of Excellence in Madagascar:...
Malaria Data Quality and Use in Selected Centers of Excellence in Madagascar:...
 
Evaluating National Malaria Programs’ Impact in Moderate- and Low-Transmissio...
Evaluating National Malaria Programs’ Impact in Moderate- and Low-Transmissio...Evaluating National Malaria Programs’ Impact in Moderate- and Low-Transmissio...
Evaluating National Malaria Programs’ Impact in Moderate- and Low-Transmissio...
 
Improved Performance of the Malaria Surveillance, Monitoring, and Evaluation ...
Improved Performance of the Malaria Surveillance, Monitoring, and Evaluation ...Improved Performance of the Malaria Surveillance, Monitoring, and Evaluation ...
Improved Performance of the Malaria Surveillance, Monitoring, and Evaluation ...
 
Use of Qualitative Comparative Analysis in the Assessment of the Actionable D...
Use of Qualitative Comparative Analysis in the Assessment of the Actionable D...Use of Qualitative Comparative Analysis in the Assessment of the Actionable D...
Use of Qualitative Comparative Analysis in the Assessment of the Actionable D...
 
Sustaining the Impact: MEASURE Evaluation Conversation on Health Informatics
Sustaining the Impact: MEASURE Evaluation Conversation on Health InformaticsSustaining the Impact: MEASURE Evaluation Conversation on Health Informatics
Sustaining the Impact: MEASURE Evaluation Conversation on Health Informatics
 
7 Steps to EnGendering Evaluations of HIV programs with Adolescent Girls and ...
7 Steps to EnGendering Evaluations of HIV programs with Adolescent Girls and ...7 Steps to EnGendering Evaluations of HIV programs with Adolescent Girls and ...
7 Steps to EnGendering Evaluations of HIV programs with Adolescent Girls and ...
 
Sexual Orientation and Gender Identity Measures for Global Survey Research
Sexual Orientation and Gender Identity Measures for Global Survey ResearchSexual Orientation and Gender Identity Measures for Global Survey Research
Sexual Orientation and Gender Identity Measures for Global Survey Research
 
What’s Next? Practical Implementation Lessons from the Partnership for HIV-Fr...
What’s Next?Practical Implementation Lessons from the Partnership for HIV-Fr...What’s Next?Practical Implementation Lessons from the Partnership for HIV-Fr...
What’s Next? Practical Implementation Lessons from the Partnership for HIV-Fr...
 
Measuring Outcomes for Vulnerable Children: A Global Snapshot
Measuring Outcomes for Vulnerable Children: A Global SnapshotMeasuring Outcomes for Vulnerable Children: A Global Snapshot
Measuring Outcomes for Vulnerable Children: A Global Snapshot
 
Sustaining the Impact: MEASURE Evaluation Conversation on Health Systems Stre...
Sustaining the Impact: MEASURE Evaluation Conversation on Health Systems Stre...Sustaining the Impact: MEASURE Evaluation Conversation on Health Systems Stre...
Sustaining the Impact: MEASURE Evaluation Conversation on Health Systems Stre...
 
Les dialogues communautaires pour diffuser des résultats de recherche Example...
Les dialogues communautaires pour diffuser des résultats de recherche Example...Les dialogues communautaires pour diffuser des résultats de recherche Example...
Les dialogues communautaires pour diffuser des résultats de recherche Example...
 
Seven Steps to EnGendering Evaluations of Public Health Programs
 Seven Steps to EnGendering Evaluations of Public Health Programs Seven Steps to EnGendering Evaluations of Public Health Programs
Seven Steps to EnGendering Evaluations of Public Health Programs
 

Recently uploaded

GERIATRIC PHARMACOLOGY Geriatric pharmacology is a specialized field focusing...
GERIATRIC PHARMACOLOGY Geriatric pharmacology is a specialized field focusing...GERIATRIC PHARMACOLOGY Geriatric pharmacology is a specialized field focusing...
GERIATRIC PHARMACOLOGY Geriatric pharmacology is a specialized field focusing...Abhinav S
 
Bedside Utility of Liaoning Score a Non-Invasive As Predictor of Esophageal V...
Bedside Utility of Liaoning Score a Non-Invasive As Predictor of Esophageal V...Bedside Utility of Liaoning Score a Non-Invasive As Predictor of Esophageal V...
Bedside Utility of Liaoning Score a Non-Invasive As Predictor of Esophageal V...semualkaira
 
Macquarie Neurosurgery Journal Club 2022 PPT
Macquarie Neurosurgery Journal Club 2022 PPTMacquarie Neurosurgery Journal Club 2022 PPT
Macquarie Neurosurgery Journal Club 2022 PPTMQ_Library
 
Autism and Savant Syndrome|Study|The Good doctor .pptx
Autism and Savant Syndrome|Study|The Good doctor .pptxAutism and Savant Syndrome|Study|The Good doctor .pptx
Autism and Savant Syndrome|Study|The Good doctor .pptxScripted Symptoms
 
Work-role of Radiation Therapists in the Consequences of Adaptive Radiotherap...
Work-role of Radiation Therapists in the Consequences of Adaptive Radiotherap...Work-role of Radiation Therapists in the Consequences of Adaptive Radiotherap...
Work-role of Radiation Therapists in the Consequences of Adaptive Radiotherap...Subrata Roy
 
Artificial Radionuclide Generators in Medicine Applications in Radiotherapy.pptx
Artificial Radionuclide Generators in Medicine Applications in Radiotherapy.pptxArtificial Radionuclide Generators in Medicine Applications in Radiotherapy.pptx
Artificial Radionuclide Generators in Medicine Applications in Radiotherapy.pptxDr. Dheeraj Kumar
 
Tolerance Hydra10P Avene trainings blink
Tolerance Hydra10P Avene trainings blinkTolerance Hydra10P Avene trainings blink
Tolerance Hydra10P Avene trainings blinkClinicians
 
ABDULRAHMAN AL RAFIQ UPDATED MANAGEMENT OF ACUTE PULMONARY EMBOLISM .pdf
ABDULRAHMAN AL RAFIQ UPDATED MANAGEMENT OF ACUTE PULMONARY EMBOLISM .pdfABDULRAHMAN AL RAFIQ UPDATED MANAGEMENT OF ACUTE PULMONARY EMBOLISM .pdf
ABDULRAHMAN AL RAFIQ UPDATED MANAGEMENT OF ACUTE PULMONARY EMBOLISM .pdfAbdurahmanRafeeq
 
Bursitis is inflammation or irritation of a bursa sac. Bursae are small fluid...
Bursitis is inflammation or irritation of a bursa sac. Bursae are small fluid...Bursitis is inflammation or irritation of a bursa sac. Bursae are small fluid...
Bursitis is inflammation or irritation of a bursa sac. Bursae are small fluid...leejalashrestha2
 
KOSHTA CONCEPT IN AYURVEDA BY VIJAY R K
KOSHTA CONCEPT IN AYURVEDA  BY  VIJAY R KKOSHTA CONCEPT IN AYURVEDA  BY  VIJAY R K
KOSHTA CONCEPT IN AYURVEDA BY VIJAY R Kvinaykumarrkumabar
 
Seminario biología molecular Kevin Duque
Seminario biología molecular Kevin DuqueSeminario biología molecular Kevin Duque
Seminario biología molecular Kevin Duquekevinestebanduque
 
Namburi phased spot test - NPST To identify bhasma and sindhura - A Qualitat...
Namburi phased spot test - NPST  To identify bhasma and sindhura - A Qualitat...Namburi phased spot test - NPST  To identify bhasma and sindhura - A Qualitat...
Namburi phased spot test - NPST To identify bhasma and sindhura - A Qualitat...Dr. Madduru Muni Haritha
 
seminario bio mol- sofia lopez valenciaa
seminario bio mol- sofia lopez valenciaaseminario bio mol- sofia lopez valenciaa
seminario bio mol- sofia lopez valenciaasofialopezvalencia
 
Cell cytoskeleton and molecular motors.pdf
Cell cytoskeleton and molecular motors.pdfCell cytoskeleton and molecular motors.pdf
Cell cytoskeleton and molecular motors.pdfMedicoseAcademics
 
Psychedelic Interventions for Neurological Conditions [NAN 2022 Poster]
Psychedelic Interventions for Neurological Conditions [NAN 2022 Poster]Psychedelic Interventions for Neurological Conditions [NAN 2022 Poster]
Psychedelic Interventions for Neurological Conditions [NAN 2022 Poster]Brian Peacock
 
How to be a successful physician intrapreneur
How to be a successful physician intrapreneurHow to be a successful physician intrapreneur
How to be a successful physician intrapreneurArlen Meyers, MD, MBA
 
Scanning electron microscope - SEM , Its applicability in rasashastra
Scanning electron microscope - SEM , Its applicability in rasashastraScanning electron microscope - SEM , Its applicability in rasashastra
Scanning electron microscope - SEM , Its applicability in rasashastraDr. Madduru Muni Haritha
 
pediatrics. epilepsy and seizures in children 8.ppt
pediatrics. epilepsy and seizures in children 8.pptpediatrics. epilepsy and seizures in children 8.ppt
pediatrics. epilepsy and seizures in children 8.pptArun170190
 

Recently uploaded (20)

GERIATRIC PHARMACOLOGY Geriatric pharmacology is a specialized field focusing...
GERIATRIC PHARMACOLOGY Geriatric pharmacology is a specialized field focusing...GERIATRIC PHARMACOLOGY Geriatric pharmacology is a specialized field focusing...
GERIATRIC PHARMACOLOGY Geriatric pharmacology is a specialized field focusing...
 
Bedside Utility of Liaoning Score a Non-Invasive As Predictor of Esophageal V...
Bedside Utility of Liaoning Score a Non-Invasive As Predictor of Esophageal V...Bedside Utility of Liaoning Score a Non-Invasive As Predictor of Esophageal V...
Bedside Utility of Liaoning Score a Non-Invasive As Predictor of Esophageal V...
 
Macquarie Neurosurgery Journal Club 2022 PPT
Macquarie Neurosurgery Journal Club 2022 PPTMacquarie Neurosurgery Journal Club 2022 PPT
Macquarie Neurosurgery Journal Club 2022 PPT
 
Autism and Savant Syndrome|Study|The Good doctor .pptx
Autism and Savant Syndrome|Study|The Good doctor .pptxAutism and Savant Syndrome|Study|The Good doctor .pptx
Autism and Savant Syndrome|Study|The Good doctor .pptx
 
Work-role of Radiation Therapists in the Consequences of Adaptive Radiotherap...
Work-role of Radiation Therapists in the Consequences of Adaptive Radiotherap...Work-role of Radiation Therapists in the Consequences of Adaptive Radiotherap...
Work-role of Radiation Therapists in the Consequences of Adaptive Radiotherap...
 
Artificial Radionuclide Generators in Medicine Applications in Radiotherapy.pptx
Artificial Radionuclide Generators in Medicine Applications in Radiotherapy.pptxArtificial Radionuclide Generators in Medicine Applications in Radiotherapy.pptx
Artificial Radionuclide Generators in Medicine Applications in Radiotherapy.pptx
 
Tolerance Hydra10P Avene trainings blink
Tolerance Hydra10P Avene trainings blinkTolerance Hydra10P Avene trainings blink
Tolerance Hydra10P Avene trainings blink
 
ABDULRAHMAN AL RAFIQ UPDATED MANAGEMENT OF ACUTE PULMONARY EMBOLISM .pdf
ABDULRAHMAN AL RAFIQ UPDATED MANAGEMENT OF ACUTE PULMONARY EMBOLISM .pdfABDULRAHMAN AL RAFIQ UPDATED MANAGEMENT OF ACUTE PULMONARY EMBOLISM .pdf
ABDULRAHMAN AL RAFIQ UPDATED MANAGEMENT OF ACUTE PULMONARY EMBOLISM .pdf
 
Anti-interferon-gamma autoantibody associated immunodeficiency
Anti-interferon-gamma autoantibody associated immunodeficiencyAnti-interferon-gamma autoantibody associated immunodeficiency
Anti-interferon-gamma autoantibody associated immunodeficiency
 
Bursitis is inflammation or irritation of a bursa sac. Bursae are small fluid...
Bursitis is inflammation or irritation of a bursa sac. Bursae are small fluid...Bursitis is inflammation or irritation of a bursa sac. Bursae are small fluid...
Bursitis is inflammation or irritation of a bursa sac. Bursae are small fluid...
 
KOSHTA CONCEPT IN AYURVEDA BY VIJAY R K
KOSHTA CONCEPT IN AYURVEDA  BY  VIJAY R KKOSHTA CONCEPT IN AYURVEDA  BY  VIJAY R K
KOSHTA CONCEPT IN AYURVEDA BY VIJAY R K
 
Seminario biología molecular Kevin Duque
Seminario biología molecular Kevin DuqueSeminario biología molecular Kevin Duque
Seminario biología molecular Kevin Duque
 
Namburi phased spot test - NPST To identify bhasma and sindhura - A Qualitat...
Namburi phased spot test - NPST  To identify bhasma and sindhura - A Qualitat...Namburi phased spot test - NPST  To identify bhasma and sindhura - A Qualitat...
Namburi phased spot test - NPST To identify bhasma and sindhura - A Qualitat...
 
seminario bio mol- sofia lopez valenciaa
seminario bio mol- sofia lopez valenciaaseminario bio mol- sofia lopez valenciaa
seminario bio mol- sofia lopez valenciaa
 
Asesmen Klinis Trauma Tangan (Versi 2024)
Asesmen Klinis Trauma Tangan (Versi 2024)Asesmen Klinis Trauma Tangan (Versi 2024)
Asesmen Klinis Trauma Tangan (Versi 2024)
 
Cell cytoskeleton and molecular motors.pdf
Cell cytoskeleton and molecular motors.pdfCell cytoskeleton and molecular motors.pdf
Cell cytoskeleton and molecular motors.pdf
 
Psychedelic Interventions for Neurological Conditions [NAN 2022 Poster]
Psychedelic Interventions for Neurological Conditions [NAN 2022 Poster]Psychedelic Interventions for Neurological Conditions [NAN 2022 Poster]
Psychedelic Interventions for Neurological Conditions [NAN 2022 Poster]
 
How to be a successful physician intrapreneur
How to be a successful physician intrapreneurHow to be a successful physician intrapreneur
How to be a successful physician intrapreneur
 
Scanning electron microscope - SEM , Its applicability in rasashastra
Scanning electron microscope - SEM , Its applicability in rasashastraScanning electron microscope - SEM , Its applicability in rasashastra
Scanning electron microscope - SEM , Its applicability in rasashastra
 
pediatrics. epilepsy and seizures in children 8.ppt
pediatrics. epilepsy and seizures in children 8.pptpediatrics. epilepsy and seizures in children 8.ppt
pediatrics. epilepsy and seizures in children 8.ppt
 

Lessons Learned In Using the Most Significant Change Technique in Evaluation

  • 1. Lessons Learned In Using the Most Significant Change Technique in Evaluation Mary Freyder, MPH, LMSW Bridgit Adamou, MPH January 8, 2020
  • 2. • Data for Impact (D4I) overview • Sharing lessons from the field – Most Significant Change (MSC) in five countries under MEASURE Evaluation Project • Deep dive - evaluating the Local Capacity Initiative (LCI) in Uganda • Questions and answers Presentation Overview
  • 3. • Generate evidence • Strengthen capacity • Ensure data quality • Integrate gender • Promote data use • Learn D4I works to:
  • 4. • Participatory approach • Participants give personal stories of significant change, directly or indirectly related to intervention • Stories typically analyzed and filtered through various organization levels Introduction to MSC
  • 5. MSC process in three steps 1. Decide the types of stories to be collected 2. Gather stories from participants 3. Share stories with all stakeholders to learn
  • 6. • A program adapts to different or changing contexts, leading to differences in implementation and outcomes • Cause and effect are poorly understood MSC: Useful when… • Multiple intervention components exist • Evaluation is focused on learning, not just accountability
  • 7. • Flexible, captures broad range of results • Addresses limitations of other, more traditional methods • Simple process MSC: Strengths • Goal-free, no predetermined outcome
  • 8. • Bias toward success stories • Subjectivity in story selection • Bias toward popular views • Bias toward views of those who are good storytellers MSC: Limitations
  • 9. MEASURE Evaluation’s applications of MSC • Feed the Future’s Integrating Nutrition in Value Chains (INVC) program in Malawi • M&E systems strengthening in Côte d’Ivoire and Nigeria • USAID’s Gender, Policy, and Measurement Program, in partnership with the Suaahara Project and the Government of Nepal • USAID’s Sustainable Comprehensive Responses (SCORE) Project in Uganda • PEPFAR’s Local Capacity Initiative (LCI) in Uganda and Ghana
  • 10. • Part of the evaluation of Feed the Future’s INVC program • Nutrition promoters trained on MSC in two districts • 26 volunteer nutrition promoters collected 277 stories from project beneficiary households RESULTS: • Stories were short, lacked details, and were repetitive • Many of the stories covered multiple domains • MSC was useful for understanding participants’ perceptions of the key benefits of INVC Application of MSC in Malawi
  • 11. • In both countries, researchers sought to determine the most significant changes experienced by stakeholders in the improvement of each country’s national HIV M&E system. • Using a facilitator well-trained in MSC, a stakeholder workshop was held in each of the countries to identify stories. • Groups used an adapted self-assessment tool to guide discussion, agree on a response, and then provide sources of evidence for the responses. • Researchers validated the stories and conducted key informant interviews to obtain additional details. Application of MSC in Côte d’Ivoire and Nigeria
  • 12. RESULTS: • Engaging stakeholders in the MSC method helped them identify important changes resulting from interventions to strengthen M&E systems. Application of MSC in Côte d’Ivoire and Nigeria • Researchers clearly heard stakeholders’ voices and could comprehensively and systematically quantify improvements in each of the 12 components of M&E systems strengthening.
  • 13. • Impact evaluation was conducted to understand the contributions of two approaches to strengthening the capacity of Health Facility Operation & Management Committees (HFOMCs). • MSC method was used for focus group discussions with HFOMC members and community members to see if significant changes had occurred in their lives. • A local research organization collected stories from community members and presented them to the HFOMC members for group discussion. Application of MSC in Nepal
  • 14. Application of MSC in Nepal RESULTS: • Using the MSC approach with community members was challenging because the intervention was implemented by the same organization that implemented another program and community members confused the two. • HFOMC members shared useful stories about how the project helped them obtain information.
  • 15. • The SCORE project aimed to benefit orphans and vulnerable children and their caregivers. • The program’s aims were to build economic resilience, enhance food security, improve child protection, and increase access to education and critical services. • Forty randomly selected beneficiaries and project staff were asked about positive and negative changes resulting from the program. Application of MSC in Uganda
  • 16. RESULTS: • Because of time and resource constraints, the MSC framework was used to develop open-ended questions for the participants. • The MSC method generated rich and interesting responses. Application of MSC in Uganda • Analyzing the answers revealed that the layered, multifaceted components of the intervention led to positive change, particularly at the family level.
  • 17. • The MSC approach is time-consuming and requires careful facilitation. • People collecting stories should be well trained and supervised. • The MSC approach is easy for participants to understand. • It is worthwhile to allow ample time for the interviews—even if this means gathering fewer stories. Lessons learned across MSC studies
  • 18. • Follow-up interviews with beneficiaries, program staff, or donors can further strengthen the learning aspect of this approach and complement MSC data. Lessons learned across MSC studies • The MSC technique is useful when evaluators must narrow down the components of participant observations of what changed. • This method tends to have a bias for positive responses.
  • 19. • Research brief: Experiences and Lessons Learned: Implementing the Most Significant Change Method • Report: Integrating Nutrition in Value Chains in Malawi: Using Most Significant Change stories to understand community experiences • Case study: A case study to measure national HIV M&E systems strengthening: Nigeria • Case study: A case study to measure national HIV M&E systems strengthening: Côte d’Ivoire • Report: Uganda’s SCORE program for vulnerable children and their families: Mixed-methods performance evaluation D4I resources on the MSC technique
  • 20. This presentation was produced with the support of the United States Agency for International Development (USAID) under the terms of the Data for Impact (D4I) associate award 7200AA18LA00008, which is implemented by the Carolina Population Center at the University of North Carolina at Chapel Hill, in partnership with Palladium International, LLC; ICF Macro, Inc.; John Snow, Inc.; and Tulane University. The views expressed in this publication do not necessarily reflect the views of USAID or the United States government. www.data4impactproject.org

Editor's Notes

  1. D4I, which is a MEASURE Evaluation associate award and is funded by United States Agency for International Development, works with countries to: Generate strong evidence for program and policy decision making Build individual and organizational capacity Enhance the use of data for global health programs and policies
  2. MEASURE Evaluation has implemented the MSC technique in evaluations of health programs. This presentation summarizes our experience and shares lessons learned in the application of MSC. But first, what is MSC? It is a participatory qualitative approach to monitoring and evaluation that involves assessing the changes and impacts as the result of a program from the perspective of program participants or beneficiaries. It is participatory because program participants themselves and stakeholders are usually involved in deciding types of change to explore, and in analyzing the data, or stories, collected. Project staff collect stories from beneficiaries, partners, or participants. Story collectors ask questions such as the following: “During the last period, in your opinion, what was the most significant change that took place for participants in the project?” Respondents describe both the change and the reasons they consider it significant. The stories are then analyzed and filtered through different levels. “Typically” is in italics here because MSC has been implemented in many different ways, ranging from the original version of it, with multiple levels of story review/analysis and voting, to just asking individuals for their stories and researchers analysing those. You will see this illustrated through the MEASURE Evaluation examples coming up.
  3. The process entails three basic steps: 1. Deciding the types of stories that should be collected from beneficiaries, partners, or participants 2. Gathering stories by asking respondents to describe both the change they experienced and the reasons they consider it significant, then collectively determining which stories are the most significant 3. Sharing the stories with stakeholders and contributors to learn about what is valued
  4. MSC is useful when you are evaluating a program that adapts to different or evolving contexts, leading to differences in implementation and outcomes. When cause and effect is not understood well. When there are multiple program components And when your evaluation is learning focused rather than solely focused on accountability.
  5. MSC is flexible: open to broader range of results, including those that may not have originally been intended by the program. Complementary to more traditional methods: can help address limitations of other methods. For example, if you are using a structured survey only, you may miss unanticipated changes, negative or positive. Simple, straightforward process that anybody can understand and use. Compared to other monitoring approaches, it is easy to communicate across cultures. There aren’t any indicators, so there is no need to explain what an indicator is. Everyone can tell stories about events they think were important. Goal-free: no assumption of what success should look like. Because it is not a linear model, there are no predetermined outcomes.
  6. This method is biased towards successes – particularly to make the storyteller look good. Evaluators can overcome this by having a domain of change for negative stories. There is subjectivity in story selection. Recording the reasons for selecting stories helps strengthen the process There are biases toward popular views. And there are also biases toward views of those who are good storytellers. This is why don’t use as stand-alone tool.
  7. MEASURE Evaluation applied most significant change in five different settings to monitor and evaluate various community based or complex programs. One was a nutrition program in Malawi. Another looked at how national government HIV M&E systems in Cote d’Ivoire and Nigeria were strengthened. In Nepal, MSC was used as part of an impact evaluation involving health facility operation and management committees. The Sustainable Comprehensive Responses, or SCORE, program in Uganda aimed to benefit orphans and vulnerable children and their caregivers. And then lastly there was the PEPFAR-supported global initiative to support local partners in HIV policy advocacy for highly vulnerable populations such as men who have sex with men, transgender women and sex workers. My colleague, Mary, will discuss this application more in-depth.
  8. MSC was intended to be one aspect of qualitative data collection for a large impact evaluation of Feed the Future’s Integrating Nutrition in Value Chains program. Volunteer nutrition promoters were trained on MSC in Malawi’s Mchinji and Lilongwe districts to obtain information from participants about their experiences with the INVC program. Over the course of a year, 26 nutrition promoters, who were also program implementers, collected 277 MSC stories from project beneficiary households. Outcomes: Because the nutrition promoters wrote the stories in their notebooks, summarizing them rather than writing them verbatim, the stories tended to be very short, had few details, and were repetitive. The similarities among stories made it difficult for the promoters to discuss, rank, and select the most significant stories. Many of the stories covered more than one domain of change, which also made it challenging for promoters to assign the stories to the domains. The MSC method was useful for understanding participants’ perceptions of the key benefits of and challenges to the INVC program. (Due to the cancellation of the impact evaluation, no other qualitative data was collected to supplement the MSC stories.)
  9. In two case studies, researchers sought to produce evidence of how Côte d’Ivoire and Nigeria’s national HIV M&E systems were strengthened. The MSC technique was applied in both countries to determine the MSCs experienced by stakeholders in the improvement of each country’s national HIV M&E system. With the support of a facilitator well-trained in MSC, a stakeholder workshop was held in each of the countries to identify stories. Participants started by self-selecting into groups. Next, each group used an adapted self-assessment tool to guide discussion, agree on a response, and then provide sources of evidence for the responses. Groups reported to the plenary to verify each other’s findings. Afterwards the researchers validated the stories and conducted key informant interviews to obtain additional details and perspectives on the MSCs that were identified in the workshop.
  10. Researchers found that engaging stakeholders in the MSC method helped them identify the important changes resulting from interventions to strengthen M&E systems, which also helped to raise interest in these interventions and the defined domains of change. Using the MSC method also meant that researchers clearly heard stakeholders’ voices and could comprehensively and systematically quantify improvements or successes in each of the 12 components of M&E systems strengthening.
  11. MEASURE Evaluation conducted an impact evaluation to understand the contributions of two approaches to strengthening the capacity of the Health Facility Operation and Management Committees (HFOMCs) in Nepal. At the end line of the evaluation, we used the MSC method to conduct eight focus group discussions with HFOMC members and community members to see if significant changes had occurred in their lives as a direct result of their participation in the program or as a result of the program’s interventions overall. The plan was to apply the MSC method in two ways: (1) HFOMC members were to engage in focus group discussions about changes they experienced through program participation and (2) HFOMC members were to collect stories of change from the community for the HFOMC to discuss and vote on. The latter story collection did not happen as planned, however, because of HFOMC members’ limited time and limited skills in facilitating discussions, among other reasons. Instead, a local research organization collected the stories from community members and presented them to the HFOMC members for group discussion.
  12. Using the MSC approach with community members proved challenging because the intervention was implemented by the same organization that implemented another program and community members confused the two. However, HFOMC members shared useful stories about how the project helped them obtain information.
  13. USAID’s Sustainable, Comprehensive Responses (SCORE) project operated in Uganda to benefit orphans and vulnerable children and their caregivers. The program’s aims were to build economic resilience, enhance food security, improve child protection, and increase access to education and critical services. MEASURE Evaluation used the MSC method to assess the project’s effects on beneficiaries and the strengths and challenges associated with implementation. Using the MSC techniquie, 40 randomly selected beneficiaries and project staff about positive and negative changes resulting from the program.
  14. Time and resource constraints prevented respondents from gathering to discuss and vote on the changes that had the most impact. Instead, we used the MSC framework to develop open-ended questions for the participants, which generated rich and interesting responses. Analysis of these answers revealed that the layered, multifaceted components of the intervention led to positive change, particularly at the family level.
  15. Before we dive deep into the LCI Uganda evaluation, I’ll share a few lessons learned across all four studies. Mary will highlight a few of these in her presentation. The MSC approach is time-consuming and requires careful facilitation. If people capturing stories lack sufficient time and skills, it is better to select others better suited to use it correctly, or plan to spend significant time training data collectors to record stories—or avoid the method altogether. People collecting stories should be well trained and supervised so the stories are thoroughly captured with detailed notes. The MSC approach is easy for participants to understand. It is worthwhile to allow ample time for the interviews—even if this means gathering fewer stories. It’s important that data collectors can fully explain the questions and that participants have enough time to think about their experiences and articulate their responses.
  16. Follow-up interviews with beneficiaries, program staff, or donors can further strengthen the learning aspect of this approach and complement the MSC data. The MSC technique is useful when evaluators must systematically narrow down the components of participant observations of what changed. As mentioned earlier, this method tends to have a bias for positive responses. Specifically asking a question about most significant negative change can address this potential drawback.