Advertisement
Monitoring, evaluation and theory of change
Nov. 30, 2020•0 likes•59 views
0 likes
Be the first to like this
Show More
views
Total views
0
On Slideshare
0
From embeds
0
Number of embeds
0
Download to read offline
Report
Science
Presented by Steven Lam and Fred Unger, ILRI at a Taskforce and Stakeholder meeting, 24-25 October 2019 in Siem Reap, Cambodia.
ILRIFollow
-- at ILRIAdvertisement
Advertisement
Advertisement
Recommended
SPLC 2018 Summit: Developing a Sustainable Purchasing Policy for Your Organiz...SPLCouncil
SPLC 2018: Developing a Sustainable Purchasing Policy for Your OrganizationSPLCouncil
Why LMI\'s Clients continue to excelswingkid555
Free PMP Sample Q & AOSP International LLC
10265754 fire fightingIzabela-Maria Catiru
validation presentation 07 11-13QAP-Northampton
Monitoring, evaluation and theory of change
- Monitoring &evaluation Theory of change Fred Unger and Steven Lam SFFF Cambodia Taskforce and Stakeholder Meeting Siem Reap, 24-25 October 2019
- Outline • What is monitoring and evaluation (M & E) • Ways to monitor and evaluate • Relationship between outcome mapping and theory of change • Group exercise
- It answers questions of are we implementing as we planne M & E e.g. to research and apply market- based approaches Goals Objecti ves Activiti es/ inputs Planne d outputs Expecte d outcom es Expecte d impact Focus of monitorin g Focus of evaluatio n e.g. to improve food safety in Cambodia It answers questions of: are we implementing as we planned It answers questions: have we make things better
- A note on outcomes and outputs Outputs: achieved immediately after implementing an activity - # of reports published - # of retailers trained - ↑ knowledge of safe food handling practices Outcomes: more medium- to long-term changes - # retailers demonstrating safe food handling - # retailers promoting new approaches to other retailers - # retailers reporting improved income
- M & E of outcomes Why does monitoring and evaluating outcomes matter? • To document project successes and why they happened other groups may learn from our experience • To inform donors of project progress Related to set up objectives and deliverables • To help secure future funding Based on positive evaluation evidence Goals Objecti ves Activiti es/ inputs Output s Outco mes Impact SFFF is complex and raises the question: will traditional M&E work?
- What are ways to monitor outcomes? 1/2 Outcome Harvesting (Wilson-Grau 2012) • An approach for evaluating development programming/projects • Helps to identify a program’s actors and their outcomes • Works backward, starting with the outcome, to determine how the program contributed to the observed change 6 key steps
- What are ways to monitor outcomes? 2/2 Outcome Mapping (IDRC 2001) • An approach for planning, monitoring, and evaluating development programming/projects • As Outcome Harvesting, also helps to identify projects actors and their outcomes (but towards outcomes not backwards) 12 key steps
- Outcome mapping: several stages Stage 1: Intentional design vision, mission, and partners we like to influence. Stage 2: Outcome and performance monitoring developing a framework to monitor the progress of boundary partners towards the achievement of outcomes. Stage 3: Evaluation planning identifying evaluation priorities and developing an evaluation plan. 12 key steps
- A focus on stage 1 Step 1: Vision large-scale development changes that SFFF hopes to achieve Step 2: Mission aligned to objectives & to support vision Step 3: Boundary partners (BP) individual/groups/organisations of stakeholders the project interacts directly to influence for change Step 4: Outcome challenges describes desired outcomes among each boundary partner Step 5: Progress markers Statement of BP gradual changes (steps of changes) towards the achievements of outcome challenge Step 6: Strategy maps How will the project contribute to the achievement of the outcome challenge Step 7: Organizational practices how do you have to be operating in order to support these changes in the boundary partners 12 key steps
- Relationship between OM and TOC The OM process of outlining expected outcomes and strategies to achieve them will help us develop / validate a Theory of Change. A Theory of Change is a visual representation linking outcomes to activities and helps explain HOW and WHY a change is expected to come about. Simply said, it helps to illustrate the outcome pathway of SFFF and improves the likelihood of program success.
- Ultimate outcome/goal
- SFFF - project aim Reducing the burden of FBD in informal, emerging formal, and niche markets and targeting small and medium scale producers, the project will have 5 objectives with associated activities, outputs and outcomes.
- SFFF Cambodia – Vision (large scale change we like to achieve) G1: To improve consumers health by reducing the burden of foodborne disease from poultry and pork products in Cambodia. G2: To improve community health through reducing the burden of FBD G3: The risks of FBD from ASF are reduced in Cambodia in the future
- SFFF Cambodia – Mission (aligned to objective & to support vision) • G1: By building capacity of national partners and generating evidence on the risk of FBD in Cambodia, we will pilot measures to improve hygienic standards among pork and poultry retailers, particularly female traders, and thus improve food safety. • G2: To provide actionable evidence on FBD to policy makers and deliver appropriate interventions for retailers and consumers. • G3: To increase awareness on food safety among stakeholders through effective sanitation, hygiene and preventive measurement.
- Boundary partners • NAHPRI • CelAgrid • RUA • RUPP • ICT • CDC • National Institute of Public Health • Ministry of Health • MAFF • WHO • FAO • IPC • Retailers • Consumer • Abattoirs • Technical working group - Taskforce Needs to be narrowed down/prioritised
- Provisional framework stakeholders engagement, and M&E process using outcome mapping Vision Mission SFFF Objectives 1 - 5 Boundary Partners (BP) B P B P OC Outcome Challenges (OC) Progress Markers (PM) PM ----- ----- ----- Strategy Research & TSs engagement strategies/ activities B P B P OC PM ----- ----- ----- B P B P OC PM ----- ----- ----- M&E + Org. Learnin g Strategy Strategy November onwards
- Some readings
Editor's Notes
- What is M&E? Many of you have probably seen this before; it is a generic program logic model which shows linkages between a program’s objectives all the way to its impacts. For example, a food safety program’s objectives are to… Monitoring is ongoing and would focus on activities and outputs. It answers questions of are we implementing as we planne Evaluation is often a one-time assessment; focusing on outcomes and sometimes impact. It answers the question what works and how to make things better. Typically a traditional M&E system will determine the program logic model, and assess the program according to whether planned objectives and expected outcomes were met.
- Before we get into outcome mapping and TOC, just a brief overview of some concepts so that we are all on the same page: Outcomes are changes in behavior, relationships, activities, or actions of an individual, group, community, organization, or institution. Monitoring is the ongoing, systematic collection of data about a program (e.g. what works well? Why? What could we do better?) Why is it important to monitor outcomes? - To document our successes and why they happened; such that other groups may learn from our experience - To inform our donors of our progress toward our objectives that we had set out to do - The evaluation evidence, if collected in a rigorous and systematic way, can be helpful for advocating for more funding
- Before we get into outcome mapping and TOC, just a brief overview of some concepts so that we are all on the same page: Outcomes are changes in behavior, relationships, activities, or actions of an individual, group, community, organization, or institution. M&E of SFFF, however is difficult as SFFF is a complex program, characterized by multiple actors, multiple components, and challenging complex (food security issue). Thus, traditional approaches to M&E (e.g. results based management are not always appropriate). Rather, flexible approaches, responsive to change, are more suitable. Why is it important to monitor outcomes? - To document our successes and why they happened; - To inform our donors of our progress toward our objectives that we had set out to do - The evaluation evidence, if collected in a rigorous and systematic way, can be helpful for advocating for more funding
- For complex programs such as ours where there are multiple stakeholders, multiple projects, an ambitious goal of address food safety in Cambodia (and uncertainty the type of outcomes achievable), a flexible evaluation approach is needed to support program learning and adaptation. Outcome harvesting, developed by Ricardo Wilson-Grau, is one such approach designed to evaluate development programming that helps identify a program’s actors and their expected outcomes. It consists of 6 steps; working backwards, starting with the outcome, to determine how the program contributed to the change. It is typically conducted toward the end of the program.
- Outcome mapping, developed by IDRC, is a more comprehensive tool that supports not only evaluation functions but also program planning and monitoring. It is a 12-step process with 3 stages compared to 6-steps of outcome harvesting.
- As we have already worked through steps 1-5 of outcome mapping for PigRisk, in this meeting we’d like to focus more on steps after 6. We likely won’t be able to get through everything today, but as brief introduction: Step 6 of intentional design is strategy maps; the purpose of the strategy map is to identify the strategies used by the program to contribute to the achievement of an outcome. It asks: How will the program contribute to the achievement of the outcome challenge? The next step is organizational practices – the program reviews the outcome challenges and identifies organizational practices that will best help it to contribute to the intended change in its boundary partners. It asks “how do you have to be operating in order to support these changes in the boundary partners? In other words, what does the team have to be doing in terms of practices in order to remain effective? Can think of this as “SafePORK’s approach” (e.g. market-based approach, community-based…) Other examples: - Seeking feedback from key informants - Assessing and redesigning products, services, systems and procedures - Sharing lessons learning - Experimenting to remain innovative - Engaging in organizational reflection During the outcome and performance monitoring stage, we develop a framework to monitor: 1) the progress of boundary partners towards the achievement of outcomes 2) The strategies to support outcomes 3) organizational practices used It involves planning who does what / when, developing data collection sheets, and implementing data collection. Finally, the evaluation planning stage is about identifying evaluation priorities (such as program effectiveness, relevance, or impact) and developing and evaluation plan. We will work through stages 2 and 3 at a later time.
- As we have already worked through steps 1-5 of outcome mapping for PigRisk, in this meeting we’d like to focus more on steps after 6. We likely won’t be able to get through everything today, but as brief introduction: Step 6 of intentional design is strategy maps; the purpose of the strategy map is to identify the strategies used by the program to contribute to the achievement of an outcome. It asks: How will the program contribute to the achievement of the outcome challenge? The next step is organizational practices – the program reviews the outcome challenges and identifies organizational practices that will best help it to contribute to the intended change in its boundary partners. It asks “how do you have to be operating in order to support these changes in the boundary partners? In other words, what does the team have to be doing in terms of practices in order to remain effective? Can think of this as “SafePORK’s approach” (e.g. market-based approach, community-based…) Other examples: - Seeking feedback from key informants - Assessing and redesigning products, services, systems and procedures - Sharing lessons learning - Experimenting to remain innovative - Engaging in organizational reflection During the outcome and performance monitoring stage, we develop a framework to monitor: 1) the progress of boundary partners towards the achievement of outcomes 2) The strategies to support outcomes 3) organizational practices used It involves planning who does what / when, developing data collection sheets, and implementing data collection. Finally, the evaluation planning stage is about identifying evaluation priorities (such as program effectiveness, relevance, or impact) and developing and evaluation plan. We will work through stages 2 and 3 at a later time. It is likely that some of these 7 steps have already been done for SFFF, but it’s a good opportunity to revisit them and revise if needed after learning from research and other activities post proposal stage.
- Here is a generic TOC developed for milk traders by A4NH in Kenya. Starting from the top is the ultimate expected outcome / goal (increased consumption of safe food). The boxes below are conditions that must be in place to reach the goal. Can think of it, from top to bottom as: “for this to happen, this must happen”. Overall, this figure is similar to a logic model, but it provides a pathway showing clearly the linkages between activities and outcomes. Another important component of ToCs are assumptions; which are conditions that must be in place for the linkages to be achieved. By creating a ToC, and testing our assumptions, we can look for possible gaps in our program logic, address them, and improve our program. What we hope to do at some point for SFFF is to develop a TOC to illustrate its impact pathway, which can be informed by working through steps of OM. OM provides a framework, while TOC provides a tool to think through the logic of the program. Both of which complement each other, and having both with help generate a strong M&E strategy.
- We have created several outputs related to M&E: - report identifying BPs and progress markers (2014) - a briefing note sharing the outcomes among researchers (e.g. knowledge and practices) (2016) - the outcome harvesting approach has been used in the context of Vietnam for an Ecohealth project (2018) - we are currently preparing a manuscript synthesizing evaluation findings for PigRisk
Advertisement