Meta Story to Communicate M&E
•Download as PPT, PDF•
0 likes•437 views
The use of meta stories can be an effective approach to communicate project monitoring and evaluation. Khulisa Senior Associate, Katharine Tjasink, sparked stimulating conversation at a recent brownbag lunch when she shared the meta story methodology. Khulisa successfully applied this approach to Farmers Voice Radio, a radio-based agricultural extension and education programme that targets small scale farmers. The approach has enabled Farmers Voice Radio to use storytelling as a tool to communicate a complex programme to donors in way that simplifies how the programme works, and the results that have been achieved. Katherine presented the concept at the American Evaluation Association’s annual conference in Washington DC in October, last year.
Recommended
Recommended
More Related Content
Similar to Meta Story to Communicate M&E
Similar to Meta Story to Communicate M&E (20)
More from Khulisa Management Services
More from Khulisa Management Services (20)
Recently uploaded
Recently uploaded (20)
Meta Story to Communicate M&E
- 5. …most development agencies report only their outputs (e.g., number of teachers trained, kilometres of roads built) rather than outcomes “ ”
- 6. THE APPROACH
- 8. EXPERIENCE DIFFERENCE LESSONS CHALLENGE VISION SOLUTION “…now we have radio, and through radio we can reach all our farmers” Christian Sambala, Radio Extension Officer
- 13. SUSTAINABLE PARTICIPATORY TIMELY & RELEVANT REACHES MORE FARMERS
- 16. THE BIG PICTURE
- 17. POTENTIAL USES Reporting tool Living document Evaluation exercise Interim evaluation tool
- 18. THANK YOU
Editor's Notes
- Just to give you a little bit of background information, FVR is a radio-based agricultural extension, or education, program that targets small scale farmers. It was initiated though the American Institutes for Research (AIR) in DC and was taken over by Khulisa Management Services, which is a monitoring and evaluation firm based in South Africa Our program started in Kenya and Malawi in 2009 and expanded to Tanzania and then Uganda in 2011. Currently, we have 35 radio partners that are broadcasting agricultural programs in 19 different languages altogether. Farmer Voice Radio began as a proof of concept program with a developmental design, to test whether it was possible to deliver agricultural extension services to farmers, using radio, sustainably and at scale. Over the course of the past four years, we have developed a complex model that we think, and we know, works. However, telling our story and getting other people, including our donors, to understand how the model works, and what we have achieved, has been challenging
- Like many others, we fell into the trap of relying on our project monitoring and evaluation plan to collect and communicate information about our progress and results. The problem with this kind of reporting is that it tends to have a linear, quantitative, approach, whereas the development context itself is not linear but complex and dynamic This is particularly true in the context of a developmental model where it becomes even more difficult to determine uniform indicators or standardize measurement upfront. In a multi-country developmental research setting, each country typically has a distinct economic, socio-political, organizational, and cultural context Implementing the same program can have very different results, particularly when you are emphasizing tailoring the program to fit local context. On the other hand, a traditional project monitoring plan is set up to measure a “gold standard” for standardized indicators and often cannot pick up on those kind of subtleties in implementation and what they mean. It also fails to capture the lessons learned that are essential to developing and refining a model
- While extremely important, and necessary, M&E reporting does not always capture the dynamics of change, or the big picture of what a project or program has achieved. What is often missing is the “full story” – the overarching project narrative of how challenges in program implementation were faced and creatively solved.
- In the context of Farmer Voice Radio, although our monitoring and internal evaluation reporting showed what we were doing at a micro level, it did not tell our story In the words of our principal consultant, and friend to the project, we were “in the weeds” – In essence we were concentrating on communicating our activities and outputs rather than communicating the big picture We were challenged by our donor to come up with a “meta story” that would give them a birds’ eye view of our program The solution we came up with, in consultation with our donor, was to integrate a qualitative, thematic, narrative with our evidence and results to make our data meaningful to our donor and the outside world It is this approach that I will outline next - the structure of the meta-story and the potential applications of this approach as a reporting tool for various practitioners
- After being challenged by our donor to produce a meta story, we went to the literature to investigate what this method looked like and how to apply it. However, we could not find any reference to a meta story framework other than in the context of advertising and marketing The literature does, on the other hand, highlight the use of narrative methods for evaluation practice, in the context of qualitative evidence inquiry. However, we wanted to seamlessly integrate our evidence and results with a narrative rather than presenting these as distinct or separate pieces of information. So, we decided to come up with our own approach. As a radio-based program, we understand the value and power of storytelling, and of framing a story in context. As researchers, we also understand the value and power of data. We wanted to find a way to combine the two meaningfully.
- In our initial attempt at producing a meta story, we structured the approach in the following way: First, we defined the challenge or problem statement for the program, answering the question “why does this program exist in the first place?”. For example, we highlighted the fact that while over 60% of the workforce of sub Saharan Africa are engaged in agriculture, agricultural extension services (which should help these farmers) are broken: they are capacity constrained, top-down, and have limited reach and impact. Outlining the problem statement located the story of our program within a larger framework. So often, programs exist within a “vacuum” and, while they may be internally consistent, they fail to take the larger development context into account. We wanted to start the story within a bigger framework before narrowing down to the specifics of what we had done.
- 2. Next, we defined the solutions that the program aimed to put in place to meet the defined challenge. For example, one of our solutions was that we wanted to give extension workers a radio ‘megaphone’ so they could reach more farmers. In Uganda, for example, there is one extension worker to 5000 farmers. On the other hand 83% of Ugandans have household access to radio. We wanted to harness the power of radio to extend the reach of the extension worker. A lot of reporting stops here, at the proposed solution stage, and then uses the M&E reporting framework to see how well a program has progressed compared against the proposed solution. However, we felt that there was more than this to the story of our program
- 3. So we then outlined our experience with implementing the proposed solution. In this section we told the reader how we operationalized each solution and what those solutions looked like on the ground. We identified the most important components of our model and presented these in this section. For example, one core component of our model was that we linked extension workers with radio broadcasters in production teams so that they could work together to create FVR programs on an ongoing basis.
- One of the most important elements of a good story is that the main characters go through trials and tribulations before reaching resolution. We wanted to include that element of storytelling in the meta story to demonstrate how we faced challenges in program implementation, and how each of those challenges was creatively solved. 4. So we decided to highlight some of the takeaway lessons that we had learned as an outcome of our experience on the ground. For example, we learned early on that it was not enough to ask extension workers and broadcasters to work independently as agricultural and radio experts in FVR, but that creating formal production teams was essential to bridging broadcasters’ lack of agricultural knowledge and extension officers’ lack of radio programming skills. We also learned that if this process was not facilitated, extension officers and radio broadcasters tended to compete rather than work together collaboratively. It was important for us to capture lessons like these so that we could share them.
- 5. Next, we wanted to demonstrate how FVR was different to other initiatives that also utilized radio for agricultural education. So we used this section to distinguish our program from others and explain what made it different. For example, we explained that FVR was not just about broadcasting radio programs, but about creating a system for linking extension workers to radio
- 6. Finally, we articulated our vision for FVR - reaching 80% of the small farmers of Sub Saharan Africa with timely, relevant information to improve the livelihoods of the poorest of the poor. We did this so that we could contextualise our current progress and achievements in relation to our ultimate goals. Again framing the meta story within a larger context. Once we had completed this first attempt at a framework for the meta story, we stepped back and realised , in collaboration with our donor, that our report was still too linear and granular, and that we were reporting the elements of the model in silos. We were missing the common threads that tied the story together
- So, we took it one step further by figuring out what the thematic change areas of our program were and organizing the narrative thematically, keeping most of the original meta story structure, but reorganizing each section around defined change themes. I’ve represented this process visually here. First, we started with a huge amount of unorganized, but important, data about the things we had changed as a result of our program We then thought about how these things could be grouped into areas that would represent what we had changed at a higher level Finally, we labelled each of these areas. For example, that we had developed a sustainable model, that was participatory, timely and relevant, and reached more farmers than traditional methods, effectively and efficiently.
- We decided to drop the difference section, because we realised that we could cover this in the challenge and solution sections of the meta story Then we reframed the meta story to follow the four thematic change areas, cascading the themes through the narrative. Finally, we integrated evidence and results from our project monitoring plan and outcomes evaluations into this framework, also arranged by theme.
- For example, while we integrated some evidence and results directly into the narrative, we also created graphics to capture the gist of the results and incorporated these into the story. This is an example of one of the graphics capturing the sustainability theme Finally, we included a results table to link evidence and results to claims in the meta story, and this accompanied the meta story for anyone who was interested in looking at the detail.
- In summary, the meta story communicated our program’s journey from the problem statement through its evolution and outcomes and told how things actually worked, not just how they were supposed to work, highlighting major themes, and lessons learned. It mapped program components to thematic change areas and cascaded them throughout, telling one continuous story. We used our monitoring and evaluation data in a creative way by integrating data into the story rather than presenting it separately. We also made the same data available in the form of a results table that broke the information down by country, and followed the logic and structure of the meta story narrative.
- We found the meta story approach to be particularly useful in the context of our complex program and we hope that this reporting approach can be useful for other practitioners. With regard to potential applications of this approach, we think the meta story could be used in different ways for different actors. As a reporting tool, it could potentially help implementers to step back from the details and tell a story of the program, project, or model from the problem statement through its evolution and outcomes. The meta story could also be used as an exercise that an evaluator could apply with a project or program to see the big picture of the program through the eyes of the implementers. In a long-term program, where outcome- or impact-level results are not readily available, the meta-story approach can present an interim ‘bigger picture’. It can provide program implementers and donors with a higher-level assessment of program performance than traditional monitoring-level data alone can achieve. It can also become a living document that is continually updated to give a current snapshot of the big picture of a program or project. As this is not a tested tool, we would like to continue investigating the potential applications of this approach in various settings. We would like to get some feedback on this approach from our evaluation colleagues and so I would appreciate if you would quickly take a minute to jot down your comments and suggestions and send these to me at the end of the session.
- If you would like a copy of my paper, please feel free to email me at ktjasink@Khulisa.com. If you would like any more information on farmer voice radio or Khulisa Management Services, please go to our websites, www.farmervoice.org and www.Khulisa.com, or send me an email