Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Evidence On Trial: weighing the value of evidence in academic enquiry, policy and everyday life


Published on

Presentation on CCAFS's experience with theories of change, results-based management, and monitoring, evaluation and learning.

Published in: Data & Analytics
  • Be the first to comment

  • Be the first to like this

Evidence On Trial: weighing the value of evidence in academic enquiry, policy and everyday life

  1. 1. Evidence On Trial: weighing the value of evidence in academic enquiry, policy and everyday life CCAFS outcomes and targets Tonya Schuetz, Philip Thornton, Wiebke Foerch Durham 11 July 2016
  2. 2. Structure CCAFS Experience Background Theories of Changes RBM MEL Framework Issue of Evidence Credibility of Outcomes Operational Support Challenges & opportunities
  3. 3. CGIAR, a global partnership that unites organizations engaged in research for a food secure future Background FTA MAIZE Fish WHEAT PIM A4NH RICE Livestock RTB DCLAS CCAFS WLE
  4. 4. Strategy and Results Framework 2017-30 Outcome Indicators and Targets Background
  5. 5. 4. Climate information services and climate-informed safety nets 1. Policies and institutions for climate-resilient food systems 2. Climate smart technologies, practices, and portfolios Background - Flagship Programs
  6. 6. • Large bodies of work (e.g. US$1.5 million) • Source of funds: any kind • Single or Multiple Centers/Partners • Single or Multiple Regions or Global • Key feature: fits into an impact pathway Projects Background – Regional Programs
  7. 7. Theory of Change & Impact Pathways – why? • Bring in focus on people + outcomes, i.e. behavioral changes, + unanticipated changes + incentives for change • Ensure that CCAFS plan of work is targeted at outcomes, requires ‘use of outputs’ tasks built into each activity plan • Strategically encourage communication and collaboration, and improved work plans through harmonizing IPs • Revisit trajectory of CCAFS contributions to change and use them as foresight/ ex-ante impact assessment – measure progress towards change, adapted according to learning
  8. 8. From output-focused supply-driven (logframe approach) to outcome- focused demand driven Theories of Change over a two-year process (starting from the regions): • Harmonization achieved through capacity building (beyond trainings: meetings and frequent virtual communication) in all flagships, regions and partners • Built with partners & projects • RBM first trialed on one Flagship • Iterative planning process for the CRP project portfolio building Creating our Theories of Change • Set global flagships • Regional priorities • Calls for concepts • Regional planning meetings • Detailed project planning
  9. 9. • CCAFS operates across multiple sectors, scales, stakeholders • Allowing for: Structured linear thinking and planning with assumptions for how desired changes can happen Complex systems thinking: flexibility to react according to lessons and opportunities arising during implementation Research Development Universities, research institutes, bio-medical facilities, genetics etc. UN, development NGOs, philanthropic foundations, governments Solutions R4D: How were we supposed to do it? R4D: What it looks like in practice: Action Research Development Leadership Learning Communications Partnerships Trust Combination of ordered & complex systems
  10. 10. MEL theoretical framework Results-based management • Accountability for outcomes: logical chain • Emphasis on systematic, constructive looped learning from past experiences and subsequent adaption -adaptive management • Three thirds principle engaging with partners to decide what needs to be done and how; doing the actual research, often in partnership; sharing results in appropriate formats and strengthening capacity of next users to utilize the research to achieve outcomes and impact.
  11. 11. Examples • Desk studies (e.g. current World Bank investment on Climate Smart Agriculture) • Third-party surveys (deforestation rates in CCAFS regions) • Other CGIAR/ CRP surveys (e.g. Climate Smart Agriculture technologies adopted in CCAFS sites) • Formal impact assessment and evaluation methods incl. the ones conducted by independent units • Project activities (CSA technologies adopted in other sites, GHG emissions in a landscape, …) • Other (emails, internal government documents, …) What kind of “evidence” is needed? Atproject,flagship,andregionallevel Changesthroughtime–withbaselines A lot of methodological development is needed to make evaluation incl. Impact Assessments to be adequate for TOC / IP approaches.
  12. 12. • Definition of an outcome: ‘use of the research by non- research partners to develop new, or change, policies and practices’, i.e. beyond new research designs, dissemination, training and need to be 1000’ of farmers not involved in the research • Evaluation of ‘outcome case studies’ By Whom? 1/3 Internal (2) from Program Management Unit (eliminated conflict of interest), 1/3 external (1) with experience in a farmer organization, and 1/3 external (1) with >20 years in R4D Criteria for scoring: three criteria with different weights, (a) quality/clarity of write up - 15%, (b) evidence - 20%, and (c) significance of outcome - 65%. Credibility of Outcomes
  13. 13. How to strengthen the Outcomes? • Resubmission of outcomes from previous years shows how these improve over time due to further scaling. • Trend over four years (2012-15) of reported outcomes: • improved quality & relevance for 2015, large jump in no. • increase in % evaluated as ‘not yet outcomes’ • satisfactory no. of very good outcomes reported for 2015 • it would seem that a lot of valuable time went into developing Outcome Case Studies which did not yet meet the outcome criteria. • seems worthwhile to remind of the criteria.
  14. 14. Contribution to some System Level Outcomes, which link with SDGs CRP defined outcome indicators: • for the flagship level 2030 and 2022 • Quantitative capturing of outcome target values combined with • Qualitative narrative descriptions What is CCAFS being held responsible for? 2 + 3 15equitable national/subnational food system policies enacted that take into consideration climate smart practices and strategies, informed using knowledge, tools and approaches derived from CCAFS science 1 + 2 10regional/global organisations inform their equitable institutional investments in climate smart food systems using CCAFS outputs. 20equitable national/subnational jurisdictions will have increased institutional investments in climate smart food systems 2015 2016 2022 2030 Flagship 4: Policies and institutions for climate-resilient food systems
  15. 15. Understand ToC Meaningful performance expectations for results (outputs, outcomes) Measure results, assess contribution of program to observed outcomes Report on performance achieved against expectations Practical mechanisms and tools to ensure balanced quantitative and qualitative monitoring A modular MEL system CCAFS M&E System Modules Harmoniza- tion of IPs & ToCs Indicators & Baselines Reflective Spaces and Activities Reporting Assessment & Bonus allocation Research on Institutional Transfor- mation Process
  16. 16. • Planning Work & $$ • System/ donor Reports • Synthesis at Flagship level • Synthesis at Regional level • Project Synthesis • Project evaluation • Deliverable ranking • Project annual reporting • Project annual planning Program Mgmt. Unit Flagship Program Leaders Regional Program Leaders Project Leaders Project SettingProject Setting PlanningPlanning ReportingReporting Synthesis & EvaluationsSynthesis & Evaluations Process supported by an Online Platform Iterative processes and built-in • Looped learning • System for adaptive management • Project evaluation (traffic light) • CRP mapped to outcome targets • Modular MEL Project Leaders Wider system CRP Managers CCAFS M&E System Modules Harmoniz ation of IPs & ToCs Indicators & Baselines Reflective Spaces and Activities Reporting Assessm ent & Bonus allocation Research on Institution al Transfor- mation
  17. 17. Dynamic program and project, planning and reporting modules CoA Technical features • Open source • Interoperable with or linkage to other data platforms • Modular
  18. 18. MEL Support Pack • To support strategy implementation through online platform for different users
  19. 19. Incentives – readiness of a system • The current CGIAR system’s incentive framework can be improved to make R4D via a TOC/IP approach work properly, including its financial system. • A portfolio approach: in which some projects revolve much more around science and others around engagement. Rarely will projects do both. Key is that science and engagement are happening within a regional or global conceptualized and coordinated programmatic manner. • Appropriate performance assessment required: Evaluation needs to be consistent with different measures required. Challenges & Opportunities
  20. 20. • Complexity and harmonization – of different levels/scales involved: System, Program, project, countries • System level – operationalization of the Strategy and Results Framework with a System MEL Framework and Monitoring Plan • Donor demands – outcome delivery promise aspirational vs. R4D reality • Balancing act of quantitative and qualitative measuring of outcomes • Results-based management for R4D - learning from the development sector – yet there are no off-the-shelf solutions • Accommodating negative outcomes and asking for those explicitly • .... Remaining Challenges
  21. 21. The “Evidencers” (the results agenda) Vs “The Complexers” (complex systems, emergent properties, messy) Oxfam blog, “From Poverty to Power”
  22. 22. Key reference documents • Schuetz, T, Förch, W, Thornton, P, Vasileiou, I. (accepted). Pathway to Impact: Supporting and evaluating enabling environments for research for development in Juha I. Uitto, Jyotsna Puri and Rob D. van den Berg, eds. Evaluating Climate Change for Sustainable Development. Springer: Dordrecht. Forthcoming in 2016. • CCAFS Planning and Reporting online Platform, Learning Brief No. 16, Nov. 2015 • CCAFS Reporting and Evaluation in a results-based management framework, Learning Brief No.15, reporting cycle, Jul. 2015 • Report to CO on RBM Trial, planning cycle, Dec. 2014 • Lessons and Insights from the CCAFS Results-Based Management Trial, RBM projects feedback/ experience, survey summary, Dec. 2014 • Lessons in theory of change from a series of regional workshops, Learning Brief No. 11, Dec. 2014 • CCAFS Theory of Change Facilitation Guide, Dec. 2014 • CCAFS RBM MEL strategy, framework from Jul. 2014