Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

FCAS M&E Seminar


Published on

David Fleming held a seminar on monitoring and evaluation in conflict-affected environments at the Post-war Reconstruction and Development Unit (PRDU), University of York.

Published in: Data & Analytics
  • Be the first to comment

  • Be the first to like this

FCAS M&E Seminar

  1. 1. Monitoring and Evaluation in Fragile and Conflict-Affected Contexts: The challenges of measurement David Fleming, Senior Consultant, Itad Date: 28th January 2015
  2. 2. Seminar Outline 1. Introducing Itad: Life as an M&E consultant 2. Introducing/recapping M&E: Why monitor and evaluate and why important in FCAS? 3. Theories of change: what they are, why they are useful and challenges in FCAS 4. M&E approaches and methods: how to monitor and evaluate in FCAS; examples from peacebuilding and humanitarian work
  3. 3. Learning objectives 1. Come away with a better understanding of why we do M&E and why it’s particularly important in FCAS 2. Learn about and put into practice some of the most important M&E methods and tools for FCAS 3. Be able to better identify the challenges of doing M&E in FCAS and how to overcome these 4. Everyone to leave the room with a burning desire to get involved in M&E at some point in the future!
  4. 4. Life as an M&E consultant…
  5. 5. 2. Introducing M&E: Why Monitor and Evaluate? “After decades in which development agencies have disbursed billions of dollars for social programs, and developing country governments and nongovernmental organizations (NGOs) have spent hundreds of billions more, it is deeply disappointing to recognize that we know relatively little about the net impact of most of these social programs” ‘When will we ever learn?’ Evaluation Gap Working Group, Center for Global Development 2006
  6. 6. • Monitoring: “Collection of data with which managers can assess extent to which objectives are being achieved” (World Bank) – Purpose: Collect information on programme outputs and outcomes to track and improve performance and results • Evaluation: “Determination of the value of a project, programme or policy” (World Bank) – Purpose: evidence-based decisions, accountability, transparency, lesson learning – Types: project, programme, policy, organisation, sector, theme, formative, summative, impact…
  7. 7. Why is M&E important in FCAS? 1. Development trends in FCAS • By 2015, 50% of world’s poor will live in fragile states (OECD); by 2030 it might be two thirds (Brookings) • Support to conflict, violence and fragility becoming a key priority for most major donors • ODA to fragile states is falling in quantity but number of actors multiplying (OECD) • DFID has been scaling up support to FCAS (commitment to increase to 30% of ODA by 2015) • DFID strategies include BSOS, cross-Whitehall CSSF, and the ‘Beyond Aid’ agenda
  8. 8. Why is M&E important in FCAS? 2. Increasing emphasis on transparency, accountability and fiduciary risk • Higher risk to investments in terms of results, security and fiduciary risk 3. More limited evidence base – need for lesson learning and evidence of what works • Support evidence-informed decisions and better programming by knowing what works and doesn’t and why and in which contexts
  9. 9. What are the biggest challenges? Risk of exacerbating conflict Hawthorne effect Insecurity Political objectives Longer-term nature of results Measurement challenges Vulnerability to biases Lack of existing data Poor data reliability Poor data accessibility Unpredictable chains of causation Complex and dynamic contexts
  10. 10. M&E within the programme cycle Identification Problem analysis Appraisal Evidence of what works Design Most cost- effective intervention/s Implementation With M&E built in from outset Completion Measure results – did it work? Post Completion Feed lessons into future decisions Lesson Learning and Feedback
  11. 11. Challenges of programming in FCAS Identification Problem analysis contested Appraisal Little robust data and research. No time Design Little evidence to assess cost effectiveness. Political imperatives Implementation Great hurry. M&E lags behind. No baselines/ measurement strategies Completion Not enough data to say. No inclination to admit failure Post Completion Not enough results published/ stored/ synthesisd. Disagreement No knowledge management/sharing and lots of uncoordinated actors
  12. 12. 3. Theories of Change (ToC)
  13. 13. Why are ToCs useful for M&E? A ToC is an iterative and collaborative process for thinking through how a programme is expect to work within the context of the broader system. It should create the space for critical reflection and learning and be adjusted and iterated over time. • Links to assumptions box in LF, but goes beyond this in focusing on iterating through learning shared mental models of how change happens • Important for developing M&E strategy – test key links and assumptions (intellectual leaps) in the causal chain over the life of the programme • Important for evaluability – provides foundation for a theory-based evaluation • Important to talk of ‘theories’ not ‘theory’ – i.e. to recognise and manage a range of theories and multiple drivers of change • Not a tick-box exercise or management tool like the LF but a way of working and thinking – it’s primarily a process rather than a product
  14. 14. What are the pitfalls in FCAS? • Time and resource-consuming – so they can often be poorly conceived/ too vague • Poorly understood/used – as linear tick-box exercise rather than iterative approach • Oversimplification of complex contextual (e.g. conflict) factors – reflexivity and feedback loops in complex conflict systems – black swan idea • Absence of/poor conflict analysis – must underpin project design • Difficulties in evidence gathering/data collection – conflict environments are often data rich but information poor – insecurity, staff turnover • Difficulties of working with and aiming to influence a range of actors • Unpacking chains of cause and effect in FCAS can be very difficult • Death by diagram • Funnel of attrition
  15. 15. The funnel of attritionOnly these people may experience improved outcomes
  16. 16. 4. M&E approaches and methods Recent explosion of new and innovative approaches to monitoring and evaluation: 1. Use of mobile technology and ICTs for data collection and analysis – e.g. Ushahidi 2. Influence of complexity science – PDIA, DDD – enabling environment for experimentation 3. Remote monitoring and verification 4. Rigorous evaluation/impact evaluation designs
  17. 17. Why evaluate? • White and Waddington (2012): ‘The use of the systematic reviews methodology is comparatively new among social scientists in the international development field, but has grown rapidly in the last 3 years...To date, there has not been a strong tradition of using rigorous evidence in international development. The evidence bar has been rather low, with many policies based on anecdote and ‘cherry picking’ of favourable cases’.
  18. 18. Why evaluate? • Accountability and lesson-learning – Accountability to taxpayers and beneficiaries – Understanding what works, why, where and for whom to underpin evidence-based programming – Priority to evaluate interventions with a weak evidence base • Inform scale up of an intervention or transfer to another context • Make mid-course corrections • To support spending decisions
  19. 19. What is impact evaluation? “Impact evaluation is a with versus without analysis: what happened with the programme (a factual record) compared to what would have happened in the absence of the programme (which requires a counterfactual)” (White, 2013) “Impact evaluation aims to demonstrate that development programmes lead to development results, that the intervention has a cause and effect” (Stern et al. 2012) • Attribution analysis to understand what difference a programme made • Counterfactual construction through experimental/quasi-experimental methods for large n (comparison groups); causal chain analysis for small n • Theory-based impact evaluation – in ideal world, an RCT should be embedded in a broader theory-based design that addresses questions across the causal chain (White, 2013) • Causal chain analysis – rigorous empirical assessment of causal mechanisms and the assumptions that underlie the causal chain
  20. 20. How do we estimate impact?
  21. 21. How do we estimate impact?
  22. 22. Pros and cons of RCTs • Pros: RCTs are the “gold standard” for addressing attribution when an ex ante design is possible with a large number of units of assignment • BUT MAJOR DRAWBACKS, ESPECIALLY IN FCAS – Not suited to complex development pathways with multiple non-linear causal factors – Less appropriate where hard to identify comparison groups – threat to validity – When extrapolated from their context, RCT findings lose claims to rigour (Pritchett and Sandefur, 2013)
  23. 23. How best to evaluate in FCAS? In increasing order of robustness: • Use of evaluation framework and robust approach to evidence assessment – e.g. humanitarian evaluations • Use of theories of change and contribution analysis to test causation and assumptions • Realist evaluation design looking at how different mechanisms operate in contexts
  24. 24. Using an evaluation framework Questions Theory/ Approach Methods Tools Establishing a framework for the evaluation provides a consistent and systematic means to designing the evaluation, collating and analysing the existing evidence and the new data created, and generating and interpreting the results. (Magenta Book para 6.1)
  25. 25. Theory or approach •Results-oriented •Theory-based •Participatory/ empowerment •Utilization-focused Methods •Qualitative •Case study •Experimental •Value for Money •Contribution analysis Tools •Document review •Key informant interview •FGD •Direct observation •Questionnaire survey •Participatory data collection •SWOT •Forcefield •Stakeholder analysis •Ranking and scoring •[Types of analysis; CBA; VfM; QCA; etc]] Methods define which tools and how to use them
  26. 26. Evaluating peacebuilding • Most useful definition of impact – understand effects of intervention on conflict drivers • Conflict analysis is critical – understand/test relevance of intervention to conflict drivers • Use of ToC to understand/test assumptions about how intervention contributes to change • Experimental approaches usually not useful – better to look at contribution
  27. 27. M&E Group Exercise • Split into 4 groups • 2 groups will be responsible for designing an outline M&E system for a peacebuilding programme • 2 groups will be responsible for designing an outline proposal to do an external evaluation of the same programme
  28. 28. Further Reading Literature on M&E approaches and methods • L. Morra Imas, Rist, R., The Road to Results (World Bank, 2009) • S. Funnell, Rogers, P., Purposeful Program Theory (Wiley, 2011) • E. Stern et al., ‘Broadening the range of designs and methods for impact evaluation’, DFID working paper 38, April 2012 • H. White, Phillips, D., ‘Addressing Attribution of cause and effect in small n impact evaluations’, 3ie Working Paper 15, June 2012 • G.Westhorp, ‘Realist impact evaluation: an introduction’, September 2014 Literature on M&E with specific reference to FCAS • DFID, ‘Results in Fragile and Conflict-affected States and Situations’, 2012 • DFID, ‘Back to Basics, A compilation of best practices in design, monitoring and evaluation in fragile and conflict-affected environments,’ March 2013 • L. Schreter, Harmer, A., Delivering Aid in Highly Insecure Environments, 2013 • S. Herbert, ‘Perceptions surveys in fragile and conflict-affected states’, GSDRC Helpdesk Research Report, March 2013 • DFID, ‘Evaluating impacts of peacebuilding interventions’, May 2014 • J. Puri et al. ‘What methods may be used in impact evaluations of humanitarian assistance’, 3ie working paper 22, December 2014
  29. 29. Thank you for listening - any questions?