Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Evaluation amidst complexity: 8 questions evaluators should ask

737 views

Published on

Ever been called into evaluate a project that wasn't working as expected? It might be that the implementers were trying to change behavior within a complex adaptive system. This presentation describes properties of complex adaptive systems, how they affect the success of a project and eight questions an astute evaluator can ask to help improve outcomes. Be sure to check the notes for more information.

Published in: Government & Nonprofit
  • Be the first to comment

  • Be the first to like this

Evaluation amidst complexity: 8 questions evaluators should ask

  1. 1. Evaluation amidst complexity Eight questions evaluators should ask Originally presented at Australasian Evaluation Society Annual Meeting, Melbourne, September 2015. Revised November 2015. Ann Larson, PhD www.socialdimensions.com.au
  2. 2. Complexity is often invoked as a reason for a project’s failures. My key message is that evaluators can use a complexity lens to understand and facilitate success – if we know what to look for.
  3. 3. Overview of presentation • Framing the problem • Explaining characteristics of complex adaptive systems • Eight questions to ask in an evaluation • Closing remarks
  4. 4. Framing the problem
  5. 5. Characteristics of projects introducing an evidence-informed intervention which are affected by complexity Signs of resistance or lack of support among some stakeholders or intended beneficiaries Slow progress in starting implementation Small or lack of adoption of new practices despite different strategies to change behaviour Not clear how to sustain gains when the project finishes
  6. 6. Context for evaluators Involved only at the end of the project, not at the beginning or middle. The task is to make a narrative to explain success or failure that is convincing to implementers and donors. We also must make recommendations that are grounded in evidence. There is rarely a logic model or an M&E system that is particularly helpful for the evaluation.
  7. 7. Explaining complex adaptive systems
  8. 8. Properties and behaviours of complex adaptive systems • Path dependency • Diversity • Multiple interdependent relationships • Self-organising emergent behaviours • External shocks • Non-linear outcomes
  9. 9. Path dependency
  10. 10. Diversity
  11. 11. External shocks & outside influences
  12. 12. Interdependent relationships
  13. 13. Self-organising and emergent behaviour
  14. 14. Non-linear outcomes
  15. 15. Harnessing complexity
  16. 16. An eclectic list of strategies for projects to harness rather than control complexity • Flexible, long term funding • New behaviour grounded in relevant history and saliency • Build coalitions around a vision for change • Understand motivations for behaviour change: introduce accountability and incentives • Start small, be flexible and experiment • Balance local initiative with quality standards • Monitor, review and act in a timely manner
  17. 17. Evaluation strategies
  18. 18. Evaluation designs for coping with CAS Suggestions from evaluators Developmental evaluations Push-back from clients Evaluation only budgeted at end, take too long, too expensive, too much like internal evaluation Intensive case studies to elucidate challenges and successes Lacks generalisability, doesn’t address original project objectives and logic models
  19. 19. 8 complexity-sensitive evaluation questions
  20. 20. Summary of complexity-sensitive evaluation questions Is there evidence that project designers and implementers …. 1. Were sensitive to history and current priorities? 2. Accommodate diversity in its design and implementation by employing different approaches depending on capacity and circumstance? 3. Understood dynamics of relevant behaviour? 4. Effectively influenced those dynamics? 5. Monitored, reviewed and took action based on regular information? 6. Recognised and embraced emergent behaviours that supported the intervention? 7. Responded to external change? 8. Were focused on what happens after the project?
  21. 21. 1) Has the project aligned itself with history and current priorities? Has that had an effect on its acceptability? Shape of the Busselton Sheds, planks from the old jetty
  22. 22. 2) Did the project take into account the differences in regions, local context, workforce or beneficiaries? How could it have been more effective if it had? States -2 Districts -5 Facilities-5 States -19 Districts -35 Facilities-53 States -19 Districts- 40 Facilities - 81 States -19 Districts – 206 Facilities - 371 States -19 Districts – 236 Facilities - 458 Different strategies employed in different states and facilities
  23. 23. In this sanitation project, households build their own latrines using standardized construction methods but individualised shelters.
  24. 24. 3) Did project designers and implementers understand relationships and dynamics in the workplace, families or agencies? This can be done by being part of the culture, involving the group whose behaviour you want to change, or running small scale pilots.
  25. 25. Projects within complex adaptive systems need to identify and work ‘with’ or work ‘around’ all of the important components, such as supply chains.
  26. 26. 4) What did they put into place to change those dynamics to change behaviour? Training alone is rarely effective in changing behaviour because there are so many other influences reinforcing the old behaviour.
  27. 27. Among strategies to change behaviour affected by local norms and reinforced by feedback loops are coaching, measures to increase accountability, rewarding performance and removing obstacles.
  28. 28. Encourage or enable communities to demand the new service
  29. 29. 5) Did the project routinely collect, review and respond to information about activities and behaviour change?
  30. 30. 6) Did the project recognise emergent behaviours and, if so, how did they respond? Local co-option of men’s health strategies
  31. 31. 7) How did the project respond and adapt in face of external shocks and changes? Why were they able to do this? Projects need good relationships and the capacity to be flexible to respond to changes in policies, funding or security issues through advocacy or problem solving at the appropriate level.
  32. 32. 8) How is the project preparing for what will happen when it ends? What aspects will government or local NGOs retain? Will the principal goal continue to inspire action? How is the implementing agency altering their approach?
  33. 33. Summary of complexity-sensitive evaluation questions Is there evidence that project designers and implementers …. 1. Were sensitive to history and current priorities? 2. Accommodate diversity in its design and implementation by employing different approaches depending on capacity and circumstance? 3. Understood dynamics of relevant behaviour? 4. Effectively influenced those dynamics? 5. Monitored, reviewed and took action based on regular information? 6. Recognised and embraced emergent behaviours that supported the intervention? 7. Responded to external change? 8. Were focused on what happens after the project?
  34. 34. Closing remarks These questions can be used in an interview guide with stakeholders or as a framework for analysis. We need more evidence on how to influence behaviour in complex adaptive systems. Evaluators can contribute to the evidence base.
  35. 35. A personal reading list Paina, L. and D. H. Peters (2012). "Understanding pathways for scaling up health services through the lens of complex adaptive systems." Health Policy and Planning 27(5): 365-373. Preskill H et al. Evaluating Complexity: Propositions for improving practice. FSG working paper, 2014. Everything by Lant Prichett but especially Pritchett, L. and F. de Weijer (2010). Fragile States: Stuck in a Capability Trap? World Development Report 2011 Background Paper. Washington DC, World Bank. Everything by Trish Greenhalgh but especially Greenhalgh, T., J. Russell, R. E. Ashcroft and W. Parsons (2011). "Why National eHealth Programs Need Dead Philosophers: Wittgensteinian Reflections on Policymakers’ Reluctance to Learn from History." Milbank Quarterly 89(4): 533-563. Axelrod, R., & Cohen, M. D. (1999). Harnessing Complexity: Organizational Implications of a Scientific Frontier Chandy, L., A. Hosono, H. Kharas and J. Linn, Eds. (2013). Getting to Scale: How to Bring Development Solutions to Millions of Poor People. Washington DC, Brookings Institutions Press. Sutton, R. I. and H. Rao (2014). Scaling Up Excellence: Getting to More without Settling for Less. New York, Random House Business Books.

×