Like this? Share it with your network

Share

Me plenary final final

on

  • 457 views

2012 AC Wed Morning Plenary

2012 AC Wed Morning Plenary

Statistics

Views

Total Views
457
Views on SlideShare
248
Embed Views
209

Actions

Likes
0
Downloads
2
Comments
0

1 Embed 209

http://www.seepnetwork.org 209

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Me plenary final final Presentation Transcript

  • 1. Facilitating Systemic Change: Improving Evaluation to Improve Practice Elizabeth Dunn, Impact LLC SEEP Annual Conference November 7, 2012
  • 2. Why We Need Systemic Thinking• My journey from small farmers to microcredit to inclusive market systems…and back again• Practitioners, evaluators and donors need systemic thinking to meet new challenges: – Measuring participation under facilitation – Maintaining accountability with flexibility – Evaluating sustainability in evolving systems
  • 3. Beneficiaries and Boundaries• Identifying participants – Currently inconsistent• Understanding spillover – Good for impact – Bad for evaluators?• Relates to boundaries of system• 3 types of participants – direct, indirect, imitators Source: Outreach, Outcomes and Sustainability in Value Chain Projects by Creevey et al., Sept. 2011, USAID AMAP microREPORT #171.
  • 4. Credibility and Accountability • Issue: Systems change and projects must adapt • Accountability to donors – Not going away – But targets can constrain project effectiveness • Credibility of evidence – Baselines, control groups and attribution • Causal modeling – Useful and essential – Should be flexible
  • 5. Sustainability as Systemic Change• Markets as systems – Value chain as network of firms/actors relationships• Sustainability as systemic change – New, better relationships – Learning and adaptation – Broadening of benefits• Sustainability as an emergent property
  • 6. Evaluation Challenges1. Adopt consistent and comprehensive measures of participation under facilitation.2. Agree with donors on ways to demonstrate accountability under flexible interventions.3. Adapt causal modeling to guide practice and to evaluate evolving systems.4. Advance knowledge of sustainability through meaningful indicators of systemic change.
  • 7. Measuring Impacts in Market Systems:Rethinking the Current Paradigm Dr Shamim Bodhanya Academic Leader: Higher Degrees and Research , Graduate School of Business and Leadership, University of KwaZulu-Natal Chairperson – Institute of Natural Resources South Africa Tel: +27 31 260 1493 Email: bodhanyas1@ukzn.ac.za http://www.linkedin.com/in/shamimbodhanya
  • 8. Bounded Rationality “The capacity of the human mind for formulating and solving complex problems is very small compared with the size of the problem whose solution is required forobjectively, rational behaviour in the real worldor even for a reasonable approximation to such objective rationality.” (Simon, 1957, p 198)
  • 9. Systems Thinking
  • 10. Wicked problems (mess)• Unbounded• Ill-defined• Multiple, conflicting goals• Goals may also be ill-defined• Multiple perspectives, values
  • 11. Complex Adaptive System“A complex adaptive system (CAS) is a system comprised of heterogeneous agents that interact locally with eachother based on local schema, such that the behavior of the system arises as a result of feedback relationships between the agents, and the systemevolves as the schemata of the agents adapt based on the feedback.” Bodhanya, 2008
  • 12. Complex Adaptive System“A complex adaptive system (CAS) is a system comprised of heterogeneous agents that interact locally with eachother based on local schema, such that the behavior of the system arises as a result of feedback relationships between the agents, and the systemevolves as the schemata of the agents adapt based on the feedback.” Bodhanya, 2008
  • 13. Agents with schemata Artefacts Emergence Self Persistence Organisation Characteristics SensitiveEgality dependence of Complex Adaptive Systems Edge of Path Chaos dependence Co- evolution History Fitness Far from Landscapes equilibrium
  • 14. Immune system Nervous Markets system Open Cities source EXAMPLESEconomies Regions Industries Countries Firms
  • 15. Paradigm changing
  • 16. Rethinking the paradigm• Social systems are complex adaptive systems• Radically challenges our worldviews• Flat earth versus spherical earth• Ability to predict and control• What does this mean for human agency and volition?• Changes our entire conception of planning• Societal change• Markets This does not in anyway imply that human actors must be fatalistic.
  • 17. Arc de Triomphe
  • 18. Towards a way forward• We are on a journey• Context – Contingent on local conditions• Theoretical frameworks <-> Social Reality• Learning systems• Models – – Soft models – Narratives / Metaphor• Language is world constituting• Strategic conversations• Facilitation• What is measurement?• Agency in measurement tools• Artifacts• Generative relationships
  • 19. Thank you
  • 20. Simplifying complexitythrough systems thinkingPanel: Measuring impact in market systems (November 7th 2012) Richard Hummelbrunner ÖAR Regionalberatung Graz, Austria
  • 21. Systemic or Systematic? Systemic focus on the whole and the parts Three core dimensions:  Interrelationships  Perspectives  Boundaries Systematic = + focus on the parts, step-by-step
  • 22. Trivial or non-trivial simplification ?Input Transformation OutputInput Internal state(s) Output ? Context
  • 23. Consequences for monitoring• Regard interventions as social systems  Unit of observation: intervention and context  Observe relevant contextual factors (scanning) during implementation, in particular relevant actions of others  Look beyond intended routes and effects, avoid tunnel view, capture broader range of effects (irrespective of intentions)• Different approach towards deviations from plan  Do not per se regard as negative (‘correction reflex’)  Do not treat as isolated phenomena, but connect with intervention logic  Information to understand the internal dynamics and self- organising forces at work within target social system
  • 24. Linear or ‚circular‘ logic models ? Inputs Outputs ResultsNeeds /Problems Mechanisms ImpactIssues Context
  • 25. Single or multiple logics ?Inputs Outputs Results
  • 26. Measuring Impacts in Market Systems:Rethinking the Current Paradigm Dr Shamim Bodhanya Academic Leader: Higher Degrees and Research , Graduate School of Business and Leadership, University of KwaZulu-Natal Chairperson – Institute of Natural Resources South Africa Tel: +27 31 260 1493 Email: bodhanyas1@ukzn.ac.za http://www.linkedin.com/in/shamimbodhanya
  • 27. WATER Keeping land GLOBAL WARMING Increasing prices: productive •Fuel (oil) – supply too Communal Land – ownership •Fertilizer ownership entity L Cheap imports Commodity groups NAFU Tariff protection Organised (SASA, FSA, Grain NERPO WTONB. Markets: agric. /•Land SA, etc) Financial orgs:•Finance •Landbank•Physical Private sector: •Private banks•Labour Infrastructure •Primary production •Donors •Transport •Microfinance 80/20 •Processing - 20% of people produce 80% of Education & training National government the food Policy & programmes Research DEA, DWA, DAFF Communal Freehold land tenure tenure Policy MAFISA2nd economy entrepreneurs 1st economy $$ Provincial •LandbankMANY FARMERS FEW FARMERS & local •DAFF•Simple technology corporates •DFIs •Highly mechanised govt•Small markets (mainly family •Involves agro-processing& neighbours) •Large markets (diverse•Unprocessed goods consumers and products) Farmers
  • 28. Issues• Practitioners have a good sense of real world complexity• Overly formalistic tools• Attempt to straightjacket that experience into inappropriate tools• Unintended consequences• Multiple perspectives• Policy resistance• Boundary judgments• Bounded rationality
  • 29. Systemic change• Events – Pattern - Structure• Structure• Relationships between actors• Mental models• Incentives• Information flows• Changing feedback loops• Strength of the loops
  • 30. Considerations• Systemic M&E is conceptually challenging• ...yet we must stay rooted to practice• Rigour and credibility without overly mechanistic, formalistic approaches to M&E• Donor centric – How to get accountability while shifting the centre of gravity to change on the ground and systemic change
  • 31. IcebergBehaviour over time Events Patterns Structure
  • 32. What are the causes of …?
  • 33. Event-oriented View of the World Goals Problem Decision Results Situation Source: Sterman 2001
  • 34. Feedback View of the World Decisions Side Goals Effects EnvironmentGoals of OtherAgents Actions of Others Source: Sterman 2001
  • 35. Closed loop thinking Poor No training managementMicro-managing No Low morale support Lack of skills Poor Lack of resources Poorcommunication performance Conflict Role confusion
  • 36. Laws of the Fifth Discipline (Senge)• Today’s problems come from yesterday’s solutions• The harder you push the harder the system pushes back (policy resistance)• Behaviour grows better before it grows worse• The easy way out usually leads back in• The cure can be worse than the disease• Faster is slower• Cause and effect not closely related in time and space• Small changes can produce big results – but the areas of highest leverage are often the least obvious• You can have your cake and eat it too – but not at once• Dividing an elephant in half does not produce two small elephants• There is no blame
  • 37. Characteristics of Complex Adaptive Systems• Agents with schemata• Emergence• Self-organisation• Sensitive dependence on initial conditions• History• Path dependence• Far-from-equilibrium• Co-evolution• Fitness Landscapes• Edge of chaos• Artifacts• Persistence• Egalitarianism
  • 38. Simplifying complexitythrough systems thinkingPanel: Measuring impact in market systems (November 7th 2012) Richard Hummelbrunner ÖAR Regionalberatung Graz, Austria
  • 39. Consequences of a systemic view of effects• Replace impacts chains with configurations/networks, permitting to  Link elements at the same level  Connect different levels (e.g. Outputs - Results)  Capture reciprocal or feed-back relationships  Allocate activities or assumptions with effects  Show different strategy options, impact pathways  Identify leverage points for interventions• More refined modelling (if appropriate/useful)  E.g. Influence or Multiple Cause Diagrammes (all or partly)  Causal Loop Diagrammes (identify Feedback Loops)  Represent qualitative features (z.B. intensity, duration, delays)
  • 40. Example of outcome configuration (Source: New GIZ Impact Model) Outcome Outcome Outcome Outcome Outcome Outcome Outcome Outcome OutcomeOutcome Outcome
  • 41. Example of outcome configuration (Source: New GIZ Impact Model) Outcome Outcome Objective Outcome Outcome Outcome Outcome Outcome Outcome OutcomeOutcome Outcome
  • 42. Example of outcome configuration (Source: New GIZ Impact Model) Outcome Outcome Objective Outcome Outcome Outcome Outcome Outcome Outcome OutcomeOutcome Outcome
  • 43. Challenges for capturing effects in complicated and complex situations• Main challenges  Contribution to objectives through many factors / outputs  Difficult to establish clear causalities and relationships between outputs and further effects (results, impacts)  Temptation to attribute effects irrespective of contribution  Impact not appropriate to hold program actors accountable• Limited utility of many monitoring systems  Predominant focus on inputs or outputs (easy to capture)  Predominant use of quantitative indicators (easy to measure, capture only narrow part of reality)  Information on result / impact indicators comes often (too) late to change course during implementation
  • 44. Lines of influence in a program (example: EU Structural Fund - Programs)High Funding conditions Project development Project selection Influence actions of others Observe implementation of projects Outcome Logical Framework RBMLow Mapping INPUTS OUTPUTS RESULTS IMPACTS Project owners, Programme Actors partners, External factors
  • 45. The ‚Process Monitoring of Impacts‘ approach Theory-based monitoring approach making use of logic models  Focus on processes, which should lead to results / impacts  Logic models are considered as hypotheses (to be modified during implementation), not as ‘blue-prints  Perspectives and observations of various stakeholders are captured and reflected, applying use of systemic methods Core rationale of the approach  Provide information for programme actors as early as possible on the likeliness of achieving results/impacts  Particular emphasis on domains that can be influenced by them or for which they are responsible.
  • 46. Monitoring of change processes Basic assumptions for change (along a result chain) Inputs are used to Outputs are used Results will lead to produce outputs (by someone, in a (expected) impacts (= projects) specific manner) in a plausible to reach results mannerInputs Outputs Results Impacts
  • 47. Example of Logic model (Enterprise support Scheme) OUTPUT (TYPES) USE of OUTPUTS RESULT IMPACT Enterprises (esp. SME) Increased linkages between collaborate in networksAdvise for co-operation enterprises at regional level/ (also with large scale enterprises) I Enterprises (esp. SME) Sustainable stabilisation of carry out reorganisation enterprises processes Adaption to international competition Enterprises (esp. SME) New/ improved services, carry out product finding products and productionExternal expertise, processes processes IProcess consulting I Increase in employment/ Enterprises (esp. SME) new jobs introduce new technologies Enterprises (esp. SME) New contacts with clients, gain new markets clients, new orders new orders Enterprises (esp. SME)Services to sensitize for conceive innovation-/ prepare innovation-/innovations investment projects I Quantifiable indicator
  • 48. Result Based Management (RBM) Emerging tendencies• Shift focus of performance information  from outputs (goods and services produced) to outcomes (benefits)• Set performance expectations for outcomes  Clarify conceptional issues (function, purpose and location of targets)• Different approach to accountability  Influencing outcomes (not achieving them)• Assess contributions to outcomes (instead of claiming attribution)  Take account of other contributing factors, gestation period of outputs• Beware of straightforward links between performance and budgets/costs Managing for outcomes requires authority for managers to do so, i.e. more flexibility on activities, resources and outputs
  • 49. Adapting Logframes to deal with complexity:Differentiate effects in line with situation• Categorize outputs via portfolio matrix:  locate in one of the three domains: simple, complicated, complex  has implications for completing other elements of logframe• If outputs predominantly lie in the ‘complicated’ domain:  carefully identify indicators and assumptions to enable monitoring of unfolding practice, relevant factors and context conditions• If many (or majority of) outputs are considered to be ‘complex:  Identify indicators that allow documenting initial conditions and - in combination with assumptions - capturing emerging phenomenaNot all interventions (or parts thereof) are treated as ‘simple’!
  • 50. Output Portfolio High disagreement Some disagreement DEGREE OF AGREEMENT (Between Stakeholders) High agreement High agreement Some disagreement High disagreement DEGREE OF CERTAINTY ( About what to do)
  • 51. Types of indicators Type of Example Indicators Time Lagging Services delivered by CentreOutput Coincident Service Centre established Leading Building permits obtained, Work contracted