This document discusses challenges and opportunities for improving evaluation of development programs to better account for systemic change. It notes the need for systemic thinking to address new challenges like measuring indirect participation and maintaining accountability with flexibility. Three key evaluation challenges are identified: 1) adopting consistent measures of indirect participation, 2) finding ways to demonstrate accountability under flexible interventions, and 3) adapting causal modeling to evaluate evolving systems and guide practice. The document advocates advancing knowledge of sustainability through meaningful indicators of systemic change beyond intended routes and effects.

1. Facilitating Systemic Change:
Improving Evaluation
to Improve Practice
Elizabeth Dunn, Impact LLC
SEEP Annual Conference
November 7, 2012

2. Why We Need Systemic Thinking
• My journey from small farmers to microcredit
to inclusive market systems…and back again
• Practitioners, evaluators and donors need
systemic thinking to meet new challenges:
– Measuring participation under facilitation
– Maintaining accountability with flexibility
– Evaluating sustainability in evolving systems

3. Beneficiaries and Boundaries
• Identifying participants
– Currently inconsistent
• Understanding spillover
– Good for impact
– Bad for evaluators?
• Relates to boundaries
of system
• 3 types of participants
– direct, indirect, imitators Source: Outreach, Outcomes and Sustainability
in Value Chain Projects by Creevey et al., Sept.
2011, USAID AMAP microREPORT #171.

4. Credibility and Accountability
• Issue: Systems change
and projects must adapt
• Accountability to donors
– Not going away
– But targets can constrain
project effectiveness
• Credibility of evidence
– Baselines, control groups
and attribution
• Causal modeling
– Useful and essential
– Should be flexible

5. Sustainability as Systemic Change
• Markets as systems
– Value chain as network of
firms/actors relationships
• Sustainability as systemic
change
– New, better relationships
– Learning and adaptation
– Broadening of benefits
• Sustainability as an
emergent property

6. Evaluation Challenges
1. Adopt consistent and comprehensive
measures of participation under facilitation.
2. Agree with donors on ways to demonstrate
accountability under flexible interventions.
3. Adapt causal modeling to guide practice and
to evaluate evolving systems.
4. Advance knowledge of sustainability through
meaningful indicators of systemic change.

7. Measuring Impacts
in Market Systems:
Rethinking the Current Paradigm
Dr Shamim Bodhanya
Academic Leader: Higher Degrees and Research
, Graduate School of Business and Leadership, University of KwaZulu-Natal
Chairperson – Institute of Natural Resources
South Africa
Tel: +27 31 260 1493
Email: bodhanyas1@ukzn.ac.za
http://www.linkedin.com/in/shamimbodhanya

8. Bounded Rationality
“The capacity of the human mind for
formulating and solving complex problems is
very small compared with the size of the
problem whose solution is required for
objectively, rational behaviour in the real world
or even for a reasonable approximation to such
objective rationality.”
(Simon, 1957, p 198)

9. Systems Thinking

10. Wicked problems (mess)
• Unbounded
• Ill-defined
• Multiple, conflicting goals
• Goals may also be ill-defined
• Multiple perspectives, values

11. Complex Adaptive System
“A complex adaptive system (CAS) is a
system comprised of heterogeneous
agents that interact locally with each
other based on local schema, such that
the behavior of the system arises as a
result of feedback relationships
between the agents, and the system
evolves as the schemata of the agents
adapt based on the feedback.”
Bodhanya, 2008

12. Complex Adaptive System
“A complex adaptive system (CAS) is a
system comprised of heterogeneous
agents that interact locally with each
other based on local schema, such that
the behavior of the system arises as a
result of feedback relationships
between the agents, and the system
evolves as the schemata of the agents
adapt based on the feedback.”
Bodhanya, 2008

13. Agents
with
schemata
Emergence
Self
Organisatio
n
Sensitive
dependence
Path
dependence
History
Far from
equilibrium
Fitness
Landscapes
Co-
evolution
Edge of
Chaos
Egality
Persistence
Artefacts
Characteristics
of Complex
Adaptive
Systems

15. Immune
system
Nervous
system
Cities
Regions
Countries
Firms
Industries
Economies
Open
source
Markets
EXAMPLES

16. Paradigm changing

17. Rethinking the paradigm
• Social systems are complex adaptive systems
• Radically challenges our worldviews
• Flat earth versus spherical earth
• Ability to predict and control
• What does this mean for human agency and
volition?
• Changes our entire conception of planning
• Societal change
• Markets
This does not in anyway imply that
human actors must be fatalistic.

23. Arc de Triomphe

24. Towards a way forward
• We are on a journey
• Context – Contingent on local conditions
• Theoretical frameworks <-> Social Reality
• Learning systems
• Models –
– Soft models
– Narratives / Metaphor
• Language is world constituting
• Strategic conversations
• Facilitation
• What is measurement?
• Agency in measurement tools
• Artifacts
• Generative relationships

25. Thank you

26. Simplifying complexity
through systems thinking
Panel: Measuring impact in market
systems (November 7th 2012)
Richard Hummelbrunner
ÖAR Regionalberatung
Graz, Austria

27. Systemic or Systematic?
 Systemic
focus on the whole
and the parts
Three core dimensions:
 Interrelationships
 Perspectives
 Boundaries
 Systematic
focus on the parts,
step-by-step
=
+

28. Trivial or non-trivial simplification ?
Transformation
Internal state(s)
Input
Input Output ?
Output
Context

29. Consequences for monitoring
• Regard interventions as social systems
 Unit of observation: intervention and context
 Observe relevant contextual factors (scanning) during
implementation, in particular relevant actions of others
 Look beyond intended routes and effects, avoid tunnel
view, capture broader range of effects (irrespective of
intentions)
• Different approach towards deviations from plan
 Do not per se regard as negative (‘correction reflex’)
 Do not treat as isolated phenomena, but connect with
intervention logic
 Information to understand the internal dynamics and self-
organising forces at work within target social system

30. Linear or ‚circular‘ logic models ?
Inputs Outputs Results
Mechanisms
Context
Needs /
Problems
Issues
Impact

31. Inputs Outputs Results
Single or multiple logics ?

32. Measuring Impacts
in Market Systems:
Rethinking the Current Paradigm
Dr Shamim Bodhanya
Academic Leader: Higher Degrees and Research
, Graduate School of Business and Leadership, University of KwaZulu-Natal
Chairperson – Institute of Natural Resources
South Africa
Tel: +27 31 260 1493
Email: bodhanyas1@ukzn.ac.za
http://www.linkedin.com/in/shamimbodhanya

33. 2nd economy
MANY FARMERS
•Simple technology
•Small markets (mainly family
& neighbours)
•Unprocessed goods
Communal
land tenure
1st economy $$
FEW FARMERS
•Highly mechanised
•Involves agro-processing
•Large markets (diverse
consumers and products)
entrepreneurs
corporates
Freehold
tenure
NB. Markets:
•Land
•Finance
•Physical
•Labour
Communal
ownership L
Land – ownership
entity
Keeping land
productive
80/20
- 20% of people
produce 80% of
the food
Infrastructure
Private sector:
•Primary production
•Transport
•Processing
Organised
agric. /
National government
Policy & programmes
DEA, DWA, DAFF
NAFU
NERPO
Commodity groups
(SASA, FSA, Grain
SA, etc)
Provincial
& local
govt
Policy
Farmers
Financial orgs:
•Landbank
•Private banks
•Donors
•Microfinance
WATER
GLOBAL WARMING Increasing prices:
•Fuel (oil) – supply too
•Fertilizer
MAFISA
•Landbank
•DAFF
•DFIs
Cheap imports
Tariff protection
WTO
Education & training
Research

34. Issues
• Practitioners have a good sense of real world
complexity
• Overly formalistic tools
• Attempt to straightjacket that experience into
inappropriate tools
• Unintended consequences
• Multiple perspectives
• Policy resistance
• Boundary judgments
• Bounded rationality

35. Systemic change
• Events – Pattern - Structure
• Structure
• Relationships between actors
• Mental models
• Incentives
• Information flows
• Changing feedback loops
• Strength of the loops

36. Considerations
• Systemic M&E is conceptually challenging
• ...yet we must stay rooted to practice
• Rigour and credibility without overly
mechanistic, formalistic approaches to M&E
• Donor centric – How to get accountability
while shifting the centre of gravity to change
on the ground and systemic change

37. Iceberg
Events
Patterns
Structure
Behaviour over
time

38. What are the causes of …?

39. Event-oriented View of the World
Goals
Situation
Problem Decision Results
Source: Sterman 2001

40. Actions of
Others
Feedback View of the World
Goals
Decisions
Environment
Goals of
Other
Agents
Source: Sterman 2001
Side
Effects

41. Closed loop thinking
Poor
communication
Poor
performance
Conflict
Role confusion
No
support
Lack of resources
Poor
management
Low morale
Micro-managing
No training
Lack of skills

42. Laws of the Fifth Discipline (Senge)
• Today’s problems come from yesterday’s solutions
• The harder you push the harder the system pushes back (policy resistance)
• Behaviour grows better before it grows worse
• The easy way out usually leads back in
• The cure can be worse than the disease
• Faster is slower
• Cause and effect not closely related in time and space
• Small changes can produce big results – but the areas of highest leverage
are often the least obvious
• You can have your cake and eat it too – but not at once
• Dividing an elephant in half does not produce two small elephants
• There is no blame

43. Characteristics of Complex Adaptive
Systems
• Agents with schemata
• Emergence
• Self-organisation
• Sensitive dependence on initial conditions
• History
• Path dependence
• Far-from-equilibrium
• Co-evolution
• Fitness Landscapes
• Edge of chaos
• Artifacts
• Persistence
• Egalitarianism

44. Simplifying complexity
through systems thinking
Panel: Measuring impact in market
systems (November 7th 2012)
Richard Hummelbrunner
ÖAR Regionalberatung
Graz, Austria

45. • Replace impacts chains with configurations/networks, permitting to
 Link elements at the same level
 Connect different levels (e.g. Outputs - Results)
 Capture reciprocal or feed-back relationships
 Allocate activities or assumptions with effects
 Show different strategy options, impact pathways
 Identify leverage points for interventions
• More refined modelling (if appropriate/useful)
 E.g. Influence or Multiple Cause Diagrammes (all or partly)
 Causal Loop Diagrammes (identify Feedback Loops)
 Represent qualitative features (z.B. intensity, duration, delays)
Consequences of a systemic view of effects

46. Example of outcome configuration
(Source: New GIZ Impact Model)
Outcome
Outcome
Outcome
Outcome
Outcome
Outcome
Outcome
Outcome
Outcome
Outcome
Outcome

47. Example of outcome configuration
(Source: New GIZ Impact Model)
Outcome
Outcome
Outcome
Outcome
Outcome
Outcome
Outcome
Objective
Outcome
Outcome
Outcome
Outcome

48. Example of outcome configuration
(Source: New GIZ Impact Model)
Outcome
Outcome
Outcome
Outcome
Outcome
Outcome
Outcome
Outcome
Objective
Outcome
Outcome
Outcome

49. Challenges for capturing effects
in complicated and complex situations
• Main challenges
 Contribution to objectives through many factors / outputs
 Difficult to establish clear causalities and relationships
between outputs and further effects (results, impacts)
 Temptation to attribute effects irrespective of contribution
 Impact not appropriate to hold program actors accountable
• Limited utility of many monitoring systems
 Predominant focus on inputs or outputs (easy to capture)
 Predominant use of quantitative indicators (easy to
measure, capture only narrow part of reality)
 Information on result / impact indicators comes often (too)
late to change course during implementation

50. Lines of influence in a program
(example: EU Structural Fund - Programs)
High
Low
INPUTS OUTPUTS RESULTS IMPACTS
Project owners,
partners, External
factors
Programme Actors
Funding conditions
Project development
Project selection
Observe implementation
of projects
Influence actions of others
Logical Framework RBM
Outcome
Mapping

51. The ‚Process Monitoring of Impacts‘ approach
 Theory-based monitoring approach making use of logic models
 Focus on processes, which should lead to results / impacts
 Logic models are considered as hypotheses (to be modified
during implementation), not as ‘blue-prints
 Perspectives and observations of various stakeholders are
captured and reflected, applying use of systemic methods
 Core rationale of the approach
 Provide information for programme actors as early as
possible on the likeliness of achieving results/impacts
 Particular emphasis on domains that can be influenced by
them or for which they are responsible.

52. Monitoring of change processes
Basic assumptions for change (along a result chain)
Inputs are used to
produce outputs
(= projects)
Outputs are used
(by someone, in a
specific manner)
to reach results
Results will lead to
(expected) impacts
in a plausible
manner
Inputs Outputs Results Impacts

53. External expertise,
Process consulting
Services to sensitize for
innovations
Enterprises (esp. SME)
collaborate in networks
(also with large
enterprises)
Enterprises (esp. SME)
carry out reorganisation
processes
Enterprises (esp. SME)
carry out product finding
processes
Enterprises (esp. SME)
introduce new
technologies
Enterprises (esp. SME)
gain new markets
Enterprises (esp. SME)
conceive innovation-/
investment projects
Increased linkages between
enterprises at regional level/
scale
Sustainable stabilisation of
enterprises
New/ improved services,
products and production
processes
Adaption to international
competition
Increase in employment/
new jobs
New contacts with clients,
new orders
External expertise,
Process consulting
Services to sensitize for
innovations
Enterprises (esp. SME)
carry out reorganisation
processes
Enterprises (esp. SME)
carry out product finding
processes
Enterprises (esp. SME)
introduce new
technologies
Enterprises (esp. SME)
gain new markets
Enterprises (esp. SME)
prepare innovation-/
investment projects
Sustainable stabilisation of
enterprises
New/ improved services,
products and production
processes
Adaption to international
competition
Increase in employment/
new jobs
New contacts with clients,
new orders
Advise for co-operation
I
I
I
OUTPUT (TYPES) USE of OUTPUTS RESULT IMPACT
I Quantifiable indicator
Example of Logic model (Enterprise support Scheme)

54. Result Based Management (RBM)
Emerging tendencies
• Shift focus of performance information
 from outputs (goods and services produced) to outcomes (benefits)
• Set performance expectations for outcomes
 Clarify conceptional issues (function, purpose and location of targets)
• Different approach to accountability
 Influencing outcomes (not achieving them)
• Assess contributions to outcomes (instead of claiming attribution)
 Take account of other contributing factors, gestation period of outputs
• Beware of straightforward links between performance and budgets/costs
 Managing for outcomes requires authority for managers to do so,
i.e. more flexibility on activities, resources and outputs

55. Adapting Logframes to deal with complexity:
Differentiate effects in line with situation
• Categorize outputs via portfolio matrix:
locate in one of the three domains: simple, complicated, complex
has implications for completing other elements of logframe
• If outputs predominantly lie in the ‘complicated’ domain:
carefully identify indicators and assumptions to enable
monitoring of unfolding practice, relevant factors and context
conditions
• If many (or majority of) outputs are considered to be ‘complex:
Identify indicators that allow documenting initial conditions and -
in combination with assumptions - capturing emerging
phenomena
Not all interventions (or parts thereof) are treated as ‘simple’!

56. Output Portfolio
DEGREE OF CERTAINTY ( About what to do)
DEGREE
OF
AGREEMENT
(Between
Stakeholders)
High agreement Some disagreement High disagreement
High
agreement
Some
disagreement
High
disagreement

57. Types of indicators
Output
Type of
Indicators
Lagging
Coincident
Leading
Time
Example
Services delivered by Centre
Service Centre established
Building permits obtained,
Work contracted