This document discusses challenges and opportunities for taking a long-term view in evaluations. It notes that most evaluations are short-term but can provide a window into the past and future. Taking a complexity lens can help understand historical influences. Qualitative and quantitative methods each have different contributions. Features of complexity relevant to a long-term view include open systems influenced by history, multiple actors and stakeholders with changing perspectives, domains of stability, and path dependency where current behavior arises from past actions. Evaluators need tools to surface stakeholders' assumptions, which are often unconscious and historically formed. Case studies, qualitative interviewing, theory of change mapping, and process tracing provide opportunities to explore these long-term aspects.

1. Taking a long term view: challenges and
opportunities
Parallel session
Norwegian Evaluation society
19/9/19
Dione Hills Tavistock Institute and Cecan
With acknowledgements to Helen Wilkinson
Risk Solutions

2. Key points
Most evaluations are short term
But can provide a ‘window’ into past and future
Complexity lens helps understand historical influences
Who you interview and what you ask often as important as
methodology you use –
particularly when exploring causal relationships (and assumptions
about these)
Qual and quant methods have different things to contribute
Some evaluation approaches give better opportunities to explore
these aspects than others

3. Features of complexity (particularly)
relevant to taking a long term view
Open systems:
context includes
history.
Multiple actors, levers and hubs:
stakeholder perspectives change
system behaviour
Domains of
stability: have we
been here before?
Path dependency:
current behaviour
arises from the
sequence actions, or
decisions that preceded
them
Change over
time: how to
current
changes
related to the
past?

4. Stakeholder perceptions and
assumptions
Assumptions – taken for granted views on what is
important, what influences what, and how*
Are often unconscious*
Are historically formed*
Evaluators need tools for surfacing these
*Schein, E. H. (1985). Organizational culture and leadership. San
Francisco: Jossey-Bass Publishers.

5. Evaluator tools
Case studies – exploring the multiple factors
influencing current affairs
Qualitative interviewing
∙ Asking the right questions (e.g. what has influenced
decisions relating to current intervention?)
∙ Finding informants with a long term view (may not still be
around)
Theory of change mapping (useful for clarifying
context and assumptions – multiple maps helpful)
System mapping and rich pictures may be helpful if
include a historical perspective
Contribution analysis, QCA and Process Tracing all
provide good opportunity to ask right questions
Time series analysis: if past data available, consistent
and reliable

6. Components of Theory of Change
Money,
skills,
people
invested
Different
element of
programm
e
What has
been
produced:
(e.g.
number of
activities,
participant
s
recruited)
Short and
medium
term
results
(changes
in attitudes
behaviour,
and
services)
Linked to
overall
aims
Inputs Outputs Outcomes Impacts
Key issues
addressed
Other
factors
which
might
impact on
success
Context
and
issues
Wider context: i.e. other (contextual or historical) factors which may be having an
influence over outcome and impact data
Assumptions and Rational for interventions – help generate hypotheses to be
tested about causal links between inputs, outcomes and impacts

7. Healthy Living Centre example
5 year long evaluation of 300 local community centres set up to
improve health and wellbeing and reduce health inequalities.
We used theory of change, case studies including in depth interviews
and observation, followed by a survey, to surface
Prior experience of community work, local policy environment and
organisations in area which influenced speed of set up and
willingness of community to engage
Key individual and organisational stakeholders prior views on
causes of health inequality
These influenced their choice of activities, outputs and outcomes
Indications that many would return to prior situation once short term
funding ended (but data not available after end of evaluation)