Multi-strand initiatives: using theory of change evaluations Karen Laing and Liz Todd, Newcastle University Alan Dyson, Kirstin Kerr, and Michael Wigelsworth, Manchester University

1. Multi-strand initiatives: using
theory of change evaluations
Alan Dyson: D.A.Dyson@manchester.ac.uk
Kirstin Kerr: Kirstin.Kerr@manchester.ac.uk
Karen Laing: k.j.c.laing@newcastle.ac.uk
Liz Todd: liz.todd@newcastle.ac.uk
Michael Wigelsworth: michael.wigelsworth@manchester.ac.uk

2. Introduction
• Aims
• The team – Alan Dyson, Kirstin Kerr, Karen Laing, Liz Todd, Michael
Wigelsworth
• Programme:
– 10.00 Introduction to theory of change evaluation
– 11.15 Coffee
– 11.30 Theory of change in action
– 12.00 Practical activity
– 12.30 Lunch
– 1.15 Using evidence
– 1.45 Practical activity
– 2.30 Coffee
– 3.00 Key issues: quantification; working with users; drawing conclusions
– 4.00 Finish

3. What is ‘theory of change’
approach to evaluation?
Professor Liz Todd
Liz.todd@ncl.ac.uk

4. Dyson & Kerr
Cummings, Dyson, Muijs, Papps,
Pearson, Raffo, Tiplady & Todd
Cummings Dyson &
Todd
Full service extended schools national
evaluation
Colleen Cummings, Alan Dyson, Ivy Papps,
Daniel Muijs, Diana Pearson, Carlo Ruffo, Lucy
Tiplady, Liz Todd: Newcastle University,
University of Manchester, Tecis Ltd
http://www.ncl.ac.uk/cflat/ESnetwork.htm

5. Theory of Change
A systematic and cumulative study of the links
between activities, outcomes and context of
the initiative
Fullbright-Anderson, Kubisch and Connell, 1998: 16

6. 6
Theories of Change
• Popularized in 1990s to capture complex
initiatives
• Outcomes-based
• Causal model
• Articulate underlying assumptions

7. 7
..and for the history buff:
Logic Models
• 30 year history
• Clear identification of goals (outcomes)
• First widespread attempt to depict program
components so that activities matched
outcomes

8. 8
What is a logic model?
Inputs Activities Outputs
Inter-
mediate
Outcomes
Long-term
Outcomes
Basic United Way format, 1996

9. Realistic evaluation
• Social programmes aim for some kind of social change
• Programmes ‘work’ by enabling participants to make different choices
through the programme program ‘mechanism’
• Making and sustaining different choices requires a change in participant’s
reasoning (for example, values, beliefs, attitudes, or the logic they apply to
a particular situation) and/or the resources (eg information, skills, material
resources, support) they have available to them. This combination of
‘reasoning and resources’ is what enables the program to ‘work’ and is
known as a programme ‘mechanism’.
• The contexts in which programmes operate make a difference to the
outcomes they achieve
• Programmes work differently in different contexts and through different
change mechanisms therefore programmes cannot simply be replicated
from one context to another and automatically achieve the same
outcomes.
• Good understandings about ‘what works for whom, in what contexts, and
how’ are, however, portable
• One of the tasks of evaluation is to learn more about ‘what works for
whom’, ‘in which contexts particular programs do and don’t work’, and
‘what mechanisms are triggered by what programmes in what contexts’
Pawson, R. and Tilley, N. (1997) Realistic Evaluation Sage

10. 10
How are they different?
Logic models graphically illustrate program components, and
creating one helps stakeholders clearly identify outcomes,
inputs and activities
Realistic evaluation emphasises outcomes produced via a
mechanis within a context
Context + Mechanism = Outcome
Theories of Change link outcomes and activities to explain HOW
and WHY the desired change is expected to come about

11. What is involved in theory of change….
• Qualitative and quantitative data collected
over time
• Identify early, intermediate and long-term
outcomes
• Theorise retrospectively and prospectively

12. Children’s Aid Society Community Schools in New York
…community schools are complex systems making
fundamental institutional changes, and the
means that events occur in many ways and on
many levels.
– It is, therefore
…necessary to look beyond standardized test
scores to understand the impact of community
schools.
Clark & Grimaldi, 2005

13. Chicago FSES Evaluation
Many efforts were underway in each school during
the FSSI period to improve student outcomes,
and no simple causal links can be drawn between
FSSI and improvement at the three schools.
Whalen, 2002

14. Constructing a theory of change

15. The starting situation
What it’s like now – and why
Strands of action
What we are going to do about it
Intended outputs
What we will need to do
differently Steps of change
How things will change
(for beneficiaries)
Intended outcomes
How the starting situation will
change for CYP

16. Steps of change

17. Lessons from experience
• The process is iterative…and human
• Start where the users are (situation, action,
outcomes…)
• Keep it simple
• Act as a critical friend
– Help to clarify
– Test the logic
– Test the evidence

18. Theory of change in action

21. Starting situation
– High levels of disadvantage on a wide range of social
indicators.
– A small local authority - can lack staff capacity to support
service reform/delivery.
– Communities which are locally very cohesive, but which
are also inward looking and ‘territorial’ – acts as barriers
to accessing services.
– A growing number of third generation unemployed, with
this helping to establish and perpetuate a non-aspirational,
non-learning, low-skills and workless culture.
– New Community Learning Centres
– Full range of extended services

22. Asking the right questions
• What is the situation you face?
• What needs to change?
• How will these changes be made?
• What actions will you take?
• What effect will those actions have? On
whom?
• How will you know if change is happening?
• What will happen for person A, person B etc

23. Community Learning Centres: a theory of change for their
community role
• The CLCs will develop a full range of extended services – either provided on site, or
facilitating access to services off site. (There will be sufficient capacity within the
voluntary and community sector, and within other services, to support this.)
• The CLCs will attract existing community groups to use their resources, and will be
approached by other services and organisations which want to develop new activities (or
continue established activities)
• The CLCs’ extended activities will be accessible and affordable to community members.
• Community members will engage in extended activities and use the CLCs’ facilities.
• The CLCs will become viable as not-for-profit organisations with regard to extended
service provision.
• Through engaging with activities either directly provided by, or co-ordinated or
commissioned by the CLCs, outcomes for families within the borough will improve.

24. Construct your own theory
• Identify a situation in which someone is taking
purposeful action…
• …or use ‘coming to a training day’, ‘getting
married’, ‘welfare reform’
• Build a possible theory of change for this
action

25. Using evidence

31. Some questions evidence is needed to answer
• CLC programme of activities (& what commissioned?)
• What community-based activities are out there?
• Which services/community groups are booking rooms?
• Which activities are new? Which are established and moving
from elsewhere?
• How much do activities cost? Who attends? Where do they
live?
• What revenue does the centre make from room hire? What –
if any – shortfall is there?
• What outcomes are there for children and families (what
outcomes are expected?)

32. I think there are some mixed messages about
is it really for the community? I mean it’s
physically in the middle of it, but there’s a
great concern that the community will be
priced out because I reckon each centre needs
to take £70,000 before it even breaks even.
Now £70,000 is a lot of bookings for squash
and badminton…

33. As far as I am aware, you cannot go and ‘pay and
play’ in the [CLC] sports facilities, so there’s this
lovely gym there and the kids and adults have got
their noses pressed against the window saying ‘can
we use this great gym?’. But you can’t just go and
use it, what you can do is hire the whole thing and
then use it. With great respect, not even the local
football team is that well organised and well
resourced to go and hire the whole of the gym to do
a fitness session. So we’ve got this thing that’s right
there in the community and for six months it hasn’t
been used by any member of the community.

34. We were having trouble with the all weather pitches
at the school with anti-social behaviour because the
people who were using it were the people who could
pay to use the outdoor sports facility and some of the
young people were throwing stones over and some of
the comments were ‘you don’t live round here, this
is ours and if we can’t afford to go on here or we’re
being kept out, you’re not playing on it either’.

35. Populating your own theory with
evidence
• What evidence could you use to populate your
theory of change?

36. Key issues: quantification

37. Theory of Change -
Quantification
Dr. Michael Wigelsworth
Manchester Institute of
Education
University of Manchester

38. Hello!
• Dr. Michael Wigelsworth
• Lecturer:
– M.Ed. Psychology of Education
– M.Sc Educational Research
• Special education and additional needs (SEAN)
– Mental health and wellbeing
– Social and emotional learning
• Programme Evaluation - Outcomes and impact
– Multi-component / complex school based programmes
– SEAL (primary and secondary)/ Achievement For All / PATHS to Success
• Pragmatic researcher
• Not a statistician

39. Quantification of ToC
• What?
• Why?
• How?
• Where /When?

40. What is quantification?
• The use of numerical data / quantitative
evidence to assess / evaluate a ToC
• A quantified theory of change becomes a
logic model

41. Quantification of ToC
• ToC  Logic model
• The theory of change is specified to the
point that individual components of the
model can be evidenced, and the casual
chains investigated
• To operationalise diffuse concepts into
definable factors - should be measurable

42. Same stages…
• Define input
• Explain effects
• Define outcomes
• Identifying moderators / mediators

43. ToC /Logic model
Effects OutcomesInput
Moderating
effects?
Mediating
effects?

44. Improved
pupils
mental
health
An example: Mental health in schools
Improved
attainment
Mental health
programme

45. Improved
pupils
mental
health
Taught curricula
Model
behaviour by
teachers
Parent materials
Reduction in in-
class exclusions
Improved
pupil’s self
reported
“readiness to
learn”
Improved KS
scores
Proximal (short term) outcomes Distal (long term)
An example: Mental health in schools
Consideration
of other factors

46. Why?(?)
• Makes ToC explicit / observable as an intervention
– Demonstrates impact – this is because the outcome is for everybody
– Move to transferability (poss. Replication)
• Demonstrate impact of the ToC specifically
– Examine the unique constructs (e.g. perceptions vs. outcomes)
– Provide empirical evidence of theoretical assumptions / chains
• Can test the relative strength of each causal chain (think baby and
bathwater)
• Identifies key effect mediators or moderators (may be relevant to
subgroups)
• Specifies differences between proximal and distal effects – manages
expectations of changes in outcomes

47. Why not?
• May be hard to capture “the effect”
• May be very hard to account for all important
moderating / mediating factors
• Requires accountability
• possibility of null or iatrogenic effects
• Requires levels of technical expertise

48. Where / When?
• A logic model can (should?) be applied
where measureable outcomes (i.e.
impact) is expected or needs to be
demonstrated
• This can apply (lesser /greater extent) at
all stages of research
– Theory  Long term implementation

49. Campbell, M., Fitzpatrick, R., Haines, A., Kinmouth, A., Sandersock, P., Spiegelhalter,
D. & Tyrer, P. (2000). Framework for design and evaluation of complex interventions
to improve health. British Medical Journal, 321, 694–696.

50. How?
• Dependent on quality…
– Full scale testing of a logic model for a complex
is a big ask
– But can you afford not to?
• Dependent on audience
– What do you want to be able to say?
– To who?

51. Take home
• A important question to ask is:
– How well specified is my ToC?
• How clearly established are my outcomes?
– Do I need to demonstrate impact?
• Is there value in investigating chains of
evidence?
– E.g. how did we arrive (not arrive) at impact?

52. Key issues: relationships with users

53. Multi-strand projects mean you are likely to need to
build relationships with many different people
• Strategic leads
• Operational staff
• External agencies/partners
• Communities
• Families and young people

54. How to build good relationships
• Understand the context people are working in
• Be aware of other demands on their time
• Understand the relationships they have with
each other
• Explore their hopes and fears about research
• Value their input and ideas

55. How a Theory of Change approach can
help
• Working together
• It is dialogical, involving conversation and
negotiation
• Enables continuous feedback
• Can feed into project planning
• Provides guidance about data collection
methods and self evaluation

56. Top tips!
• Be knowledgeable and respectful of those you work
with
• Expect trust to build gradually – earn it, not expect it
• Be willing to fit in with them – don’t expect them to
work to your demands!
• Factor in a ‘getting to know you’ visit
• Share information
• Theories are often implicit – projects need time and
support to work through this without feeling
inadequate or pressured

57. Impact on users is evidenced by a deputy head of a
primary school in Manchester, who said:
‘We’ve used your change document… it’s helped
us to ensure the impact of things in a way that
we hadn’t thought of’

58. Key issues: drawing conclusions
• Formative:
– ‘Initiative X is triggering changes a.b.c….which are
likely/unlikely to lead to outcomes 1, 2, 3….’
• Summative:
– ‘Initiative X triggered changes a.b.c….leading to outcomes
1, 2, 3….’
– ‘Initiative X also triggered changes l, m, n….which may yet
lead to outcomes 1, 2, 3….’
• But:
• ‘Initiative X# may not trigger changes a.b.c….or lead to
outcomes 1, 2, 3….’

59. Drawing conclusions
• Building a theory:
– Initiatives X, X#, X##...triggered similar changes
leading to similar outcomes’
– Initiatives of type X are likely to lead to similar
outcomes
• ToC and ‘what works’
• Using a D & R approach