CONUL Teaching and Learning Annual Seminar. Long Room Hub, Trinity College Dublin, 16th November, 2017.

1. David Streatfield
Impact Consultant
Bill and Melinda Gates Foundation

2. “ … any effect of the service (or of an
event or initiative) on an individual, group
or community.”
 may be positive or negative
 may be intended or accidental
 may affect users, library staff, senior
managers – others
Markless and Streatfield, 2013

3. The impact can show itself:
 in individual cases, or
 more general changes, such as shifts in:
• quality of life: e.g. self-esteem; confidence;
feeling included; work or social engagement
• educational and other outcomes: e.g. skills
acquired; educational attainment; levels of
knowledge.

4. Changes in:
 What people do (behaviour)
 How they do things
 How much they know
 Their attitudes (e.g. confidence;
valuing library staff!)

5.  Phase 1: Stealing from educational evaluation
 Phase 2: Stealing from international
development evaluation
 Phase 3: Stealing from the programme theory
community

6.  The simple logic model
 Hypothesis refuting ... and rewriting
 Illuminative evaluation
 Impact evaluation
 The problem of complexity

7. Some people who may be affected by library and
information services:
 History students
 Library staff who teach
 Mathematics teachers and learners
 Music students
 Ph D candidates
 People who wish to be (more) employable
 School archive users
 Students becoming more digitally literate
 Students with intellectual disabilities

8.  Recognising complexity: the complex logic
model
where the paths from action to impact are
complex, with disproportionate
relationships (in which, at critical levels,
a small change can make a big difference)
 Emergent evaluation
emergent impacts (which cannot readily
be specified at the outset).
Rogers, 2008

9.  In the 1990s education and development
evaluation (especially in the USA) was
focused on attribution studies based on
experimental designs and randomly
controlled trials.
 Now (especially in Europe) evaluators seek
to gauge the impact of complex change
programmes, and new services in complex
settings using rigorous theory-based
evaluations.

10.  These:
◦ make programme assumptions and
implementation issues transparent
◦ enable evaluators to recognise different
levels and types of contributions to
change, and
◦ pave the way towards credible causal
claims, not relying on direct attribution.

11. Various theory-based approaches have been
developed but these all start with articulation
of a Theory of Change. This underpins a
programme or intervention, to create a
framework for focusing and conducting its
evaluation and explaining any effects.

12. A ToC describes how and why a desired change is
expected to happen in a particular context. It
focuses on mapping out or ‘filling in’ (what has
been described as the ‘missing middle’) between
what a programme does (its activities,
mechanisms or interventions) and how these can
lead to objectives being achieved.
HOW?
It identifies the desired long-term objectives and
then works back from these to identify how and
why people expect things to work to achieve
change.

13.  Plausible - do evidence and common sense
suggest that the activities will lead to the desired
outcomes?
 Doable – are the resources be available to carry
out the initiative?
 Testable – is the ToC specific and complete
enough to track its progress in credible ways?
“ A good ToC is embedded in the context of the
intervention and is developed incorporating the
perspectives of key stakeholders, beneficiaries
and the existing relevant research.”
Mayne (2012)

14.  Complexity is now the norm and complex situations and
initiatives require flexible, agile evaluation approaches to
deal with multiple factors, relationships and layers.
 Rigorous and systematic early articulation of the ways in
which the programme expects to bring about clearly
identified changes in individuals and communities at a
range of different levels (programme theory)
 Systematic use of relevant research literature from
appropriate disciplines to inform the ToC.
 Using the ToC to carefully focus data collection activities,
increasingly by using mixed methods
Greene, 2008

15.  Theory-based evaluation approaches offer an
important change in focus for the LIS field, in
which attribution is hard to prove.
 Adopting these will make it easier for LIS
leaders to be seen as credible. They will less
often be required to make simplistic and
unconvincing attribution claims for their
services.
 However, more credible LIS evaluation requires
a significant and sustained investment of time
and resources.

16.  Greene, J.C. (2008), Is mixed methods social Inquiry a
distinctive methodology? Journal of Mixed Methods
Research, 2, 1, 7-22; article first published online: June
29, 2016 https://doi.org/10.1177/1558689807309969
 Mayne, J. (2012), Contribution analysis: coming of age?
Evaluation, 18, 3, 270-280
https://www.researchgate.net/publication/254091562_
Contribution_Analysis_Coming_of_Age
 Rogers, P. J. (2008). Using programme theory to
evaluate complicated and complex aspects of
interventions. Evaluation, 14(1), 29–48.
http://dx.doi.org/10.1177/1356389007084674

17.  Realist evaluation – addressing ‘What works for whom, in what
circumstances, in what respects and how?’; evaluation participants
offer context, mechanism and outcomes statements (CMOs) against
which to evaluate (Pawson and Tilley, 2004)
 Contribution analysis – designs a ToC to make credible causal
claims: how and to what extent has an intervention plausibly
contributed to programme/service goals? Evidence is systematically
collected in relation to these claims, to build a contribution story
 Outcomes harvesting – a six step process that systematically and
collaboratively identifies and classifies outcomes (changes in
individuals, groups or communities), then gathers views about
these and how they are achieved. The results are analysed and
tabulated to provide evidence-based contribution statements.