Presented at the European Conference on Information Literacy, September 2021 by Dr Pamela McKinney and Sheila Webber A video of this presentation is available at https://digitalmedia.sheffield.ac.uk/media/Using+Theories+of+Change+to+evaluate+Information+Literacy+initiatives/1_v1g05eav

1. Using Theories of Change to evaluate
Information Literacy initiatives
Information School, University of Sheffield, UK
ECIL 2021
Dr Pam McKinney
Sheila Webber
Photo:
Sheila
Webber

2. Presentation outline
1. Introductions and our experiences of using Theory of Change
2. Overview of Theory of Change
3. How a Theory of Change is constructed
4. The role of reflection in Theory of Change
5. Developing an evaluation plan
6. Examples of using Theory of Change

3. Introducing ourselves - Sheila
● Senior Lecturer in the Information School,
University of Sheffield
● Teach on the Information School's research
methods modules
● Was leader for one of the projects in Centre
for Inquiry-based Learning in the Arts and
Social Sciences (CILASS)
● Am currently using Theory of Change
working with educators in virtual worlds
Pam McKinney/Sheila Webber 2021

4. Introducing ourselves - Pam
● Background as an educational developer, working at CILASS:
Centre for Inquiry-based Learning in the Arts and Social
Sciences
● Theory of Change methodology applied at project and
programme level to evaluate the impact of the Centre
● I led Theory of Change creation process with stakeholders,
and supported the development of evaluation plans
● Supported reflective evaluation process, and jointly published
with project leaders using evaluation data
● Now: lecturer at the University of Sheffield Information School
Logo:
Sabine
Little
Pam McKinney/Sheila Webber 2021

5. What is Theory of Change?
● A theory-based, participatory approach to project evaluation developed by the Aspen
Institute (Connell & Kubisch 1998)
● Designed to create a link between funding and outcomes for stakeholders
● Can be used as an organisational level tool across multiple projects to standardise
evaluation practice and identify and share good practice.
● Theory-based evaluation explains how and why results are achieved through
understanding the links stakeholders construct between activities and outcomes
● Participatory evaluation avoids feelings that evaluation is “done to” projects, instead
it is a flexible tool applied throughout the project that involves multiple stakeholders.
● Longitudinal evaluation focuses on intermediate and longer term impacts, and can
be revisited long after a project has ceased.
Hart, Diercks-O’Brian & Powell, 2009
Pam McKinney/Sheila Webber 2021

6. The role of the facilitator: creating a Theory of Change
with stakeholders
● Literature is positive about the role of an external facilitator to collaboratively
design an educational intervention, define a Theory of Change and evaluation
plan, and carry out evaluation activities
● In the CILASS project an educational developer led the ToC creation process
with project stakeholders (e.g. academic staff, students, professional services
staff and librarians)
● An audio-recorded planning session, featuring interviews with key
stakeholders, with a draft ToC created
● The educational developer then led the revision and finalisation of the ToC,
and the resulting evaluation plan.
Hart, Diercks-O’Brian & Powell, 2009
Pam McKinney/Sheila Webber 2021

7. Constructing a Theory of Change
Drivers Resources/
enabling
factors
Activities Outcomes Anticipated
impact
What is the
current situation
that has led to
the project?
What support is
needed to do
the project
activities?
What activities
need to take
place to
achieve the
project
outcomes?
What are the
desirable and
feasible
outcomes for
the project?
What is the
longer term
impact of the
project?
There should be a consistent and credible narrative developed across the 5 columns, where
an outcome has specific activities and enabling factors associated with it.
Pam McKinney/Sheila Webber 2021

8. Example Theory of Change from the CILASS programme
Pam McKinney/Sheila Webber 2021

9. Role of reflection in Theory of Change
● Reflection is a key aspect of the Theory of Change design
● Based on “reflective practitioner” model (Kolb, 1984; Lewin, 1946; Schön,
1983).
● Recognition that change is a complex, socially-constructed process, and
reflection can help articulate this.
● Critical reflection from project stakeholders helps
identify relationships between actions and effects
● Reflection is a desirable aspect of the evaluation
plan
Hart, Diercks-O’Brian & Powell 2009
Photo
Nigel
Ford:
Pam
McKinney/Sheila
Webber
2021

10. Next step - developing an evaluation plan
● Once a Theory of Change has been agreed by project stakeholders, an
evaluation plan is created.
● Each Enabler, Process and Outcome indicator should have an evaluation
process associated with it
● Multiple data collection methods can be used, qualitative and quantitative
● A significant role for reflection in evaluation, often taking the form of reflective
interviews with project stakeholders
● An evaluation matrix is created which articulates the evaluation method to be
used for each indicator
Pam McKinney/Sheila Webber 2021

11. Evaluation matrix
ToC indicator Student focus
group
Librarian
questionnaire
Project leader
reflection
Enabler: Effective support structures are
developed for librarians to help them
design effective online IL tutorials
X
Process: A suite of online IL tutorials is
created based on ACRL IL framework
X
Outcome: students have confidence in
referencing practices and can accurately
cite sources in their work.
X X
Pam McKinney/Sheila Webber 2021

12. Examples of Theory of Change in practice
USA's Public Library Association (2018) - developed a ToC for Professional
Development
● Identified goals, outcomes, strategies & competencies required to achieve
outcomes
● Identified 6 key roles for LIS professionals to achieve outcomes
The Youth Cafe - a "pan-African youth empowerment organisation" - uses ToC
and stresses participative element "We have incorporated the views of young
people, youth-led and youth-serving organizations, and experts so that our map
reflects young people’s view of how change occurs" (The Youth Cafe, 2021)
Pam McKinney/Sheila Webber 2021

13. The
Youth
Cafe
(2021)

14. The
Youth
Cafe
(2021)
Pam McKinney/Sheila Webber 2021

15. How could you use
Theories of Change to
evaluate Information
Literacy initiatives?
Photo:
Sheila
Webber

16. Sheila Webber
Information School
University of Sheffield
s.webber@shef.ac.uk
Twitter: @sheilayoshikawa
http://information-literacy.blogspot.com/
http://www.slideshare.net/sheilawebber/
Dr Pamela McKinney
Information School
University of Sheffield
p.mckinney@shef.ac.uk
Twitter: @ischoolpam
https://www.slideshare.net/PamelaMcKinney/