Using Theories of Change to evaluate Information Literacy initiatives ECIL 2021.pptx
•
0 likes•4 views
Workshop at the European Conference on Information Literacy 2021 Theories of Change (ToC) is a participative approach to evaluating the impact of projects, programmes and initiatives. Facilitators help stakeholders to construct ToC at the initial stages of the initiative, and support them in monitoring and in impact evaluation. ToC has been used to evaluate the success and impact of projects in a variety of sectors (often community and public sector initiatives; Mason & Barnes, 2007), and in educational development (Hart, Dierks-O’Brien & Powell, 2009) including Information Literacy initiatives (McKinney, 2014; McKinney, Jones & Turkington, 2011). McKinney was part of the core team facilitating ToC evaluation of projects in the multi-million pound Centre for Inquiry Based Learning in the Arts and Social Sciences (CILASS) project (McKinney, 2014) and Webber was a stakeholder involved in two projects and a CILASS Academic Fellow. In the version of the ToC process used in CILASS projects, stakeholders are asked to identify the drivers for change in the current situation; the longer term impact they envisage the project will have; the intermediate outcomes that the project is expected to achieve ; activities that would need to be undertaken to achieve outcomes and enabling factors and resources required to support the project (Hart, Dierks-O’Brien & Powell, 2009). Stakeholders collaboratively design a Theory of Change poster that defines key project indicators and develops a causal narrative between project activities and outcomes. A plan and evaluation framework is then developed from these indicators, and stakeholders design data collection instruments. Connell & Kubisch (1998) have identified that a good ToC should be plausible, doable and testable. Objectives and outcomes for the Workshop Objectives will be: (1) To explain ToC, its value and application (2) To enable participants to plan how they could use ToC to improve practice and impact. By the end of the workshop participants will (1) understand what ToC involves; (2) have learnt the key steps in facilitating a ToC approach; and (3) will have identified how ToC could be used in their own workplace Workshop outline There will be five portions: (1) A presentation describing ToC, identifying why it is useful, giving examples and outlining the steps in the ToC process. (2) Participants will, individually, identify an project, intervention, activity or class where ToC could be used. (3) Participants will form small groups, briefly explain each of their projects (etc.) and choose one per group to focus on. (4) The groups will use prompt questions to start drawing up a ToC evaluation plan for their chosen project. (5) Sharing of ideas, and questions. The target audience is anyone who wishes to evaluate projects, programmes, curricula or other initiatives. Equipment should include presentation facilities, flipchart paper and pens. We propose a workshop of 90 minutes.
Recommended
Recommended
More Related Content
Similar to Using Theories of Change to evaluate Information Literacy initiatives ECIL 2021.pptx
Similar to Using Theories of Change to evaluate Information Literacy initiatives ECIL 2021.pptx (20)
More from Pamela McKinney
More from Pamela McKinney (13)
Recently uploaded
Recently uploaded (20)
Using Theories of Change to evaluate Information Literacy initiatives ECIL 2021.pptx
- 1. Using Theories of Change to evaluate Information Literacy initiatives Information School, University of Sheffield, UK ECIL 2021 Dr Pam McKinney Sheila Webber
- 2. Presentation outline 1. Introductions and our experiences of using Theory of Change 2. Overview of Theory of Change 3. How a Theory of Change is constructed 4. The role of reflection in Theory of Change 5. Developing an evaluation plan 6. Examples of using Theory of Change
- 3. Introducing ourselves - Sheila ● Senior Lecturer in the Information School, University of Sheffield ● Teach on the Information School's research methods modules ● Was leader for one of the projects in Centre for Inquiry-based Learning in the Arts and Social Sciences (CILASS) ● Am currently using Theory of Change working with educators in virtual worlds Pam McKinney/Sheila Webber 2021
- 4. Introducing ourselves - Pam ● Background as an educational developer, working at CILASS: Centre for Inquiry-based Learning in the Arts and Social Sciences ● Theory of Change methodology applied at project and programme level to evaluate the impact of the Centre ● I led Theory of Change creation process with stakeholders, and supported the development of evaluation plans ● Supported reflective evaluation process, and jointly published with project leaders using evaluation data ● Now: lecturer at the University of Sheffield Information School Logo: Sabine Little Pam McKinney/Sheila Webber 2021
- 5. What is Theory of Change? ● A theory-based, participatory approach to project evaluation developed by the Aspen Institute (Connell & Kubisch 1998) ● Designed to create a link between funding and outcomes for stakeholders ● Can be used as an organisational level tool across multiple projects to standardise evaluation practice and identify and share good practice. ● Theory-based evaluation explains how and why results are achieved through understanding the links stakeholders construct between activities and outcomes ● Participatory evaluation avoids feelings that evaluation is “done to” projects, instead it is a flexible tool applied throughout the project that involves multiple stakeholders. ● Longitudinal evaluation focuses on intermediate and longer term impacts, and can be revisited long after a project has ceased. Hart, Diercks-O’Brian & Powell, 2009 Pam McKinney/Sheila Webber 2021
- 6. The role of the facilitator: creating a Theory of Change with stakeholders ● Literature is positive about the role of an external facilitator to collaboratively design an educational intervention, define a Theory of Change and evaluation plan, and carry out evaluation activities ● In the CILASS project an educational developer led the ToC creation process with project stakeholders (e.g. academic staff, students, professional services staff and librarians) ● An audio-recorded planning session, featuring interviews with key stakeholders, with a draft ToC created ● The educational developer then led the revision and finalisation of the ToC, and the resulting evaluation plan. Hart, Diercks-O’Brian & Powell, 2009 Pam McKinney/Sheila Webber 2021
- 7. Constructing a Theory of Change Drivers Resources/ enabling factors Activities Outcomes Anticipated impact What is the current situation that has led to the project? What support is needed to do the project activities? What activities need to take place to achieve the project outcomes? What are the desirable and feasible outcomes for the project? What is the longer term impact of the project? There should be a consistent and credible narrative developed across the 5 columns, where an outcome has specific activities and enabling factors associated with it. Pam McKinney/Sheila Webber 2021
- 8. Example Theory of Change from the CILASS programme Pam McKinney/Sheila Webber 2021
- 9. Role of reflection in Theory of Change ● Reflection is a key aspect of the Theory of Change design ● Based on “reflective practitioner” model (Kolb, 1984; Lewin, 1946; Schön, 1983). ● Recognition that change is a complex, socially-constructed process, and reflection can help articulate this. ● Critical reflection from project stakeholders helps identify relationships between actions and effects ● Reflection is a desirable aspect of the evaluation plan Hart, Diercks-O’Brian & Powell 2009 Photo Nigel Ford: Pam McKinney/Sheila Webber 2021
- 10. Next step - developing an evaluation plan ● Once a Theory of Change has been agreed by project stakeholders, an evaluation plan is created. ● Each Enabler, Process and Outcome indicator should have an evaluation process associated with it ● Multiple data collection methods can be used, qualitative and quantitative ● A significant role for reflection in evaluation, often taking the form of reflective interviews with project stakeholders ● An evaluation matrix is created which articulates the evaluation method to be used for each indicator Pam McKinney/Sheila Webber 2021
- 11. Evaluation matrix ToC indicator Student focus group Librarian questionnaire Project leader reflection Enabler: Effective support structures are developed for librarians to help them design effective online IL tutorials X Process: A suite of online IL tutorials is created based on ACRL IL framework X Outcome: students have confidence in referencing practices and can accurately cite sources in their work. X X Pam McKinney/Sheila Webber 2021
- 12. Examples of Theory of Change in practice USA's Public Library Association (2018) - developed a ToC for Professional Development ● Identified goals, outcomes, strategies & competencies required to achieve outcomes ● Identified 6 key roles for LIS professionals to achieve outcomes The Youth Cafe - a "pan-African youth empowerment organisation" - uses ToC and stresses participative element "We have incorporated the views of young people, youth-led and youth-serving organizations, and experts so that our map reflects young people’s view of how change occurs" (The Youth Cafe, 2021) Pam McKinney/Sheila Webber 2021
- 15. How could you use Theories of Change to evaluate Information Literacy initiatives?
- 16. Sheila Webber Information School University of Sheffield s.webber@shef.ac.uk Twitter: @sheilayoshikawa http://information- literacy.blogspot.com/ http://www.slideshare.net/sheilawebber / Dr Pamela McKinney Information School University of Sheffield p.mckinney@shef.ac.uk Twitter: @ischoolpam https://www.slideshare.net/PamelaMcKinney/ Workshop outline and references at https://tinyurl.com/5dybntaa
Editor's Notes
- Sheila
- Sheila
- Pam Hi I’m DrPam McKinney, currently i’m a lecturer at the Information school, but my previous role was as an educational developer in the Centre for Inquiry-based Learning n the Arts and Social Sciences, a Centre for Excellence on Teaching and learning based at the University of Sheffield CILASS adopted the Theory of Change methodology to evaluate over 120 curricueum development projects, and we also used it to evaluate the programme as a whole. I facilitated the theory of change approach with academic departments and individual project leaders, and helped them with collecting and anlysing evaluation data. The CILASS evaluation process relied quite heavily on reflective interviews with project leaders at 6 monthly intervals, which I conducted, and the project evaluation process led to 2 journal publications. I’m now a lecturer at the Informtion school.
- Pam Theory of Change was developed in the 1990s by the Aspen Institute in colorado along the principles of developing an approach to impact evaluation that was both theory based and participatory. This method of evaluation was intended to provide a demonstrable link between funding for (initially) community projects and their outcomes, something that is notoriously hard to establish. It was adopted at the University of Sheffield as an organisationsal approach to the evaluation of internally funded curriculum development projects, with the intention to standardise evaluation practice, leading to an enhanced ability to identify and share good practice across the institution. I’ going to outline these principles in a little more detail. Theory of change encourages stakeholders to articulate the expected outcomes for the project and then link these to specific activities they are going to do, thus creating the narrative of the project and a theory of how the desired outcomes can be achieved. Theory of Change is participatory, it is designed to be collaboratively created by multiple project stakeholders and increases a sense of ownership of the evaluation process, which otherwise might feel as if it’s simply an external process that is done to projects. One of the strengths of Theory of Change is that is facilitates a longitudinal evaluation process, as it encourages project stakeholders to articulate the longer term impact of their project. A theory of Change can be be revisited long after a project has formally completed to understand and evaluate its impact. He reference here is to a paper written by the learning enhancement team at the university and gives a lot nore detail about the philosophy and implementation of Theory of change in higher education.
- Pam Usually, a theory of change is created collaboratively with stakeholders, and this process is often led by a facilitator, In the exercise later, some of you will act as facilitators to develop the theory of change with a stakeholder. The facilitator role is generally seen to be very positive, it is helpful to have someone who can bring a neutral perspective, and use active questioning to help stakeholders conceptualise their what their project hopes to achieve and how it is going to realise these objectives. In the CILASS project an educational developer led the Theory of Change creation process, usually this would involve academic staff, professional services staff, students, librarians, sometimes all working in the same room, or sometimes a smaller stakeholder group would meet, and the resulting Theory of Change sent to stakeholders for comment and amendment. We usually worked with sheet of A3 paper, to handwrite notes, and audio record the conversation so that we could go back over crucial points and create indicators that were aligned with the views of stakeholders. Then a ToC word document was drafted by the educational developer and sent to all stakeholders for comment and revision. Once the Theory of Change was finalised, the educational developer would then help with stakeholders define the evaluation plan
- Pam This table is the starting point for the Theory of Change, and forms the basis of the document that is created and shared with stakeholders The process starts with the articulation of the anticipated longer term impacts of the project = the “blue skies” thinking about what the stakeholders hope to achieve in 5-10 years time Then we address the drivers, the current situation that has prompted the need for the project, asking stakeholders to define why the project has come about, what is the situation that can be changed? Then we define the specific measurable and achievable outcomes for the project - these need to be plausible, doable and testable , this is often the trickiest section of the document to complete. Once you have your outcomes, the stakeholders need to define the activities that will take place that will directly lead to the outcomes, and the facilitator needs to help stakeholders identify the links between the two columns. I would often ask questions such as “you have this outcome, what are you going to do to make this happen?” Finally we look at the enabling factors, these can be varied, but can involve securing time and other resources for the project, buy in from senior management, and ways to support the deliver of the project. Again here the role of the facilitator is to ask “what needs to change so that this activity can take place?”
- Pam This is an example theory of change developed by the information school, previously the Department of information studies at the university. It is too small to read on the slide during this presentation, but we’ll make the slides available so you can zoom in to have a look in your own time. The project focused on developing approaches to inquiry-based learning with an inquiry-based level 1 module, and also in supporting the development of information literacy in the departmental teaching.
- Pam I mentioned reflection briefly previously, but it really is a key aspect of Theory of Change. The approach draws heavily on the idea of the reflective practitioner, and the value this has for teachers and other professionals. The reflective process is vital to articulate the changes that have happened, and how these link to the initial intentions of the project. Change is a complex, socially constructed process, so people are vital to the interpretation of the change process. Project stakeholder reflections can provide different perspectives on the change and help construct a ore holistic interpretation of the project. In the CILASS programme project leaders were invited to a reflective interview every 6 months after the completion of their project, and this helped keep track of the longer term impacts.
- Pam Once a Theory of Change has been agreed by project stakeholders, an evaluation plan is created. Ideall, each enabler, process and Outcome should have an evaluation method attached to it, and a variety of evaluation methods can be used, depending on the indicator, the resources available, and other aspects of the situation. As mentioned previously, reflective interviews with project leaders or stakeholders are a valid and informative source of ealuation data. The evaluation plan is articulated as an evaluation matrix, and we can se an example on the next slide
- Pam Process of negotiation between stakeholders and facilitator to identify reasonable and informative evaluation methods - lots of factors can influence the choice of evaluation methods e.g. access to participants, availability of analysts, epistemological position of stakeholders The evaluation matrix is a shared document that is approved by the stakeholders and implemented by the facilitator and the project team.
- Sheila