Slicing the evaluation cake


Published on

An overview of the approach to evaluation and impact for projects part of the JISC Content programme 2011-2013

Published in: Education, Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Slicing the evaluation cake

  1. 1. Slicing the evaluation cake JISC Content programme mtg - 14-15 Nov 2011Paola Marchionni, JISC Programme Manager
  2. 2. Before we start baking,we need to think of what we wantto make, the ingredients we needand how we’re going to mix them.
  3. 3. So, what do we want to make? What do we want to achieve? Why?What are the (evaluation and impact)questions we want to ask, and be able to answer?
  4. 4. Need to consider and align: JISC impact areas Content programme aims, objectives and key issuesProjects and institutions aims, objectives and priorities
  5. 5. Think of slicing a layered cake…
  6. 6. JISC funders JISC 5 impact areas Content programmeProjects andinstitutions
  7. 7. JISC layer: 5 impact areas: - Research excellence - Learning and teaching excellence- Increasing effectiveness/saving money - Positive impact on society- Helping with future technology needs
  8. 8. Content programme layer: aims, objectives and key issues - Notion of critical mass- User engagement/responding to users needs (incl embedding in research and teaching) - Partnerships and collaborations - benefit of - Innovation in any aspect of project life cycle - Sustainability and business models - Resource discovery through multiple channels - Skills development- QA: quality, technical standards and best practice (eg IPR and licencing...)
  9. 9. Projects layer: Evaluation and Impact questions Evaluation (formative and summative): during and at the end of the project, more about how we have done things, processes, workflows, skills… and the quality of outputs created, their fitness for purpose Impact (short – medium – long term): more about the kind of “change” or benefit that aproject has brought about. Answers to the evaluation questions above will feed into the measurement of impact.
  10. 10. Some examples of evaluation questions (projects in different strands may have different emphasis) - Have projects created/clustered content? - Does this follow best practice for creating, delivering and sustaining digital resources? - Is content fit for purpose, ie for use/re-use in teaching and research (eg licencing, formats, usability…) - Can content be discovered through multiple channels? - Have partnerships worked effectively? - Have new skills been developed?
  11. 11. Some examples of impact questions, or the 5 cake slices (mapped to JISC impact areas, Content programme and projects/inst concerns)
  12. 12. Slice 1: Learning and teaching excellence- How have the new resources been embedded in teaching and learning (prog key issue)? - Have users (prog key issue) (eg students) and other relevant stakeholders been involved in the process of creation? - What change or benefit has this brought to learning and teaching? - How does this align with institutional priorities (prog key issue and inst issue) in teaching and learning?
  13. 13. Slice 2: Research excellence- Is the digitised content suitable for researchers needs (prog key issue) (eg quality, detail, reliability, critical mass, tools...?) - Has it opened up new research opportunities? - Are there any specific activities or plans to embed content (prog key issue) in research activity? - What does this content allow researchers to do that they couldn’t have done before? - What change has it set in motion? - How does this align with institutional priorities (prog key issue) in research?
  14. 14. Slice 3: Increasing effectiveness/saving money - Has there been fruitful collaboration (prog key issue) within different departments and other external partners? - What have been the benefits/dis-benefits (?) of this collaboration?- Is collaboration and partnership an effective process (in terms of time and resources) for producing and sharing resources (use, re-use)? - Has the project added value/critical mass (prog key issue) to existing repositories? - Are innovative sustainability and business models (prog key issue and inst) contributing to increasing effectiveness of the resource?
  15. 15. Slice 4: Positive impact on society- Is content accessible (prog key issue) beyond the formal education sector and usable by a variety of users (prog key issue)the public at large? - Are there any innovative partnerships (prog key issue, also inst) that help content reach wider audiences?- Have different delivery channels (prog key issue) been used to provide access to these resources?
  16. 16. Slice 5: Helping with future technology needs - Have any innovative technologies (prog key issue) been used to deliver the project?- Do these have wider application within the sector? - Do they have the potential to benefit other institutions?
  17. 17. In summary• Be aware of the bigger picture• Identify how your project is contributing to it• Chose your “slice/s” of cake: in what area is your project going to make more impact? Project are likely to make impact in more than one area• Formulate key evaluation and impact questions (at least short term impact)• Identify your baseline so that you can document change more easily• Plan for medium-long term impact and what you have to put in place in order to be able to measure that• Look at JISC guidance for how to tell your impact stories
  18. 18. Guidance and toolsJISC Evaluation guidance - and concise guidance on formative and summative evaluationJISC Impact guidelines - to the JISC 5 impact areas and pragmatic approach to gathering evidence and measuring impact during and after a projectToolkit for the Impacts of Scholarly Digitised Resources - useful, includes qualitative and quantitative methodologies to measure online impact of resources and case studies. Useful also for planning measurement of long term impact.Open Educational Resources: The value of reuse in higher education - Useful not just for OER but digital resources in general.
  19. 19. Image credits• Cover slide: Front display window of G. E. Adams cake shop, Brisbane Arcade, ca. 1938 No known copyright restrictions -• Slide 3: Cake and mold pans, Cornell University Library, No known copyright restrictions• Slide 6: Photograph of Newlyweds Ronald Reagan and Nancy Reagan cutting their wedding cake, 03/04/1952 - 03/04/1952, Ronald Reagan Library, US National Archives, No known copyright restrictions -• Slide 11 and subsequent, Slice o cake, thepinkpeppercorn CC-BY