An overview of the approach to evaluation and impact for projects part of the JISC Content programme 2011-2013

1. Slicing the evaluation cake
JISC Content programme mtg - 14-15 Nov 2011
Paola Marchionni, JISC Programme Manager p.marchionni@jisc.ac.uk

2. Before we start baking,
we need to think of what we want
to make, the ingredients we need
and how we’re going to mix them.

3. So, what do we want to make?
What do we want to achieve? Why?
What are the (evaluation and impact)
questions we want to ask, and be able
to answer?

4. Need to consider and align:
JISC impact areas
Content programme aims, objectives
and key issues
Projects and institutions aims, objectives
and priorities

5. Think of slicing a layered cake…

6. JISC funders
JISC 5 impact
areas
Content
programme
Projects and
institutions

7. JISC layer: 5 impact areas:
- Research excellence
- Learning and teaching excellence
- Increasing effectiveness/saving money
- Positive impact on society
- Helping with future technology needs

8. Content programme layer:
aims, objectives and key issues
- Notion of critical mass
- User engagement/responding to users needs (incl
embedding in research and teaching)
- Partnerships and collaborations - benefit of
- Innovation in any aspect of project life cycle
- Sustainability and business models
- Resource discovery through multiple channels
- Skills development
- QA: quality, technical standards and best practice
(eg IPR and licencing...)

9. Projects layer:
Evaluation and Impact questions
Evaluation (formative and summative): during and
at the end of the project, more about how we have
done things, processes, workflows, skills… and the
quality of outputs created, their fitness for purpose
Impact (short – medium – long term):
more about the kind of “change” or benefit that a
project has brought about. Answers to the evaluation
questions above will feed into the measurement of
impact.

10. Some examples of evaluation questions
(projects in different strands may have different emphasis)
- Have projects created/clustered content?
- Does this follow best practice for creating,
delivering and sustaining digital resources?
- Is content fit for purpose, ie for use/re-use in
teaching and research (eg licencing, formats,
usability…)
- Can content be discovered through multiple
channels?
- Have partnerships worked effectively?
- Have new skills been developed?

11. Some examples of impact questions,
or the 5 cake slices
(mapped to JISC impact areas, Content programme and
projects/inst concerns)

12. Slice 1:
Learning and teaching excellence
- How have the new resources been embedded in teaching and
learning (prog key issue)?
- Have users (prog key issue) (eg students) and other relevant
stakeholders been involved in the process of creation?
- What change or benefit has this brought to learning and
teaching?
- How does this align with institutional priorities (prog key
issue and inst issue) in teaching and learning?

13. Slice 2:
Research excellence
- Is the digitised content suitable for researchers needs (prog
key issue) (eg quality, detail, reliability, critical mass, tools...?)
- Has it opened up new research opportunities?
- Are there any specific activities or plans to embed content
(prog key issue) in research activity?
- What does this content allow researchers to do that they
couldn’t have done before?
- What change has it set in motion?
- How does this align with institutional priorities (prog key
issue) in research?

14. Slice 3: Increasing
effectiveness/saving money
- Has there been fruitful collaboration (prog key issue) within
different departments and other external partners?
- What have been the benefits/dis-benefits (?) of this
collaboration?
- Is collaboration and partnership an effective process (in terms
of time and resources) for producing and sharing resources
(use, re-use)?
- Has the project added value/critical mass (prog key issue) to
existing repositories?
- Are innovative sustainability and business models (prog key
issue and inst) contributing to increasing effectiveness of the
resource?

15. Slice 4:
Positive impact on society
- Is content accessible (prog key issue) beyond the
formal education sector and usable by a variety of
users (prog key issue)the public at large?
- Are there any innovative partnerships (prog key
issue, also inst) that help content reach wider
audiences?
- Have different delivery channels (prog key issue)
been used to provide access to these resources?

16. Slice 5:
Helping with future
technology needs
- Have any innovative technologies (prog key issue)
been used to deliver the project?
- Do these have wider application within the sector?
- Do they have the potential to benefit other
institutions?

17. In summary
• Be aware of the bigger picture
• Identify how your project is contributing to it
• Chose your “slice/s” of cake: in what area is your project
going to make more impact? Project are likely to make impact
in more than one area
• Formulate key evaluation and impact questions (at least
short term impact)
• Identify your baseline so that you can document change
more easily
• Plan for medium-long term impact and what you have to put
in place in order to be able to measure that
• Look at JISC guidance for how to tell your impact stories

18. Guidance and tools
JISC Evaluation guidance - http://bit.ly/tkE5F1
Brief and concise guidance on formative and summative evaluation
JISC Impact guidelines - http://bit.ly/rzQ9lS
Background to the JISC 5 impact areas and pragmatic approach to gathering
evidence and measuring impact during and after a project
Toolkit for the Impacts of Scholarly Digitised Resources -
http://microsites.oii.ox.ac.uk/tidsr/
Very useful, includes qualitative and quantitative methodologies to measure
online impact of resources and case studies. Useful also for planning
measurement of long term impact.
Open Educational Resources: The value of reuse in higher education -
http://bit.ly/rvAxwP
Useful not just for OER but digital resources in general.

19. Image credits
• Cover slide: Front display window of G. E. Adams cake shop, Brisbane
Arcade, ca. 1938 No known copyright restrictions - http://bit.ly/vlo0mL
• Slide 3: Cake and mold pans, Cornell University Library, No known
copyright restrictions http://flic.kr/p/6SJBQR
• Slide 6: Photograph of Newlyweds Ronald Reagan and Nancy Reagan
cutting their wedding cake, 03/04/1952 - 03/04/1952, Ronald Reagan
Library, US National Archives, No known copyright restrictions -
http://flic.kr/p/9nJ1Np
• Slide 11 and subsequent, Slice o cake, thepinkpeppercorn CC-BY
http://flic.kr/p/6mJyiU