Handout to accompany presentation by Jay Dempster

1. JISC Guidance for Evaluating BCE Support
Activities
Demonstrating EVIDENCE of outcomes & impact
Services are often asked to report on aims, methodologies and lessons learned and to
demonstrate outcomes and impact of their activities & initiatives, in other words to show ‘value
for money’.
This is certainly a requirement for formal evaluation of your service. But capturing data from
your day-to-day activities offers a continuous “product development cycle” that is immensely
beneficial as a means of reflective practice. This is not only important for ongoing engagement
with your clients, to consult upon whether their needs are being met, but also when it comes to
later reporting can mean the evidence is readily at hand.
Taking this approach may mean being somewhat more pro-active seeking out and collating a
rich variety of different types of evidence to back up statements or claims about impact. But the
pay-offs are worth it, particularly when it comes to justifying your existence – an inevitable
necessity with a dwindling resource base.
The purpose here is to provide evidence of impact, in particular. However, it can be unclear to
practitioners what is meant by 'evidence' in the context of BCE related activity. This guide aims
to clarify and provide some examples against some typical kinds of outcomes from hard
quantitative data to the softer qualitative and illustrative evidence.
First, let’s consider who the evidence is aimed at. This is where evaluation should feed into
dissemination, and vice versa, both being based upon an acute awareness of one’s audience
(clients or stakeholders) and leading to an appropriate and effective “message to market
match”.
- Who are your different stakeholders and what are they most likely to be interested in?
- What questions or concerns might they have?
- What form of advice or guidance is most likely to suit their needs?
And from that starting point, consider what ‘evidence’ is most likely to demonstrate impact, and
indeed how well you’ve hit your target aims.
Second, let’s look at the nature of the evidence itself.
'Evidence' is anything that helps to confirm or illustrate an outcome (see Examples below).
Evidence may be arrived at through formal evaluation processes – when it is likely to be
supported by data collected and analysed for that purpose – but also through reflection, data
traces gathered during set up, implementation and follow up of an activity or product, as part of
a collective 'story' of your service’s deliverables and outcomes.
Creative forms of ‘rich media’ evidence, such as vision boards, multimedia, talking heads,
screenshots and images, have an important role to play in communicating outcomes. However,
the type of evidence needs to match the type of message or the claim being made. It needs to
be right for the purpose, occasion and audience it is used for (e.g. formal report or presentation,
informal consultations, website, blog, newsletter or published paper).
For example, if we are claiming that particular service activities generated a significant number
of new contacts and partnership opportunities, we need quantitative data, numbers and
examples. If we are saying that awareness of BCE has been raised (an outcome that cuts
across all BCE projects & services), the data will tend to be more qualitative, analysed from
interviews, questionnaires and/or surveys, and possibly include quotes from clients & other
stakeholders to illustrate how their attitudes, roles or strategies/policies may have changed.
© Belanda Consulting 2010 Jay Dempster & Helen Beetham

2. JISC Guidance for Evaluating BCE Support
Activities
1. An illustrative ‘Activity System’ as a basis for evaluating
services ‘product development cycle’
Evidence generated by complex processes – such as for BCEct, the intersection of new
technologies with different organisational agendas – is often itself complex and difficult to
generalise. This is fine, providing the complexity and the limits of generalisability are
acknowledged.
Indeed, you may seek to use theoretical frameworks to model the dynamics of supporting BCE
activity and draw out the specific contextual factors at play, e.g. Engstrom’s activity theory used
in BCEct (see below) or Wenger’s communities of practice.
Engestrom's activity system triangles:
This framework is not an evaluation methodology as such but the activity system is a
very tight fit with the context of JISC services & BCE projects, investigating how diverse
objectives (of subjects: institutions & B&C partners) are taken forward through the
medium of shared tools & services, in the context of community/institutional strategies
for engagement (rules, division of labour).
How might you interpret the dynamics of your stakeholder
context and where/how your service might best engage and
contribute?
© Belanda Consulting 2010 Jay Dempster & Helen Beetham

3. JISC Guidance for Evaluating BCE Support
Activities
Taking the different points in turn gives us a set of evaluation questions that
could form the basis for gathering data around a product development cycle.
Objects What were the objectives of the activity/product? How did they fit with
clients’ (or other stakeholders’) strategic goals?
What have the subjects (participants, end users) said about why they got
involved in the activity/use of the product?
What have the subjects said about what they wanted from engaging with
your service?
Subjects Who were involved as participants in the activity/use of the product?
(Give roles and departments/organisations rather than names)
Who else had an interest in the collaboration (stakeholders)?
Tools/resources What tool(s) or resource(s) were used by the participants?
How fit-for-purpose or effective did they feel the tool/resource was?
What issues, if any, arose in implementing the ideas/resource(s)
provided?
What guidance would you give other institutions on using the
tool(s)/resource?
BCEct specifically would include here questions around ‘choosing and
using collaborative tools’ (examples available if appropriate to your
project)
Roles/division of How (much) did different participants engage with you?
labour How comfortable were different participants with their role in
implementing the approaches?
How has use of approaches/tool(s) changed roles and modes/levels
/type of participation?
How have attitudes towards BCE changed, if at all?
How have participants' different motivations played out? Have they
changed?
Community How were relevant communities engaged, developed, facilitated, funded
(if relevant), sustained as a result of using your services/products?
What indicators are there of any changes in community (institutional)
practice and/or policy around BCE? And whether these changes
sustainable?
Outcomes What happened as a result of clients using your services/products?
What engagement activities or partnerships came about (that might not
have happened, or not in that way/at that time)?
What evidence is there – of achieving objectives and furthering strategic
goals?
What changes have been observed – in practice, process, attitude, level
of engagement?
What benefits have stakeholders identified from using BCE approaches
as a result of working with your service?
What indirect benefits can be inferred or observed?
Were there any unexpected outcomes?
© Belanda Consulting 2010 Jay Dempster & Helen Beetham

4. JISC Guidance for Evaluating BCE Support
Activities
2. Examples of Evidence
Examples of the kind of evidence you might provide for different outcomes are given below. The
list is meant to be illustrative and not exhaustive.
Outcome Evidence
(benefits, direct/indirect /unintended, (gathered from clients or at events)
success/impact indicators)
successful implementation of …(a Feedback, screenshots, guidance notes
tangible deliverable)
successful collaborative activities with Reports, recording (podcast, excerpt), photos,
partners (screenshots if online)
identification and follow-up of new BCE Reports, articles/press releases, transcripts, quotations
opportunities; new partnerships 'Talking heads', audio and video clips, especially with
new partners or potential partners
improved awareness and perception of Feedback questionnaires (include quotes)
BCE within the institutions involved 'Talking heads', audio and video clips
Outcomes of e.g. interviews, survey, questionnaire,
focus group
improved skills and expertise Surveys, email follow ups (include quotes)
Workshop programme and feedback from participants
more strategic approach to BCE New institutional policies, strategies, mission
statements (excerpts)
internal emails indicating new practice or approach
sustainable approach to the use of any Evidence of use being cascaded to other areas of
online tools institution; changes in IT policy
Evidence of guidelines being adopted.
© Belanda Consulting 2010 Jay Dempster & Helen Beetham

5. JISC Guidance for Evaluating BCE Support
Activities
Consider your own service in terms of aims, outcomes and
evaluation resources (what, who, where, how, when) to decide
what will be manageable for you.
From your list, choose a selection to prioritise that would lend maximum credibility to your
narrative about benefits and impact from your activities to the different internal and external
audiences.
Outcome Example evidence
(tangible outputs, benefits, direct/indirect (existing or new data gathered from
/unintended, success/impact indicators) clients/stakeholders, specify what, who, how,
where, when as far as possible)
© Belanda Consulting 2010 Jay Dempster & Helen Beetham