This document summarizes a workshop on evaluation and impact measurement for mentoring programs. The workshop aims to explore evaluation principles, impact measurement, and sharing best practices. It defines evaluation and impact, and explains that evaluation is needed to measure impact. Process and outcome evaluation are discussed. Case studies are presented on evaluating a medical e-mentoring scheme and measuring the impact of an Imperial STEM enrichment program. Key points are that evaluation should be kept simple, short, and quantitative, and that impact measurement demonstrates how a program has made a measurable difference.

1. Dr Annalisa Alexander
Head of Outreach
Imperial College London

2. Workshop Aims:
To understand and explore (in an hour):
Evaluation principles
Impact measurement
What… Why… How…
Case studies and sharing best practise
Group work
Please ask questions, share your thoughts and interact!

3. Definitions….
Evaluation
“To Judge or assess the worth of…”
Impact
“To have an effect upon….”
You can evaluate a scheme in isolation
e.g. Are the participants communicating regularly? What topics
are they discussing?
But you can’t easily measure impact without first evaluating
the programme
e.g. What effect is the programme having on raising aspirations?
Are mentees more likely to achieve higher grades as a result of
being part of it?

4. Evaluation
What is Evaluation?
Process Evaluation – Is it doing what it should be?
Outcome Evaluation – Impact measurement - Has it achieved
what you intended?
Evaluation in a mentoring context:
Critical to ensure the scheme is working
 Are mentors and mentees interacting appropriately?
 Do boundaries need to be changed/set?
 Is the training sufficient?
 Do the mentors and mentees require more support?

5. Evaluation
Case Study – medical e-mentoring scheme
Piloted in 2003 with a few mentoring pairs (range of ages in
three schools)
Feedback from schools suggested Year 10 to Year 13 would be
suitable
Evaluation in 2005 showed Year 10s were not engaged
Scheme then focused in on Year 11+
2010 onwards, OFFA required Year 12 interaction
STEM-wide scheme launched 2012 following feedback from
schools and pupils
Currently 250 mentoring pairs across STEM

6. www.imperialmentoring.org

7. Evaluation
Why is evaluation necessary?
Funders often require it
External bodies (e.g. OFFA) require it
Internal rigor should demand it!
Gathering feedback is meaningless unless you are
prepared to do something with it….

8. Evaluation
How do we do it?
Keep It SIMPLE!
Keep it SHORT!
Keep it quantitative
Free text is hard to interpret but can be useful
Encourage participation through prizes or incentives
Pre and post course evaluation works well – as long as the
questions are comparable….
 Be clear about what you are trying to find out though – a good
evaluation form has a strong focus
 but beware of bias in your questioning

9. Evaluation – Group Work
The Brief:
You have been running a mentoring scheme for sixth formers at
a local school for 18-months to 2 years
The purpose of the scheme was to help them make choices
about careers/university
Another school has expressed an interest in the scheme but
wants to know what is involved in running it
Create a short evaluation form
No more than one side of A4
Remember that you should be able to turn the answers into a
coherent report on the nuts and bolts of the scheme

10. Impact Measurement
What is Impact measurement?
Demonstrating through your evaluation work that your scheme
has made a measurable difference
Impact measurement in a mentoring context:
Critical to ensure the scheme has done what you intended it to
do
 The mentors’ and mentees’ expectations have been met
 The mentors/mentees have grown in confidence and understanding
If the scheme hasn’t achieved what you intended – how can you
improve or change it?
 Feedback is crucial (ideally 360 if possible!)

11. Impact Measurement
Why is Impact measurement so important?
Often a requirement of funders/partner organisations
Ensures that your scheme is on track (E.g. OFFA)
Allows you to make critical changes along the way if it is not
meeting expectations or hitting targets
How do we measure impact?
Keep it simple!
Its easy to develop complicated multi-faceted methods but just a
basic demonstration of cause and effect will often suffice

12. Impact Measurement
 Ref: National Council for Voluntary Organisation

13. Impact Measurement
Case Study – Imperial STEM Enrichment Programme
A series of tailored AS level STEM and personal effectiveness
workshops for high achieving Year 12 students
All from disadvantages backgrounds, under represented in
Higher Education
First cohort was 96 Year 12 students from across London
Fully funded through Access money
Rigorous evaluation and impact measurement is required

14. Impact Measurement
Q1: How would you rate the event overall?
95% rated the event Good or Very good
Response
Number of
responses
Adequate 21
Good 191
Very Good 210
Very Poor 1
Grand Total 423
5%
45%
50%
0%
Adequate
Good
Very Good
Very Poor
Q2: I now feel more motivated about my studies.
88% feel more motivated about their studies
Response Number of responses
Agree 234
Disagree 2
Not Sure 41
Strongly Agree 139
Strongly Disagree 5
(blank) 2
Grand Total 423
55%
1%
10%
33%
1% 0%
Agree
Disagree
Not Sure
Strongly Agree
Strongly Disagree

15. Impact Measurement
Q3: I understand the importance of studying the subject of this event.
92% agree or strongly agree
Response
Number of
responses
Agree 216
Disagree 4
Not Sure 25
Strongly Agree 173
Strongly Disagree 2
(blank) 3
Grand Total 423
51%
1%6%
41%
0% 1% Agree
Disagree
Not Sure
Strongly Agree
Strongly Disagree
(blank)
Q4: I will consider applying to courses related to the subject of this event at University.
91% believe they will consider applying to courses related to the subject at University.
Response
Number of
responses
Agree 156
Disagree 4
Not Sure 24
Strongly Agree 230
Strongly Disagree 5
(blank) 4
Grand Total 423
37%
1%
6%
54%
1% 1% Agree
Disagree
Not Sure
Strongly Agree
Strongly Disagree
(blank)

16. Impact Measurement
Q2-5: I am more likely to go on to University.
84% believe they are more likely to go to university following the event
Response
Number of
responses
Agree 139
Disagree 14
Not Sure 41
Strongly Agree 216
Strongly Disagree 4
(blank) 9
Grand Total 423
33%
3%
10%
51%
1% 2%
Agree
Disagree
Not Sure
Strongly Agree
Strongly Disagree
(blank)
• The impact of this scheme was therefore:
 Participants felt more confident in their studies
 Participants felt more motivated to continue on to Higher
Education
 Participants felt they were now more likely to apply to study a
course relating to the subject they had received support with

17. Impact Measurement – Group
Work
The Brief:
A corporate partner has expressed an interest in your mentoring
scheme and would like to potentially offer three years funding
They have requested a more detailed report that looks at the
impact your scheme has on pupils’ decisions and choices post A-
level
Now revisit your evaluation form
What do you need to change/add to get a more detailed picture
of impact?
What measurements are you interested in?
 e.g. how will you measure change in aspiration?

18. Pitfalls….
Over complicated evaluation forms lose effectiveness
A danger of investing too much time and resource
You can sink a lot of money into it….!
Bias in designing tools - don’t just focus on things you know
will give good feedback
Asking questions that are meaningless or can’t be
interpreted
Avoid free text unless you wish to use comments later
Data protection issues – always be sure to state what you
intend to do with the information

19. References
Rhodes, J (2002): Gauging the Effectiveness of Youth mentoring. The
National Mentoring Partnership
Lee, C (2012): Keeping Impact Measurement Simple. Third Sector (
www.thirdsector.co.uk)
National Council for Voluntary Organisations (www.ncvo-vol.org.uk)
Parsons et al (2008): Formalised Peer Mentoring Pilot Evaluation.
Canterbury Christ Church University (Research Report DCSF-RR033)
Garbarine, S & Holland, J (2009): Quantitative and qualitative methods in
impact evaluation and measuring results. Governance and Social
Development Resource Centre
Berk et al (2005): Measuring the effectiveness of faculty mentoring
relationships. Academic Medicine, Vol. 80, 1.

20. Ref : Dilbert.com, Scott Adams