Dr Annalisa Alexander
Head of Outreach
Imperial College London
Workshop Aims:
To understand and explore (in an hour):
Evaluation principles
Impact measurement
What… Why… How…
Case ...
Definitions….
Evaluation
“To Judge or assess the worth of…”
Impact
“To have an effect upon….”
You can evaluate a sche...
Evaluation
What is Evaluation?
Process Evaluation – Is it doing what it should be?
Outcome Evaluation – Impact measurem...
Evaluation
Case Study – medical e-mentoring scheme
Piloted in 2003 with a few mentoring pairs (range of ages in
three sc...
www.imperialmentoring.org
Evaluation
Why is evaluation necessary?
Funders often require it
External bodies (e.g. OFFA) require it
Internal rigor...
Evaluation
How do we do it?
Keep It SIMPLE!
Keep it SHORT!
Keep it quantitative
Free text is hard to interpret but ca...
Evaluation – Group Work
The Brief:
You have been running a mentoring scheme for sixth formers at
a local school for 18-m...
Impact Measurement
What is Impact measurement?
Demonstrating through your evaluation work that your scheme
has made a me...
Impact Measurement
Why is Impact measurement so important?
Often a requirement of funders/partner organisations
Ensures...
Impact Measurement
 Ref: National Council for Voluntary Organisation
Impact Measurement
Case Study – Imperial STEM Enrichment Programme
A series of tailored AS level STEM and personal effec...
Impact Measurement
Q1: How would you rate the event overall?
95% rated the event Good or Very good
Response
Number of
resp...
Impact Measurement
Q3: I understand the importance of studying the subject of this event.
92% agree or strongly agree
Resp...
Impact Measurement
Q2-5: I am more likely to go on to University.
84% believe they are more likely to go to university fol...
Impact Measurement – Group
Work
The Brief:
A corporate partner has expressed an interest in your mentoring
scheme and wo...
Pitfalls….
Over complicated evaluation forms lose effectiveness
A danger of investing too much time and resource
You ca...
References
Rhodes, J (2002): Gauging the Effectiveness of Youth mentoring. The
National Mentoring Partnership
Lee, C (20...
Ref : Dilbert.com, Scott Adams
Upcoming SlideShare
Loading in …5
×

Evaluation and Impact Measurement - Imperial College London

649 views

Published on

Published in: Education, Technology

Evaluation and Impact Measurement - Imperial College London

  1. 1. Dr Annalisa Alexander Head of Outreach Imperial College London
  2. 2. Workshop Aims: To understand and explore (in an hour): Evaluation principles Impact measurement What… Why… How… Case studies and sharing best practise Group work Please ask questions, share your thoughts and interact!
  3. 3. Definitions…. Evaluation “To Judge or assess the worth of…” Impact “To have an effect upon….” You can evaluate a scheme in isolation e.g. Are the participants communicating regularly? What topics are they discussing? But you can’t easily measure impact without first evaluating the programme e.g. What effect is the programme having on raising aspirations? Are mentees more likely to achieve higher grades as a result of being part of it?
  4. 4. Evaluation What is Evaluation? Process Evaluation – Is it doing what it should be? Outcome Evaluation – Impact measurement - Has it achieved what you intended? Evaluation in a mentoring context: Critical to ensure the scheme is working  Are mentors and mentees interacting appropriately?  Do boundaries need to be changed/set?  Is the training sufficient?  Do the mentors and mentees require more support?
  5. 5. Evaluation Case Study – medical e-mentoring scheme Piloted in 2003 with a few mentoring pairs (range of ages in three schools) Feedback from schools suggested Year 10 to Year 13 would be suitable Evaluation in 2005 showed Year 10s were not engaged Scheme then focused in on Year 11+ 2010 onwards, OFFA required Year 12 interaction STEM-wide scheme launched 2012 following feedback from schools and pupils Currently 250 mentoring pairs across STEM
  6. 6. www.imperialmentoring.org
  7. 7. Evaluation Why is evaluation necessary? Funders often require it External bodies (e.g. OFFA) require it Internal rigor should demand it! Gathering feedback is meaningless unless you are prepared to do something with it….
  8. 8. Evaluation How do we do it? Keep It SIMPLE! Keep it SHORT! Keep it quantitative Free text is hard to interpret but can be useful Encourage participation through prizes or incentives Pre and post course evaluation works well – as long as the questions are comparable….  Be clear about what you are trying to find out though – a good evaluation form has a strong focus  but beware of bias in your questioning
  9. 9. Evaluation – Group Work The Brief: You have been running a mentoring scheme for sixth formers at a local school for 18-months to 2 years The purpose of the scheme was to help them make choices about careers/university Another school has expressed an interest in the scheme but wants to know what is involved in running it Create a short evaluation form No more than one side of A4 Remember that you should be able to turn the answers into a coherent report on the nuts and bolts of the scheme
  10. 10. Impact Measurement What is Impact measurement? Demonstrating through your evaluation work that your scheme has made a measurable difference Impact measurement in a mentoring context: Critical to ensure the scheme has done what you intended it to do  The mentors’ and mentees’ expectations have been met  The mentors/mentees have grown in confidence and understanding If the scheme hasn’t achieved what you intended – how can you improve or change it?  Feedback is crucial (ideally 360 if possible!)
  11. 11. Impact Measurement Why is Impact measurement so important? Often a requirement of funders/partner organisations Ensures that your scheme is on track (E.g. OFFA) Allows you to make critical changes along the way if it is not meeting expectations or hitting targets How do we measure impact? Keep it simple! Its easy to develop complicated multi-faceted methods but just a basic demonstration of cause and effect will often suffice
  12. 12. Impact Measurement  Ref: National Council for Voluntary Organisation
  13. 13. Impact Measurement Case Study – Imperial STEM Enrichment Programme A series of tailored AS level STEM and personal effectiveness workshops for high achieving Year 12 students All from disadvantages backgrounds, under represented in Higher Education First cohort was 96 Year 12 students from across London Fully funded through Access money Rigorous evaluation and impact measurement is required
  14. 14. Impact Measurement Q1: How would you rate the event overall? 95% rated the event Good or Very good Response Number of responses Adequate 21 Good 191 Very Good 210 Very Poor 1 Grand Total 423 5% 45% 50% 0% Adequate Good Very Good Very Poor Q2: I now feel more motivated about my studies. 88% feel more motivated about their studies Response Number of responses Agree 234 Disagree 2 Not Sure 41 Strongly Agree 139 Strongly Disagree 5 (blank) 2 Grand Total 423 55% 1% 10% 33% 1% 0% Agree Disagree Not Sure Strongly Agree Strongly Disagree
  15. 15. Impact Measurement Q3: I understand the importance of studying the subject of this event. 92% agree or strongly agree Response Number of responses Agree 216 Disagree 4 Not Sure 25 Strongly Agree 173 Strongly Disagree 2 (blank) 3 Grand Total 423 51% 1%6% 41% 0% 1% Agree Disagree Not Sure Strongly Agree Strongly Disagree (blank) Q4: I will consider applying to courses related to the subject of this event at University. 91% believe they will consider applying to courses related to the subject at University. Response Number of responses Agree 156 Disagree 4 Not Sure 24 Strongly Agree 230 Strongly Disagree 5 (blank) 4 Grand Total 423 37% 1% 6% 54% 1% 1% Agree Disagree Not Sure Strongly Agree Strongly Disagree (blank)
  16. 16. Impact Measurement Q2-5: I am more likely to go on to University. 84% believe they are more likely to go to university following the event Response Number of responses Agree 139 Disagree 14 Not Sure 41 Strongly Agree 216 Strongly Disagree 4 (blank) 9 Grand Total 423 33% 3% 10% 51% 1% 2% Agree Disagree Not Sure Strongly Agree Strongly Disagree (blank) • The impact of this scheme was therefore:  Participants felt more confident in their studies  Participants felt more motivated to continue on to Higher Education  Participants felt they were now more likely to apply to study a course relating to the subject they had received support with
  17. 17. Impact Measurement – Group Work The Brief: A corporate partner has expressed an interest in your mentoring scheme and would like to potentially offer three years funding They have requested a more detailed report that looks at the impact your scheme has on pupils’ decisions and choices post A- level Now revisit your evaluation form What do you need to change/add to get a more detailed picture of impact? What measurements are you interested in?  e.g. how will you measure change in aspiration?
  18. 18. Pitfalls…. Over complicated evaluation forms lose effectiveness A danger of investing too much time and resource You can sink a lot of money into it….! Bias in designing tools - don’t just focus on things you know will give good feedback Asking questions that are meaningless or can’t be interpreted Avoid free text unless you wish to use comments later Data protection issues – always be sure to state what you intend to do with the information
  19. 19. References Rhodes, J (2002): Gauging the Effectiveness of Youth mentoring. The National Mentoring Partnership Lee, C (2012): Keeping Impact Measurement Simple. Third Sector ( www.thirdsector.co.uk) National Council for Voluntary Organisations (www.ncvo-vol.org.uk) Parsons et al (2008): Formalised Peer Mentoring Pilot Evaluation. Canterbury Christ Church University (Research Report DCSF-RR033) Garbarine, S & Holland, J (2009): Quantitative and qualitative methods in impact evaluation and measuring results. Governance and Social Development Resource Centre Berk et al (2005): Measuring the effectiveness of faculty mentoring relationships. Academic Medicine, Vol. 80, 1.
  20. 20. Ref : Dilbert.com, Scott Adams

×