Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
EvaluationBruce Etherington15th May 2013
This presentation is developed from a number ofpresentations originally created by the NationalCoordinating Centre for Pub...
Aims of day• To help develop a shared set of approaches toevaluating engagement across Wellcome TrustCentres• To develop s...
Timetable9am Arrive (tea and coffee)9.15 Introductions9.20 Why Evaluate?9.40 How do I know what I am evaluating?10.30 Brea...
Introductions• Who you are• What experience you have in evaluation• What you are hoping from the day
WHY EVALUATE?
1. Why evaluate?Beginner’s Guide to Evaluation
Why and Who of evaluationPost up as many reasons as you can think ofwhy we evaluate our activities on the WHYflipchart.
HOW DO I KNOW WHAT I AMEVALUATING?
Source: Ingenious evaluations: A guide for grant holders, The Royal Academy of Engineers2a: What are you aiming to do?Begi...
1. Aim (what do you want to achieve? Big picture!)2. Objectives (what you need to do to achieve your aim?)3. Evaluation qu...
• Pick an activity that you know well• Pair up with someone you do not know andexplain you activity to each other– Why you...
Inputs Outputs OutcomesBasic Logic Model
Term Definition ExampleInputs Public sector resources required toachieve the policy objectiveResources used to deliver the...
• Pick an project you are familiar and start towork out the steps of a logic model for it• Consider:– Inputs – Resources U...
• Understanding the theory of the change youare aiming for improves evaluation– You can see what you need to evaluate– You...
Pool of long termunemployed wholack skillsObtain Placementsand undertaketrainingImproveQualifications andworkplace skills ...
• What evaluation questions might you want toask about this project?– Are we promoting it sufficiently to the targetaudien...
Other TemplatesAssumptionsAssumptionsNick Temple/School for Social EntrepreneursAssumptionsActivitiesImmediateEffectsMediu...
1. Analysis of the project’s Context2. Stakeholder Analysis3. Problem Analysis/Situation Analysis4. Objectives Analysis5. ...
• Participatory approach to evaluation• Looks to understand the contribution of aproject to changes in practice of stakeho...
• “So That” chains• UNDP template:– Identify the desired change– Identify the agents of change– Identify the assumptions– ...
• HM Treasury, The Magenta Book: Guidance forevaluation (2011)• Annie E Casey Foundation, Theory of Change:A Practical Too...
WHO IS THE EVALUATION FOR?
Why and Who of evaluationPost up as many audiences that you can thinkof for our evaluation work on the WHOflipchart.
What do these audiences want?In groups consider the following questions:• Why is this audience interested in yourevaluatio...
2. Making sure your evaluation is fit for purpose
ActivityLook at the example evaluation strategies.Is the approach suitable for all the potentialaudiences of evaluation?Wh...
Corrosion Summer BallA family activity during the Manchester Science Festival with 4 table-top interactive experiences rel...
Challenges of measuring impactWhat are the key challenges to measuringimpact?ResultsBehaviourL e a r n I n gR e a c t I o n
MAKING THE CASE FORENGAGEMENT AND EVALUATION
3. Making an impact with your evaluationHow can you make use of evaluation?• Self reflection• Reports– Case studies and ot...
 What worked well? Why? What did not work well? Why not? What will I do the same next time? What will I do different...
What are the keythings you needto include in areport?Reports
Other ways of reportingCase studies/ Video etcWhat are the pros and cons of usingcase studies as a way of reporting onyour...
Making a case• Things you can evidence–History of evaluative practice informingdevelopment of activities–Learning (self/te...
Beyond the reportWhat are the opportunities for sharing yourevaluation with others?• On your website• With funders• With p...
Top tips• Think about your audience• Develop your evaluation plan at the beginning• Don’t collect data you can’t use• Bewa...
Evaluation can always havean impact.............if you let it
NCCPE http://www.publicengagement.ac.uk/how/guides/evaluation/resourcesManchester Beacon Evaluation Guide http://www.manch...
Evaluation training for wellcome trust 15th may
Evaluation training for wellcome trust 15th may
Evaluation training for wellcome trust 15th may
Upcoming SlideShare
Loading in …5
×

Evaluation training for wellcome trust 15th may

579 views

Published on

Traininf

Published in: Education, Business, Technology
  • Be the first to comment

  • Be the first to like this

Evaluation training for wellcome trust 15th may

  1. 1. EvaluationBruce Etherington15th May 2013
  2. 2. This presentation is developed from a number ofpresentations originally created by the NationalCoordinating Centre for Public Engagement and theBeacons for Public Engagement through the HESTEM Programme.http://www.publicengagement.ac.uk/evaluating-stem-outreachAs such, this presentation is released under thesame Creative Commons licence of Attribution-Share Alike 3.0 Unportedhttp://creativecommons.org/licenses/by-sa/3.0/
  3. 3. Aims of day• To help develop a shared set of approaches toevaluating engagement across Wellcome TrustCentres• To develop skills of participants to develophigh quality evaluation strategies• To help participants to make strong cases forengagement with research and for theevaluation of this activity
  4. 4. Timetable9am Arrive (tea and coffee)9.15 Introductions9.20 Why Evaluate?9.40 How do I know what I am evaluating?10.30 Break10.45 How do I know what I am evaluating? (cont)11.45 Who is the evaluation for?12.30 Lunch13.30 Making the case for engagement (and evaluationof engagement)15.00 End
  5. 5. Introductions• Who you are• What experience you have in evaluation• What you are hoping from the day
  6. 6. WHY EVALUATE?
  7. 7. 1. Why evaluate?Beginner’s Guide to Evaluation
  8. 8. Why and Who of evaluationPost up as many reasons as you can think ofwhy we evaluate our activities on the WHYflipchart.
  9. 9. HOW DO I KNOW WHAT I AMEVALUATING?
  10. 10. Source: Ingenious evaluations: A guide for grant holders, The Royal Academy of Engineers2a: What are you aiming to do?Beginner’s Guide to Evaluation
  11. 11. 1. Aim (what do you want to achieve? Big picture!)2. Objectives (what you need to do to achieve your aim?)3. Evaluation questions (what do you want to know?)4. Methodology (what strategy will you use?)5. Data collection (what techniques will you use to collectyour evidence?)6. Data analysis (how will you analyse your data?)7. Reporting (who will be reading your report?)What goes in an evaluation plan?Beginner’s Guide to Evaluation
  12. 12. • Pick an activity that you know well• Pair up with someone you do not know andexplain you activity to each other– Why you do the activity– What you hope to achieve by doing the activityActivity
  13. 13. Inputs Outputs OutcomesBasic Logic Model
  14. 14. Term Definition ExampleInputs Public sector resources required toachieve the policy objectiveResources used to deliver thepolicyActivities What is delivered on behalf of thepublic sector to the recipientProvision of seminars, trainingevents, consultations etc.Outputs What the recipient does with theresources, advice/trainingreceived, or intervention relevantto themThe number of trainingcourses completedIntermediateOutcomesThe intermediate outcomes of thepolicy produced by the recipientJobs created, turnover,reduced costs or trainingopportunities providedImpacts Wider societal and economicoutcomesThe change in personalincomes and ultimatelywellbeingHM Treasury Definitions (p22)
  15. 15. • Pick an project you are familiar and start towork out the steps of a logic model for it• Consider:– Inputs – Resources Used– Activities – What the project did/does– Outputs – What the participants did/do– Intermediate Outcomes – What changed in theparticipants– Impact – Wider societal effectsYour activity
  16. 16. • Understanding the theory of the change youare aiming for improves evaluation– You can see what you need to evaluate– You can see what you do not need to evaluate– You can see the assumptions you may be makingLogic Models & Evaluation
  17. 17. Pool of long termunemployed wholack skillsObtain Placementsand undertaketrainingImproveQualifications andworkplace skills ofattendeesJob Training scheme exampleObtainInterviews andJob OffersIncrease injobs andincomesLower overallunemploymentHM Treasury, Magenta Book, p23
  18. 18. • What evaluation questions might you want toask about this project?– Are we promoting it sufficiently to the targetaudience?– Are the training courses at the right level?– Are they improving the skills and qualifications ofattendees?– Are they getting more interviews? If not, why not?– Etc.Job Training scheme example
  19. 19. Other TemplatesAssumptionsAssumptionsNick Temple/School for Social EntrepreneursAssumptionsActivitiesImmediateEffectsMediumtermoutcomesLong termimpact
  20. 20. 1. Analysis of the project’s Context2. Stakeholder Analysis3. Problem Analysis/Situation Analysis4. Objectives Analysis5. Plan of Activities6. Resource Planning7. Indicators/Measurements of Objectives8. Risk Analysis and Risk Management9. Analysis of the AssumptionsOther Templates – Logical Framework
  21. 21. • Participatory approach to evaluation• Looks to understand the contribution of aproject to changes in practice of stakeholders• Needs skilled facilitation and a budgetOther Templates – Outcomes Mapping
  22. 22. • “So That” chains• UNDP template:– Identify the desired change– Identify the agents of change– Identify the assumptions– Pathways to Change– Indicators of Change• Theory U (www.presencing.com)Other Templates
  23. 23. • HM Treasury, The Magenta Book: Guidance forevaluation (2011)• Annie E Casey Foundation, Theory of Change:A Practical Tool for Action, Results andLearning (2004)http://www.aecf.org/upload/publicationfiles/cc2977k440.pdf• www.outcomemapping.ca• www.theoryofchange.orgReferences
  24. 24. WHO IS THE EVALUATION FOR?
  25. 25. Why and Who of evaluationPost up as many audiences that you can thinkof for our evaluation work on the WHOflipchart.
  26. 26. What do these audiences want?In groups consider the following questions:• Why is this audience interested in yourevaluation?• What are the top three things they wouldwant to know?• What are the things you, as the organiser,would like them to know?
  27. 27. 2. Making sure your evaluation is fit for purpose
  28. 28. ActivityLook at the example evaluation strategies.Is the approach suitable for all the potentialaudiences of evaluation?What else could the organisers do to help improvetheir evaluation plan?Read the feedback – do you agree/ disagree withthe suggestions?
  29. 29. Corrosion Summer BallA family activity during the Manchester Science Festival with 4 table-top interactive experiences related tocorrosion science. The aims of the activity were to:• inspire the general public with an introduction to corrosion.• communicate that corrosion is interesting and relevant to peoples daily lives.• provide an exciting and memorable learning experience.• make universities more accessible to the general public.What could be the possible outputs, outcomesand impact of this activity?http://www.publicengagement.ac.uk/how/case-studies/corrosion-summer-ballOutputs, outcomes and impact
  30. 30. Challenges of measuring impactWhat are the key challenges to measuringimpact?ResultsBehaviourL e a r n I n gR e a c t I o n
  31. 31. MAKING THE CASE FORENGAGEMENT AND EVALUATION
  32. 32. 3. Making an impact with your evaluationHow can you make use of evaluation?• Self reflection• Reports– Case studies and other formats eg presentations/ video/audio etc• Making a case for future funding/ support
  33. 33.  What worked well? Why? What did not work well? Why not? What will I do the same next time? What will I do differently next time?
  34. 34. What are the keythings you needto include in areport?Reports
  35. 35. Other ways of reportingCase studies/ Video etcWhat are the pros and cons of usingcase studies as a way of reporting onyour evaluation?
  36. 36. Making a case• Things you can evidence–History of evaluative practice informingdevelopment of activities–Learning (self/team reflection)–Approach is informed by target audiences–Effective practice–Commitment to future evaluation toinform activity
  37. 37. Beyond the reportWhat are the opportunities for sharing yourevaluation with others?• On your website• With funders• With partners• With others e.g. NCCPE; Collective Memory
  38. 38. Top tips• Think about your audience• Develop your evaluation plan at the beginning• Don’t collect data you can’t use• Beware of misrepresenting your data• Back up qualitative data with quantitative data• Don’t hide mistakes – learn from them• Reflect on what you would do differently next time• Recognise the challenges of measuring impact• Be realistic about what you can measure• Remember the value of using evaluation during theproject• Share what you have learnt
  39. 39. Evaluation can always havean impact.............if you let it
  40. 40. NCCPE http://www.publicengagement.ac.uk/how/guides/evaluation/resourcesManchester Beacon Evaluation Guide http://www.manchesterbeacon.org/about/UCL Evaluation Toolkithttp://www.ucl.ac.uk/public-engagement/research/toolkits/Event_EvaluationRCUK Evaluation Guidehttp://www.rcuk.ac.uk/documents/publications/evaluationguide.pdfHE STEM http://www.hestem.ac.uk/evaluationInspiring Learning for All http://www.inspiringlearningforall.gov.uk/toolstemplates/Useful Resources

×