Measuring Impact - An Engage Workshop on Monitoring & Evaluation

Measuring ImpactAn Engage Workshop onMonitoring & Evaluation

Workshop Aims• To deepen ROEYP grant holders’ understanding of monitoring and evaluation.• To assist grant holders in starting to identify appropriate methods and means of measuring the impact of their work.• To help ROEYP grant holders network and make connections between their projects

Intended outcomes for workshop• Increased confidence in using the language/ terminology of monitoring & evaluation• Greater understanding of evaluation methods and means to measure impact• More connections between projects funded under the ROEYP Programme

Workshop overview• Why measure impact (evaluate)?• Definitions & clarification of terms• Learning from other participants: approaches & tools Evaluation Template, Youth Concern Rickter Scale, Include Youth, Wemwebs, YANI Assessment Tool based on Farrington’s Factors, NIACRO• Involving young people in monitoring & evaluation – some ideas• Discussion: questions, queries and challenges• Resources, evaluation & close

Why measure impact (evaluate)?Some reasons ...• Accountability (is the money being spent as it should be?)• To find out what is the impact of the project• To explore how the project could be improved• To understand user perspectives (what is their experience of the project?)• To help with decision-making about the project (cut, or continue – all or part?)• To inform the replication or expansion of a pilot project• To inform future marketing or publicity about the project

What is evaluation?• Derived from the word ‘value’ (how much? how good?)• Relates to a specific project, programme or service, at a particular point in time• A series of planned and systematic activities• Has a clearly defined purpose• Gathering evidence in order to make judgements• Multi-method (typically)• Multiple perspective (if possible)

Evaluation: a working definition• Evaluation consists of a set of systematic, planned activities designed to assess the worth or value of a project or programme.

Over-arching evaluation questions• What / how much did we do? (monitoring)• How well did we do it? (process)• Did we make a difference? (outcomes/impact)

Monitoring• Underpins any evaluation• Answers the ‘what/how much/how many?’ question• A systematic record of relevant information about the project• Recording ‘inputs’ and ‘outputs’• Keeping track of project progress• Relevant information about service users• Monitoring data will usually be in numerical form (quantitative)• Monitoring is not simply collecting this information, it is about using it (reviewing, reflecting, comparing, analysing)

Outcome / impact (evaluation)• Answers the ‘did we make a difference?’ question• Assessing the ‘difference’ that the project has made – eg to users (and others?)• Outcomes and impacts need to be expressed in the language of change (improve, decrease, enhance etc)• To measure change conclusively, a before-after model is required (although there may still be issues of attribution)• Sometimes ‘softer’ assessments of change (eg qualitative views, or retrospective accounts) may be more practical for a DIY evaluation• It may be appropriate to consider longer-term or broader impacts (including unintended outcomes)

Methods of collecting evidence• Adapt or extend existing systems• ‘Spoken word’ methods (face-to-face or phone, individually or in groups)• Written methods eg questionnaires, diaries• Standard ‘tests’ or assessments• Observation• ‘Creative’ methods eg pictures, photos

Evaluation Template• Ricky Linton• Youth Concern, Whitehead

The Rickter Scale• Nicci Smith• Include Youth

Wemwbs• Eliz McArdle• YANI

Assessment tool(bases on Farrington’s Factors)• Tony Martin• NIACRO

Methods of collecting evidence• Evaluation does not have its own exclusive set of methods• No one method is ‘better’ than any other, each have their advantages and disadvantages• Important to select the most appropriate method(s) for your specific needs and circumstances• Can mix and match methods for different aspects of your evaluation• Different methods can provide complementary data

When selecting methods, things to bear in mind• Who are your informants? (age, skills, expectations)• What kind of information are you collecting – Qualitative and/or quantitative? – Sensitive? – Complex?• Likely numbers involved• Anonymity / confidentiality issues• Your own analysis skills / confidence• Resource implications• Time implications

Involving children & young peoplein monitoring and evaluation• Some ideas….

When to collect evidence?• Ongoing (for monitoring data)• After the ‘intervention’ is complete (for retrospective view)• Before and after the intervention (if you want to measure change, or compare expectations with reality)• Additional follow-up (if you want to look at longer term impact or outcomes)• At other appropriate points eg midway point if you want to assess progress and make changes, or ‘snapshot’ to gather wide range of views at particular point in time

AnalysisThe last minor (!) steps:• Analysis - making sense of all the information you have collected• Keep it simple! – Think back to your evaluation questions – Collate your data into a workable format – Describe the facts in a balanced way (include positives and negatives) – Interpreting the facts – what do they signify and why?

Reporting• Reporting – presenting it in a way that makes sense to target audience – Planned structure (logical sequence) – Be concise and keep it simple – Include evidence to support your findings (e.g frequencies or percentages for quantitative data, quotations for qualitative, cameos for observations) – Tables and graphs a useful way of presenting quantitative data

Measuring impact…• Challenges• Concerns• Queries

Measuring impact• Resources• Evaluation

Want more info?Teresa Geraghtytgeraghty@ncb.org.uk028 90891733

Measuring Impact - An Engage Workshop on Monitoring & Evaluation

by Participation Works

Accessibility

Categories

Upload Details

Usage Rights

Statistics

Measuring Impact - An Engage Workshop on Monitoring & Evaluation Presentation Transcript