This presentation discusses the importance of measuring outcomes to demonstrate the impact and effectiveness of projects. It emphasizes that funders want to see evidence that a project will achieve measurable changes. The presentation provides guidance on defining needs, identifying appropriate outcomes and indicators, and measuring progress. It also explains why randomized controlled trials provide the strongest level of evidence and discusses key criteria like minimizing bias that are important for reliable evidence.

1. Presentation Title
Evidence – helping understand
what works
Michael O’Toole
Chief Executive, Mentor

2. Presentation Title
• Introduction
• Thinking about projects impact
• Funding & commissioning for outcomes
• Why measure outcomes
• How to measure
• Why good evidence is important
• Standards for good evidence
Agenda
More than a
numbers
game

3. Presentation Title
• Outputs
• Outcomes
• Impact
• Aim or mission
• ‘So What?’
• Thinking about projects impact
More than a
numbers
game

4. Presentation Title
• Project in 3 sentences of no more than
20 words each
• like an impact elevator pitch
• What the programme does
• Immediate changes brought about
• Longer-term change project
contributes to
• Theory of change
More than a
numbers
game

5. Presentation Title
• Statement 1 describes what we do &
who you work with
• outputs & target group
• Statement 2 describes the changes we
hope to make in your client group
• Outcomes
• Theory of change
More than a
numbers
game

6. Presentation Title
• Statement 3 talks about your impact
• broader, longer-term changes,
often beyond your client group
• could be impacts in a community,
systemic impacts, social value
impact or inter-generational
impacts
• This statement is like a mission
statement
• Theory of change
More than a
numbers
game

7. Presentation Title

8. Presentation Title
What’s the problem?
Easy mistake to start by describing who you want
to work with and what you want to do without
first being clear about the need you are
addressing.
Careful analysis of the need is an important first
step in project planning and developing effect
evaluation.
Funding & commissioning for outcomes
More than a
numbers
game

9. Presentation Title
• Meeting needs
• A problem or issue, or situation
where something needs to change
to make things better for a person
or a community
Funding & commissioning for outcomes
More than a
numbers
game

10. Presentation Title
• Careful analysis of need is an
important first step in planning a bid
• How do you know the unmet need
exists and why?
• Provide recent and relevant evidence:
• people with experience of the need
• research into the scale of the need
• local statistics about the population in
the area or levels of deprivation
Funding & commissioning for outcomes
More than a
numbers
game

11. Presentation Title
Needs checklist
What is the problem or issue the project will
address?
What evidence is there that this need exists?
What are the reasons for the need?
What research supports this?
Why prioritise this need?
How is your project an appropriate response
to the need?
Have you consulted with the people your project
will benefit?
What other initiatives, or other projects work in
this field and how your project will fit in?
Funding & commissioning for outcomes
More than a
numbers
game

12. Presentation Title
• Measuring the effectiveness of an
intervention
• How do you know if a program was
effective?
• Identify effective practices
• Determine which activities to continue,
build upon or modify
• Identify practices that need
improvement
• Change in order to improve effectiveness
or stop!
• Clarity & consensus on purpose
• Prove value to existing and potential
funders
Why measure outcomes
More than a
numbers
game

13. Presentation Title
• Funders will want to know:
• scale or level of the change the project
will make
• how you will measure the change in order
to track progress to outcomes
• Outcome indicators can be
quantitative or qualitative
Why measure outcomes
More than a
numbers
game

14. Presentation Title
• Consider measuring progress by
imagining meeting a beneficiaries
prior to the intervention then again
after a month or year:
• What will be different for them in terms
of their wellbeing, skills, attitude,
knowledge, etc?
Why measure outcomes
More than a
numbers
game

15. Presentation Title
• Indicators help understand change &
track progress towards achieving
change
• An indicator is a signal that the
outcome is being achieved
• Indicators can be
• the extent of...
• the ability to...
• the amount of...
• Should have at least one indicator for
each outcome
How to measure outcomes
More than a
numbers
game

16. Presentation Title

17. Presentation Title
• Evidence-based programmes provide
funders reliable and validated
information
• Establish “what works,” as well as
what counts as good evidence
• Usually comprise of two components:
• strong magnitude of impact
• a fair and rigorous methodological
approach
• A causal relationship between
implementation of the programme
and outcomes of the intervention
Why good evidence is important
More than a
numbers
game

18. Presentation Title
• Level of evidence pyramid
Why good evidence is important
More than a
numbers
game

19. Presentation Title
• Randomised control trials
• Gold standard
• Repeated, systematic review RCTs
• recording same or similar result each
time a study is repeated
• minimising biases linked to procedures of
the study
• Ethical dilemmas
Why good evidence is important
More than a
numbers
game

20. Presentation Title
• Important criteria for conducting an
RCT:
• Independent evaluation
• Transferability and generalisability
(degree to which results go beyond the
initial study. Generalisability is relevance
of programme to a definable group
• Statistical power of the analysis
(study design and methodology)
• Minimum bias
(measures taken to eliminate biases)
• These criteria are fundamental to the
reliability and appropriateness of a fair
and well-conducted RCT
Why good evidence is important
More than a
numbers
game

21. Presentation Title
• Levels of evidence grades
• Further information about repository
of CAYT repository of impact studies -
here
Why good evidence is important
More than a
numbers
game

22. Presentation Title
• Twitter: otoole_michael
• Email: michael.otoole@mentoruk.org
• Web: www.mentoruk.org.uk
Contact details
More than a
numbers
game