This document outlines an agenda and content for a workshop on monitoring and evaluation techniques. The all-day workshop covers key concepts, logic chains, quantitative and qualitative data collection methods, quantitative evaluation techniques like sampling and extrapolation, using case studies and qualitative stories, and calculating value for money. Participants will work in groups to develop logic chains for their own projects, discuss best practices for gathering beneficiary feedback, and begin drafting an evaluation plan. The goal is to provide practical skills and strategies for organizations to effectively demonstrate their impact.

1. Proving Your Impact:
monitoring & evaluation techniques
4th February 2015
Robin Beveridge and Tony Sacco

2. • Domestics
• Who’s who
• What’s in store today
• Ground rules
• Networking and learning from each other
Introduction

3. Structure of the day: AM
9.30 Introductions and warm up
9.45 Key concepts and logic chains
Break
10.45 Capturing information from beneficiaries – what works?
11.30
Lunch
Quantitative evaluation techniques

4. Structure of the day: PM
14.10
15.00
Workshop feedback and close
Evaluation in practice
Wowing the world
13.00
15:30
‘Telling the story’ using case studies
13.30 Value for money

5. Getting started
• Over the next 5 minutes…
Write down 3 things you hope to get out of today on ‘post-its’
Speak with someone you have never met before today
Introduce yourselves and swap one of your ‘hopes’
Repeat this with two more people
• Then…
…post your ‘post-its’ up on the wall.

6. Setting the ground rules…

7. Setting the ground rules…

8. Key Concepts
• Monitoring
• Evaluation
• Outputs
• Outcomes
• Impact
• Aims & Objectives
• Quantitative
• Qualitative
• Beneficiaries
• Stakeholders
• Sample/Sampling
• Extrapolation
• Value for Money
• Additionality
• Social Return
• Gross and Net

9. Logic chain example 1: This workshop
Caused by
Output
Need Activity
Impact Outcome
Limited ability to
demonstrate
impact
Workshops and
peer learning
People trainedHigher level of
skills and
knowledge
Better
demonstration of
impact
Lack of
Monitoring and
evaluation
knowledge
Rationale

10. Logic chain example 2: Streetwise
Caused by
Output
Need Activity
Impact Outcome
High level of
teenage
pregnancy
Providing
C-cards
Young people
accessing
condoms
Less un-protected
sex
Lower level of
teenage
pregnancy
Poor access to
contraceptives

11. Working in pairs
• Using templates provided, develop a ‘Logic
Chain’ for a project you are involved with
• Share that with someone on your table
• Feed back to the workshop:
– What was easy, what was hard?
– Does it make sense?

12. Break

13. Capturing information from beneficiaries –
What works?
• In groups consider…
–What are the different ways of capturing
information from participants /
beneficiaries?
–Identify some ‘pros and cons’ of each
approach.

14. Capturing information from beneficiaries –
Some thoughts
Questionnaires
• Wide coverage
• Tick box / narrative approach?
• Accessibility
• Self-completed or ‘managed’
• ‘Survey Monkey’
• Before / after / distance travelled
• Confidentiality
• Response rate

15. Capturing information from beneficiaries –
Some thoughts

17. Capturing information from beneficiaries –
some thoughts
Feedback forms
• Immediate feedback from ‘captive
audience’?
• More of / less of
• Most useful / least useful
• Improvements
• Can be superficial and ‘too immediate’
• Follow up again later?

18. Capturing information from beneficiaries –
some thoughts
In-depth interview
• Richer content
• Particular insights
• Range of views / perspectives
• Resource intensive

19. Capturing information from beneficiaries –
some thoughts
Focus Groups
• A group with similar characteristics
(e.g. single parents)
• ‘Managing’ the group
• Recording the outcomes
Round Table
• Different perspectives

20. Capturing information from beneficiaries –
some thoughts
Comments / concerns / compliments
• Forms / Box?
• Managing the process
• The feedback loop

23. Timing issues…
• Captive audience
• Getting buy-in from the outset
• ‘When they’re gone they’re gone!’
• Bias?
• Incentives?

25. Storage issues…
• Paper based
• Spreadsheets
• Databases
• Bespoke approaches?

26. Questionnaire Design Considerations
• Questionnaire
– Introduction / opening questions
– Meaningful order / clear sections
– Thank you / return information
• Questions
– Numbered
– Clear instructions
– Routing
– Layout
• Testing
– Peer review; Cognitive interview;
– Focus group; pilot/field test

27. Don’ts…
• Try to avoid…

28. …and do’s
• Try to avoid…

29. What more creative ways can you come up
with to gather information from beneficiaries?
Outside the box…

30. Quantitative techniques

31. • Why numbers matter
• What numbers do we
need?
• How to get useful
numbers
• Cross-tabulation
• Sampling and
extrapolating
• How to present data
Quantitative techniques
0
10
20
30
40
50
%
Reaction to the word
“Spreadsheet”
Men
Women
n.b. fictional data!

32. Key numbers
The project has achieved significant positive
outcomes, with 447 young people supported to
become enterprise ready up to December 2011.
To date, roughly 1 in 4 of these beneficiaries
have gone on to establish a business, and
around 30% have moved into education, training
or employment.”
“An independent evaluation found
that the service costed £145,000 over
three years, and had resulted in
significant cost savings to statutory
services: conservatively estimated at
£1,211,630.”
• How many
beneficiaries?
• Of what sort?
• What happened as a
result?
• What did it cost?
• How much did it save?
• Was it good value for
money?

33. CDCF/ESF Requirements

34. • Filters
• Count
• Pivot Tables
• Cross
Tabulations
Getting the numbers: 1) Excel
% of beneficiaries Under 25 26-50 Over 50 Total
Northumberland 0 5 15 20
Tyne & Wear 10 20 10 40
County Durham 15 10 0 25
Tees Valley 0 15 0 15
TOTAL 25 50 25 100
Count of Completed Column Labels
Row Labels Jan Feb Mar Apr May Jun Jul Aug Sep Oct
Grand
Total
f 12 3 3 6 6 3 3 6 9 51
m 3 3 6 9 6 6 3 3 39
Grand Total 12 6 3 9 6 15 9 9 9 12 90

35. CDCF/ESF Data – Radar Charts
0
1
2
3
4
5
6
7
8
9
10
Motivation
Confidence
Forms
CV
Interview
IT
Average
Before
After
0
1
2
3
4
5
6
7
8
9
10
Motivation
Confidence
Forms
CV
Interview
IT
Adriana
Before
After

36. Getting the numbers: 2) Other
• Survey Monkey
• Star Online
• Databases:
– Access
– Off the shelf systems
– Bespoke design

37. Sampling and extrapolation
• Most evaluation is based on a sample
• How representative is your sample?
• How diverse is your ‘population’?
• Evaluation is NOT a science
• Be aware of potential ‘bias’ in the sample
• Use caveats in extrapolating

38. Sampling example
0
20
40
60
80
100
120
1 2 3 4-6 7-12 12+
Months from completion
Length of time to find work
We found that 100% of respondents had
found work within 3 months of
completing the course. If this is true for
all beneficiaries, we will have helped 300
people to find work”

39. Sampling thoughts
Try to get at least 10% of
each part of the
population:
• By geography
• By gender
• By age
• By time/stage
• Other
characteristics…
To address bias in your sample:
• Use a variety of methods
• Ask searching/critical
questions
• Triangulate with stakeholder
views
• Pay attention to outlier views

40. Presenting Data – some tips
• Not too much information
• Order bars from high to low
• Use a variety of formats, but not too many
• Keep a consistent colour scheme
• Remember titles, axes and legends
• Keep it as clean as possible

41. For example
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
Ethicity
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
White: British White: Other Other Ethnicity
Ethnicity Other Ethnicity
Mixed: White and Black Caribbean
Mixed: White and Black African
Mixed: White and Asian
Mixed: Other Mixed
Asian or Asian British: Indian
Asian or Asian British: Pakistani
Asian or Asian British: Bangladeshi
Asian or Asian British: Other Asian
Black or Black British: Black Caribbean
Black or Black British: Black African
Black or Black British: Other Black
Chinese or Other Ethnic Group: Chinese
Chinese or Other Ethnic Group: Other

42. Presenting Data – some tips
Use free tools, eg:
• Google Fusion
Tables
• Tableau public
• Plot.ly
• Any others?

43. That’s quite enough about data

44. Lunch

45. Recap

46. Structure of the day: PM
14.10
15.00
Workshop feedback and close
Evaluation in practice
Wowing the World
13.00
15:30
‘Telling the story’ using case studies
13.30 Value for money

47. Telling the Story
Using Case Studies
Spend a couple of minutes discussing with
one or two others…
Things to consider when producing an ESF
Community Grants Case Study

48. Case Studies: ESF Approach
About the person:
• Details of the client’s background, education/training
and employment history and current situation.
Training/support:
• Details of what activity took place during the project
• What the client feels s/he achieved in regard to soft
skills (e.g. more confidence) or hard outcomes (e.g. a
qualification, employment, or volunteering).

49. Case Studies: ESF Approach
Plans for the future
• How does the client hope to progress their
employability, training or job searching?
• Have they been referred to another organisation for
ongoing support – if so which and what type ?
Quote
• A short quote from the client on how the project has
made a difference.
Signature

50. Telling the Story
Using Case Studies
• The power of the story
• The power of the actual words…
• … and a picture paints 1,000 of them

51. The Power of the Story

52. The Power of the Actual Words
• ‘If I didn’t have this I would
have reoffended. I would
have ended up getting
drunk, fighting with
someone and be back in
prison.’
• ‘I felt miserable, I had no
money and I was on
benefits but now I’m
independent. I feel good
about working for my
money.’

53. …and a picture…

54. Case Studies: A Quick Checklist
• Before you start
• Setting up the interview
• On the day
• After the event
• Other thoughts

55. Brandon’s Story

56. Value for Money
• What is the additional
outcome impact?
• What is the cost per
outcome?
• What is the return to
society?

57. Gross No of
people into employmentPeople into employment
who wouldn’t otherwise
Would have got a job
anyway (deadweight)
Eligible people into
employment who
wouldn’t otherwise
Would have got a job
anyway (deadweight)
Ineligiblepeople
(leakage)
Additional spend locally
(multiplier)
Additional
eligible people
into
employment
who wouldn’t
otherwise
Net/additional impact
Ineligiblepeople
(leakage)
Would have got a job
anyway (deadweight)Otherpeoplewhodon’t
getorlosejobs
(displacement)
Additional
eligible people
into
employment
who wouldn’t
otherwise
Net additional impact
Additional
eligible people
into
employment
who wouldn’t
otherwise

58. Net additional impact
Work though
part 1 of the
worksheet

59. Net additional impact Results
Gross Outcomes 250
Deadweight effect (those who’d have got a job anyway) 50
Gross Outcomes less Deadweight 200
Leakage effect (ineligible beneficiaries) 10
Gross Outcomes less Deadweight and Leakage 190
Displacement effect (other people losing jobs as a result) 0
Gross Outcomes less Deadweight, Leakage and Displacement 190
Multiplier effect 19
Gross Outcomes less Deadweight, Leakage and Displacement,
plus Multiplier effect (“Net Additional Outcomes”)
209

60. Cost per outcome
• Simple concept: TOTAL COST
NO. of OUTCOMES
• BUT: Gross or Net Outcomes?
• Total cost or Total Funding?
• See Part 2 of Workbook

61. Cost per outcome Results
Total Cost per Outcome (Gross) £3,000
ESF per Outcome (Gross) £1,350
Total Cost per Net Outcome £6,000
ESF per Net Outcome £2,700

62. Social Return
• The financial value to society of the outcomes
• e.g. moving someone off benefit into work:
– Benefits saved
– Taxes contributed
– Local spend
– Reduced risk of costs arising from ill health, crime,
anti-social behaviour

63. Social Return
• Depends on lots of assumptions (rate of pay,
length of employment, behaviour of the
individual, age, personal circumstances etc)
• Can calculate/estimate on a case-by-case –
difficult, lengthy and expensive
• Or use ready-made estimates e.g. Global
Value Exchange

64. Global Value Exchange
• Free to join and use
• Open source
estimates of the
financial value of a
huge range of
outcomes
www.globalvaluexchange.org

65. Social Return Results
Value to individuals of securing work £323,010
Value of volunteering £2,600
Value of increased earnings potential (excluding those
who have secured work)
£131,040
Value of fiscal savings to DWP £277,020
Total Social Return of the project £733,670

66. Social Return continued…
• Can then divide Total Social Return by Total
Cost to get return on investment e.g. if Project
Z cost £200,000, SROI is
£733,670 / £200,000 = £3.67
• Could have included other factors, e.g.
– Value of volunteering
– Increased earnings potential
– Improved health

67. SROI warnings
• Result depends on the factors included
• Also depends on assumptions made
• Similar projects making different assumptions will
generate very different SROI scores
• Dealing with time properly is complex
• Doing SROI properly is expensive
• Most projects can demonstrate high positive
SROI, so even an high positive result may not
unlock funding

68. Developing an Evaluation Plan
for your project

69. Developing an Evaluation Plan:
Some things to think about
• Scope and Purpose
– What?
– Level – e.g. Project, Programme, Organisation
– Formative / Summative
– Audience?

70. Developing an Evaluation Plan:
Some things to think about
• Timing and Phasing
– When?
– Single phase or multiphase?
• How do they fit together?
– Previous evaluations?
• The same approach or different?
– Key deadlines?
• ‘Doomed to success’

71. Developing an Evaluation Plan:
Some things to think about
• Collecting Data
– What is required?
• Back to the logic chain
– Where can you get the data
– What have you got already?
– Storage and analysis
– Any barriers to using data?

72. Developing an Evaluation Plan:
Some things to think about
• Who does it?
– Internal / External?
– Buy-in?
– Objectivity
– Budget

73. Developing an Evaluation Plan:
Some things to think about
• Collecting views
– Options for beneficiaries?
– Key stakeholders?
– Options for Stakeholders

74. Developing an Evaluation Plan:
Some things to think about
• Managing the evaluation process
– Who is responsible?
– Sign off?
• Dissemination
– Who and how?

75. Developing an Evaluation Plan
for your project

76. Wowing the World
• Why are we doing this again?
– Because we have to?
– To improve our services to clients?
– To attract new clients?
– To demonstrate to ourselves and / or others that
what we’re doing is effective?
– To show Value for Money?
– To convince people to give us some more?

77. Wowing the World
• Whatever the reason we need
to ‘broadcast it’ in some way
• What approaches have you
used that have been
particularly effective in
broadcasting what you’ve
achieved and the impact that it
has made?

78. Wowing the World

79. The word on the street
• Evaluation Reports
• Summary Documents
– Selected highlights
• Newspapers / Newsletters
• Annual Reports

80. The word on the street

81. The word on the street

82. The writing on the wall…

83. Multi Media

84. Summary and questions

85. Your Evaluation!
• Any immediate thoughts?
• Evaluation sheet – Free prize
draw!
• A more detailed conversation?
• Peer support network?