Slides from a workshop on Monitoring & Evaluation delivered for County Durham Community Foundation on Monday 10 November 2014.

1. Proving Your Impact:
Pain-free Monitoring and Evaluation
10th November 2014
Robin Beveridge and Tony Sacco

2. Introduction
• Domestics
• Who’s who
• What’s in store today
• Ground rules
• Networking and learning from each other

3. Structure of the day: AM
9.30 Introductions and warm up
9.45 Monitoring & Evaluation: Key Concepts
10.00 Practical session: developing project ‘logic chains’
Break
11.00 Capturing information from beneficiaries – what works?
11.30 ‘Telling the story’ using case studies
Lunch

4. Structure of the day: PM
13.00 Recap and Review
13.10
13.50
14.10
14.50
Quantitative Evaluation techniques
Evaluation in practice
Monitoring & Evaluation plans
Wowing the World
Workshop feedback and close
15:30

5. Getting started
• Over the next 5 minutes…
Write down 3 things you hope to get out of today on ‘post-its’
Speak with someone you have never met before today
Introduce yourselves and swap one of your ‘hopes’
Repeat this with two more people
• Then…
…post your ‘post-its’ up on the wall.

6. Setting the ground rules…

7. Setting the ground rules…

8. Key Concepts
• Monitoring
• Evaluation
• Outputs
• Outcomes
• Impact
• Aims
• Objectives
• Quantitative
• Qualitative
• Beneficiaries
• Stakeholders
• Value for Money
• Additionality
• Social Return

9. Logic chain example 1: This workshop
Limited ability to
demonstrate
impact
Need Activity
Caused by
Output
Impact Outcome
Workshops and
peer learning
Higher level of People trained
skills and
knowledge
Better
demonstration of
impact
Lack of
Monitoring and
evaluation
knowledge
Rationale

10. Logic chain example 2: Streetwise
High level of
teenage
pregnancy
Need Activity
Caused by
Output
Impact Outcome
Providing
C-cards
Young people
accessing
condoms
Less un-protected
sex
Lower level of
teenage
pregnancy
Poor access to
contraceptives

11. Logic chain example 3: Bright Futures
Young women in
South Tyneside
facing health and
social problems
Need Activity
Caused by
Output
Impact Outcome
Courses, advice
sessions, peer
education,
volunteering
Young women
engaged in
programmes/cou
rses or given
advice
Young women
better equipped
to succeed in life
Reduced health
and social
problems for
young women in
South Tyneside
Lack of life skills
and support
systems

12. Logic chain example 4: Action Foundation
Vulnerable
asylum –seekers
without
accommodation
Need Activity
Caused by
Output
Impact Outcome
Provision of free
emergency
accommodation
Asylum-seekers
housed and
supported
Asylum-seekers
have stability and
ability to access
services for basic
needs
Chronic cycle of
poverty / poor
health / illegal
work prevented
No recourse to
public funds and
no permission to
work

13. Working in pairs
• Using templates provided, develop a ‘Logic
Chain’ for a project you are involved with
• Share that with someone on your table
• Feed back to the workshop:
– What was easy, what was hard?
– Does it make sense?

14. Break

15. Capturing information from beneficiaries –
What works?
• In groups consider…
–What are the different ways of capturing
information from participants /
beneficiaries?
– Identify some ‘pros and cons’ of each
approach.

16. Capturing information from beneficiaries –
Some thoughts
Questionnaires
• Wide coverage
• Tick box / narrative approach?
• Accessibility
• Self-completed or ‘managed’
• ‘Survey Monkey’
• Before / after / distance travelled
• Confidentiality
• Response rate

17. Capturing information from beneficiaries –
Some thoughts

19. Capturing information from beneficiaries –
some thoughts
Feedback forms
• Immediate feedback from ‘captive
audience’?
• More of / less of
• Most useful / least useful
• Improvements
• Can be superficial and ‘too immediate’
• Follow up again later?

20. Capturing information from beneficiaries –
some thoughts
In-depth interview
• Richer content
• Particular insights
• Range of views / perspectives
• Resource intensive

21. Capturing information from beneficiaries –
some thoughts
Focus Groups
• A group with similar characteristics
(e.g. single parents)
• ‘Managing’ the group
• Recording the outcomes
Round Table
• Different perspectives

22. Capturing information from beneficiaries –
some thoughts
In-depth interview
• Richer content
• Particular insights
• Different perspectives
• Resource intensive

23. Capturing information from beneficiaries –
some thoughts
Comments / concerns / compliments
• Forms / Box?
• Managing the process
• The feedback loop

26. Timing issues…
• Captive audience
• Getting buy-in from the outset
• ‘When they’re gone they’re gone!’
• Bias?
• Incentives?

28. Storage issues…
• Paper based
• Spreadsheets
• Databases
• Bespoke approaches?

29. Outside the box…
What more creative ways can you come up
with to gather information from beneficiaries?

30. Telling the Story
Using Case Studies
Spend a couple of minutes discussing with
one or two others…
When and why might we use a case study?

31. Telling the Story
Using Case Studies
• The power of the story
• The power of the actual words…
• … and a picture paints 1,000 of them

32. The Power of the Story

33. The Power of the Actual Words
• ‘If I didn’t have this I would
have reoffended. I would
have ended up getting
drunk, fighting with
someone and be back in
prison.’
• ‘I felt miserable, I had no
money and I was on
benefits but now I’m
independent. I feel good
about working for my
money.’

34. …and a picture…

35. Case Studies: ESF Approach
About the person:
• Details of the client’s background, education/training
and employment history and current situation.
Training/support:
• Details of what activity took place during the project
• What the client feels s/he achieved in regard to soft
skills (e.g. more confidence) or hard outcomes (e.g. a
qualification, employment, or volunteering).

36. Case Studies: ESF Approach
Plans for the future
• How does the client hope to progress their
employability, training or job searching?
• Have they been referred to another organisation for
ongoing support – if so which and what type ?
Quote
• A short quote from the client on how the project has
made a difference.
Signature

37. Case Studies: A Quick Checklist
• Before you start
• Setting up the interview
• On the day
• After the event
• Other thoughts

38. Brandon’s Story

39. Lunch

40. Recap

41. Structure of the day: PM
13.00 Recap and Review
13.10
13.50
14.10
14.50
Quantitative Evaluation techniques
Evaluation in practice
Monitoring & Evaluation plans
Wowing the World
Workshop feedback and close

42. Quantitative techniques

43. Quantitative techniques
• Why numbers matter
• What numbers do we
need?
• How to get useful
numbers
• Cross-tabulation
• Sampling and
extrapolating
• How to present data
50
40
30
20
10
0
%
Reaction to the word
“Spreadsheet”
Men
Women
n.b. fictional data!

44. Key numbers
The project has achieved significant positive
outcomes, with 447 young people supported to
become enterprise ready up to December 2011.
To date, roughly 1 in 4 of these beneficiaries
have gone on to establish a business, and
around 30% have moved into education, training
or employment.”
• How many
beneficiaries?
• Of what sort?
• What happened as a
result?
“An independent evaluation found
that the service costed £145,000 over
three years, and had resulted in
significant cost savings to statutory
services: conservatively estimated at
£1,211,630.”
• What did it cost?
• How much did it save?
• Was it good value for
money?

45. Getting the numbers
• Excel
– Pivot Tables
– Filters
• Survey Monkey
• Database Systems

46. Cross tabulation
% of beneficiaries Under 25 26-50 Over 50 Total
Northumberland 0 5 15 20
Tyne &Wear 10 20 10 40
County Durham 15 10 0 25
Tees Valley 0 15 0 15
TOTAL 25 50 25 100

47. Sampling and extrapolation
• Most evaluation is based on a sample
• How representative is your sample?
• How diverse is your ‘population’?
• Evaluation is NOT a science
• Be aware of potential ‘bias’ in the sample
• Use caveats in extrapolating

48. Sampling example
120
100
80
60
40
20
0
Length of time to find work
We found that 100% of respondents had
found work within 3 months of
completing the course. If this is true for
all beneficiaries, we will have helped 300
people to find work”
1 2 3 4-6 7-12 12+
Months from completion

49. Presenting Data – some tips
• Not too much information
• Order bars from high to low
• Use a variety of formats, but not too many
• Keep a consistent colour scheme
• Remember titles, axes and legends
• Keep it as clean as possible

50. For example

51. Presenting Data – some tips
Use free tools, eg:
• Google Fusion
Tables
• Tableau public
• Any others?

52. That’s quite enough about data

53. Developing an Evaluation Plan:
Some things to think about
• Scope and Purpose
– What?
– Level – e.g. Project, Programme, Organisation
– Formative / Summative
– Audience?

54. Developing an Evaluation Plan:
Some things to think about
• Timing and Phasing
– When?
– Single phase or multiphase?
• How do they fit together?
– Previous evaluations?
• The same approach or different?
– Key deadlines?
• ‘Doomed to success’

55. Developing an Evaluation Plan:
Some things to think about
• Collecting Data
– What is required?
• Back to the logic chain
– Where can you get the data
– What have you got already?
– Storage and analysis
– Any barriers to using data?

56. Developing an Evaluation Plan:
Some things to think about
• Who does it?
– Internal / External?
– Buy-in?
– Objectivity
– Budget

57. Developing an Evaluation Plan:
Some things to think about
• Collecting views
– Options for beneficiaries?
– Key stakeholders?
– Options for Stakeholders

58. Developing an Evaluation Plan:
Some things to think about
• Managing the evaluation process
– Who is responsible?
– Sign off?
• Dissemination
– Who and how?

59. Developing an Evaluation Plan
for your project

60. Wowing the World
• Why are we doing this again?
– Because we have to?
– To improve our services to clients?
– To attract new clients?
– To demonstrate to ourselves and / or others that
what we’re doing is effective?
– To show Value for Money?
– To convince people to give us some more?

61. Wowing the World
• Whatever the reason we need
to ‘broadcast it’ in some way
• What approaches have you
used that have been
particularly effective in
broadcasting what you’ve
achieved and the impact that it
has made?

62. Wowing the World

63. The word on the street
• Evaluation Reports
• Summary Documents
– Selected highlights
• Newspapers / Newsletters
• Annual Reports

64. The word on the street

65. The word on the street

66. The writing on the wall…

67. Multi Media

68. Summary and questions

69. Your Evaluation!
• Any immediate thoughts?
• Evaluation sheet – Free prize
draw!
• A more detailed conversation?
• Peer support network?