A training session which Victoria Pelka, former Enactus Sussex team leader ran at the Enactus UK training weekend in 2013.

1. Impact Evaluation
with Vicky Pelka

2. What awaits you…
• Enactus UK first ever Impact
Evaluation session!!!
• 1 hour…
• Reflect
• Ask Questions
• electronic Hand-outs available on
request!

3. Who am I?
• Vicky Pelka
• Ex- Enactus Sussex President
• Research and Evaluation Coordinator
at a London-based charity
• I evaluate the impact of our projects

4. Learning Outcome
Statements
1. I can define impact evaluation.
2. I understand why impact evaluation is important
and useful.
3. I know at what points in a project cycle impact
evaluation should take place.
4. I know what kind of things can be measured in
order to do impact evaluation.
5. I can name some research methods used in
impact evaluation.
6. I can name some ways to communicate the
impact of a project.

5. Definitions
• Need
• Empowerment
• Quality of Life (QOL)
• Standard of Living (SOL)
• Impact
• Evaluation

6. (PEOPLE in )Need
Lack of something requisite, desirable, or useful.
Needs range from basic survival needs, to cultural,
intellectual, and social needs (Maslow’s pyramid of
Needs).
Empowerment
The process of increasing the capacity of individuals
(or groups) to apply new skills and gain self-confidence,
so they achieve a stage of
independence.

7. Quality of Life (QOL)
Intangible components of someone’s life
e.g. satisfaction with life, confidence, self-esteem,
social status, emotional / psychological well-being,
etc.
Standard of Living (SOL)
Tangible components of someone's’ life
e.g. employment, income, housing, access to clean
drinking water or sanitation, healthcare, food
provision, etc.

8. Impact
The positive and negative, short-term and long-term
changes produced by a project.
Impact can be direct or indirect, intended or
unintended.
Evaluation
The objective assessment of an on-going or completed
project, its design, implementation and results.
The aim is to determine the relevance and fulfilment of
objectives, effectiveness, impact and sustainability.

9. WHY is it important?
It’s in the Enactus Criterion
Considering the relevant social, economic and
environmental factors,
which Enactus team most effectively
empowered people in need
by applying business and economic concepts
and an entrepreneurial approach,
to improve their quality of life and standard of
living.

10. WHY is it important?
It’s in the in the Enactus Criterion
Considering the relevant social, economic and
environmental factors,
which Enactus team most effectively
empowered people in need
by applying business and economic concepts
and an entrepreneurial approach,
to improve their quality of life and standard of
living.

11. Why is it important?
• To know for yourself
• To tell others
(university, students, committed and
potential funders)
• To improve
• To let beneficiaries have a say!

12. What Impact Evaluation are YOU
already doing in your team
and when?
(Take 2 minutes to tell your neighbour)

13. Project plans
you have seen so far
Need
• Find people in
need
Solution
• Apply business and
economic concepts
• Take an
entrepreneurial
approach
• Empower
Impact
• Improve quality
of life and
standard of
living

14. Bringing in
Impact Evaluation
Need
• Find
people in
need
Solution
• Apply business and
economic concepts
• Take an
entrepreneurial
approach
• Empower
Impact
• Improve
quality of life
and standard
of living
Conduct Needs
Assessment
Collect Baseline Data
Record Outputs
Monitor Progress
Adjust/ Improve
if necessary
Measure Outcomes:
Measure QOL and
SOL changes
Review effectiveness
and sustainability

15. What Impact Data
is worth collecting?
• Outputs – what you have done
e.g. delivered 5 business workshops, carried out
30 hours of teaching
• Outcomes - what this means…
e.g. successfully established one business,
doubled beneficiaries’ income, enabled a family
to send 5 children to high school

16. What Impact Data
is worth collecting?
• Quantitative – relates to number, statistics,
quantities, specifics
e.g. “We increased the number of orphans
attending school from 10 to 20”
• Qualitative – relates to quality, but also
subjective opinion, judgements or perceptions
e.g. “The beneficiaries reported to be more
confident in their writing skills“

17. What Impact Data
is worth collecting?
• Get into groups of 3
• Brainstorm data that you could collect to prove:
Group1: A Need, e.g. unemployment
Group 2: changes in SOL, e.g. number of
households with access to clean water
Group 3: changes in QOL, e.g. self-confidence in
public speaking
Group 4: Empowerment, e.g. ability to produce
quality furniture independently

18. What Impact Data
is worth collecting?
Need Average income, unemployment rate, current state of business, child
mortality rate, local poverty line, confidence, independence,
SOL Income, literacy rate, number of books in a library, attendance at
events, access to medicine, electricity, school attendance rate ,
employment, transport,
QOL Self-confidence in own skills, appearance, mental well being, social
inclusion, self-fulfilment, independence,
Empower
ment
Occupational success, ability to provide for family, independence,

19. Collecting Impact Data
Where do I find this data?
• Government statistics, Census Data
• International organisations e.g. UN / WB /
OECD / Transparency International
• Other charity publication
Can’t find it anywhere?
Collect it yourself using Research Methods…

20. What are Methods
of Impact Evaluation?
1. Questionnaires
2. Interviews
3. Focus Groups
4. Conversations and Observations
• Find a research method that suits you!
• Tailor methods to your project and evaluation
• All methods have benefits and weaknesses

21. Method 1:
Questionnaire
Method for qualitative and quantitative data
collection
Useful for e.g.
knowledge testing,
gathering baseline data for needs assessment,
collecting testimonials

22. Method 1:
Questionnaire
Things to consider:
• Language
• Length of the questionnaire
• Using open or closed questions
• Using textboxes or tick boxes
• An absolute no-go: leading questions!
• Scales of rating (e.g. 1-5; strongly agree to
disagree) to give you QUANT data
• Logic of the question order
• Online (surveymonkey) or paper?

23. Method 1:
Questionnaire
Things to do:
• Trial the questionnaire
• Explain purpose of the questionnaire
• Ensure confidentiality
Weaknesses:
• Misunderstanding your question
• Possible misreporting

24. Method 2: Interview
Method for qualitative and quantitative data
collection
Useful for e.g.
individual needs assessments,
Project progress review,
in-depth analysis of impact on an individual

25. Method 2: Interview
Things to consider:
• Language and use of translator (issues of
culture, gender, etc.)
• Choice between structures and semi-structured
• Length of the interview
• Location

26. Method 2: Interview
Things to consider:
• Incentive for interviewee to come
• Open or closed questions
• An absolute no-go: leading questions!
• Logic of the question order
• Video recording?

27. Method 2: Interview
Things to do:
• Practice the interview
• Explain purpose of the interview
• Ensure confidentiality
Weaknesses:
• Possible misreporting
• Bad chemistry with interviewee
• Only get perspective of one individual

28. Method 3:
Focus Group
Method for qualitative data collection mainly
“A focus group is a planned discussion among a small
group (4-12 persons) of stakeholders facilitated by a
skilled moderator.
It is designed to obtain information about people’s
preferences, values, beliefs, perceptions pertaining to a
defined topic and why these are held by observing the
structured discussion of an interactive group.”
Participatory Method Toolkit, Slocum, 2003

29. Method 3:
Focus Group
• Useful for e.g. needs assessment, project design,
market research, end of project impact evaluation
Things to consider:
• Group size (4 -12 is recommended)
• Makeup of the group (gender, caste, social class, etc.)
• Location (indoor, outdoor, private, public?)
• Providing incentives for people to attend (food,
money?)

30. Method 3:
Focus Group
Things to do:
• Use a local facilitator
• Think of ice breakers and warm ups
• Plan a series of questions
• Appoint someone in charge of taking notes
and observing. Record discussion points,
arguments, behaviour.

31. Method 3:
Focus Group
Weaknesses:
• Only reflects opinions of the people present
• Domination of discussion by powerful group
members
• Unpredictable group dynamics
• Facilitator is key!

32. Method 4:
Conversation and Observation
Method for qualitative data collection
Conversational reporting such as:
“I send my daughter to school now.”
“I really like coming here.”
“Used in marketing and the social sciences,
observational research is a social research
technique that involves the direct observation of
phenomena in their natural setting.” Wikipedia

33. Method 4:
Conversation and Observation
Things to consider:
• In what context did someone say something?
• What do you want to observe – a process or
result? Interactions between people; individual
behaviour; physical facts?
Things to do:
• Write down quotes ASAP
• Make an observation checklist
• Separate observation and thought/interpretation!

34. Method 4:
Conversation and Observation
Weaknesses:
• Possible misreporting
• Observers’ bias
• Presence of an observer can influence
behaviour

35. How do you
communicate your impact?
• Consider the different ways of publicising
/ communicating your findings
• Consider your audience!
• Consider mix of Quant and Qual, Outputs
and Outcomes

36. Figures and Stats
Example:
• 100 young people took part in a business
skills course in 2009-2010.
• 60% of homeless individuals now live in
safe housing conditions.

37. Graphs and charts

38. The written word….
Describing what you do
• Do not use jargon
• Take the reader on a journey
(Need, Solution, Impact, Meaning)
• Explain Evaluation Process,
methodology

39. The written word….
Back it up with a Testimonial
"I feel I’m a changed person. I had some
problems at school and I was badly bullied and
threatened. On the Outward Bound course I was
challenged to face my fears and get over them
but when I finished it I felt really proud of
myself.“ Ryan, 13 years

40. Pictures –
Choose them well!
From the Oxfam and
ActionAid website

41. Video
e.g. Windows Movie Maker
• Of the community in which you work
• Of project activities
• Testimonials from beneficiaries
• Animation to explain a complex project
concept

42. Go home and put all
information into one awesome Action Plan:
1. Know what you want to evaluate (what data you
want to collect)
2. Think when you want to record this data
(before, after, mid-project)
3. Decide what method of data collection you
want to use?
4. Who will carry out the evaluation?
5. What resources do you need for it?
6. Do it!
7. Communicate it!

43. Learning Outcome
Statements
1. I can define impact evaluation.
2. I understand why impact evaluation is important
and useful.
3. I know at what points in a project cycle impact
evaluation should take place.
4. I know what kind of things can be measured in
order to do impact evaluation.
5. I can name some research methods used in
impact evaluation.
6. I can name some ways to communicate the
impact of a project.

44. Get in touch!
• Vicky Pelka
• Skype: vickyp101
• Email: pelka.v@gmail.com
• London-based meetings
• Emergency calls: 07 580 430 866
• Sign up/ email me to receive the hand-out