With the support of Jim Rugh, author of ‘Real World Evaluation’, 4th Wheel Social Impact organised a one day workshop on Real World Evaluation to spread this approach among personnel involved in designing, implementing, monitoring, documenting and evaluating social programs in India. The RealWorld Evaluation approach was developed specifically to address the need for practical strategies on how to ensure the highest level of methodological rigour for evaluation studies, consistent with the circumstances under which the evaluation has to be conducted. Sage first published the book in 2006 and the 2nd edition in 2012. (See www.RealWorldEvaluation.org)

1. WORKSHOP ON
REAL WORLD
EVALUATION
16th December, 2017
4TH WHEEL SOCIAL IMPACT

2. 4th Wheel Social Impact was founded with an ambitious vision to strengthen social development
programs. We offer four areas of support to organisations: tracking baseline data, impact
evaluation, using development communication for social change and building capacities of
implementation staff. We regularly organise training programs to strengthen professional
capacities for performance improvements and facilitate discussions to scope problems and
explore solutions, through interactive, participatory and engaging workshops and events.
With the support of Jim Rugh, author of ‘Real World Evaluation’, 4th Wheel Social Impact
organised a one day workshop on Real World Evaluation to spread this approach among
personnel involved in designing, implementing, monitoring, documenting and evaluating social
programs in India.
The RealWorld Evaluation approach was developed specifically to address the need for practical
strategies on how to ensure the highest level of methodological rigour for evaluation studies,
consistent with the circumstances under which the evaluation has to be conducted. Sage first
published the book in 2006 and the 2nd edition in 2012. (See www.RealWorldEvaluation.org)
1

3. 2
An evaluator coming from a research background may find it challenging to cope with
a number of constraints when asked to design and conduct an evaluation of a ‘real-
world’ program. Typical constraints include lack of comparable baseline data, no
data on a relevant comparison group, and insufficient time and budget allocated by
clients. The workshop offered solutions to conduct adequately valid evaluations under
such circumstances, especially while conducting impact evaluations of complex and
evolving programs in complex and evolving contexts.
THE WORKSHOP SUMMARIZED THE APPROACHES ADVOCATED
IN THE REALWORLD EVALUATION BOOK.
SESSIONS INCLUDED
- TECHNICAL INPUTS SESSIONS
- WORKSHEETS AND REFLECTIVE EXERCISES
- KNOWLEDGE AND EXPERIENCE SHARING
- CASE STUDY LEARNING EXERCISE
WORKSHOP OVERVIEW

4. The day began with a game of Human Bingo!
Participants were given a sheet of paper that had different traits, hobbies
and activities. Each participant had to find 5 people who matched one of
those given in the bingo sheet, to win the game. The winner was given a
free ‘document review’ by the 4th wheel team ☺. The game helped
participants to get to know each other on a personal level, share their
own experiences with evaluation and network amongst themselves.
HUMAN BINGO!
caff.co3

5. WE ALWAYS STRUGGLE BETWEEN DECIDING WHETHER
TO DEPLOY RESOURCES FOR BETTER PROGRAM
IMPLEMENTATION, OR TO EVALUATE IT
WE REQUIRED DATA TO UNDERSTAND THE IMPACT OF
OUR PROJECT, BUT THE SECONDARY DATA WAS EITHER
MISSING OR UNRELIABLE
IT IS ALWAYS CHALLENGING TO UNDERTAKE
EVALUATIONS PERTAINING TO CONCEPTS THAT ARE
ABSTRACT SUCH AS PERSONALITY OR
EMPOWERMENT.
– AKSHAT, THE BLUE RIBBON MOVEMENT
- NIKHIL, PAANI FOUNDATION
- BARAT THOMBRE, LIGHT OF LIFE TRUST
caff.co
4
The workshop commenced with participants sharing
instances of challenges they have faced in their
evaluations. Real life examples helped contextualize the
Real World Approach for the attendees and locate it in
their own experiences.

6. Participants were handed out worksheets that
helped them evaluate a program and/or evaluation
they are/ have been a part of. They were taken
through 7 steps of the Real World Approach and
asked to rate their evaluation work on each aspect.
This exercise helped participants understand the
factors they must incorporate in their evaluations, as
well as reflect on their current evaluation practices.
This exercise was substantiated with a technical
input session describing key considerations of the
Real World Approach.
WORKSHEET
EXERCISE
5

7. Participants were taken through different
evaluation designs depending on when the
evaluation events take place (baseline,
midterm, end line), selection of different
evaluation designs (experimental, quasi-
experimental, other) and the qualitative &
quantitative methods used for filling in
“missing data”.
DESIGNING AN
EVALUATION
caff.co
6

8. TACKLING
CONSTRAINTS
caff.co
7
Solutions were shared in terms of addressing budget and time constraints. These
included simplifying the evaluation design, rationalising data needs, finding reliable
secondary sources, reducing sample size and seeking ways to reduce costs of data
collection and analysis.
To cater to data constraints, solutions and examples were discussed with
participants on ways to reconstruct baseline conditions and related issues, the
importance of documentation and maintaining records, regular monitoring and
triangulation, along with using technology for data collection.

9. ELEMENTS OF A
RIGOROUS EVALUATION
caff.co
8
1. Consultation with a variety of stakeholders
2. Articulating a comprehensive logic model that includes relevant external
influences
3. Stakeholder agreement on desirable ‘impact level’ goals
4. Adapting evaluation design according to ground realities
5. Adequate monitoring through the life of the program
6. Appropriate combination of methods to triangulate evidence
7. Flexible to account for evolving contexts
8. Estimating the potential sustainability of changes
9. Communicating the findings in useful ways

10. Some methods discussed were evaluation rubrics,
reality checks, trend analysis, body mapping, ten
seed technique, role-plays and appreciative inquiry.
PARTICIPATORY
TOOLS WERE
SHARED AND
DISCUSSED!
IN OUR ORGANIZATION WE USE COOPERATIVE
INTERVIEWING AS A METHOD TO UNDERSTAND THE
PERSPECTIVE OF ALL STAKEHOLDERS – THE INVESTOR,
THE ENTREPRENEUR AND OUR ENTERPRISE.
WE HAVE USED TEMPERATURE GAUGE WITH
CHILDREN WE WORK WITH. AND ALL METHODS
SHOULD BE CONTEXTUALISED ACCORDING TO THE
TARGET GROUP
- POOJA RAVINDRA, GRAY MATTERS CAPITAL
KRISSTINA RAO, ATMA EDUCATION

11. All the participants had to fill out four questions about
themselves and put the chits in a bowl. Each one had to
pick a chit and find the person on the basis of the answers
on the chit. This activity was combined with lunch during
which the participants got to know each other and their
work engagements better. Three winners were awarded free
passes for our next event!
FIND ME!
caff.co10

12. BUDGET TIME DATA
CASE STUDY
LEARNING EXERCISE
caff.co
11
For the Case Study Learning Exercise, participants were divided into three groups,
which worked on three cases related to budget, data and time constraints.

13. 12
CASE STUDY 1
The primary intervention strategy of the project is the establishment of Farmer Field Schools
(FFS) to provide agricultural skills to farmers in the target communities and to promote shared
learning and knowledge extension between farmers.
• An evaluation of the pilot phase is to be conducted which, if positive, would be an important
factor in deciding whether to expand it.
• A sample survey of farmers’ households would be administered in 2017 and again in 2020 to a
sample of 1,500 project households covering all 5 project locations, and 1,500 comparison group
households.
• The donor agency indicated that the cost exceeded their budget. They asked for a reduction
in budget by 25% to 50%.
The techniques in the Real World Approach to tackle budget constraints are simplification of
evaluation design, rationalizing data needs, reliable secondary resources, reducing sample size
and reducing costs of data collection and analysis. The team decided that they would not
remove the comparison group, as it would diminish the validity of the project and would
consider reducing the sample size to reduce costs.

15. 14
CASE STUDY 2
An international development agency intends to improve the healthcare provider-patient
communication and levels of safety, comfort and privacy experienced by the women patients in
health facilities.
• An evaluation of the pilot phase is to be conducted which, if positive, would be an important
factor in deciding whether to expand it.
• The proposal seeks to adopt a pretest-posttest design with comparative groups to assess the
impact on both the women patients and the healthcare providers.
• The evaluation consultants were asked to resubmit the proposal with the timeline reduced,
bearing in mind the priorities and role of each partner in this project.
When it comes to tackling time constraints in evaluation it is important to bear in mind the
what information is essential, the precision and detail that is required, the necessity of
analyzing all project components and look for avenues to speed up data collection and analysis.
The team suggested that they would collect data during the implementation phase itself. They
will employ the use of technology for the same in order to save time. They would use secondary
data to draw comparisons and save time.

17. 16
CASE STUDY 3
A project activity primarily includes distributing biodegradable pads to women and girls and
providing menstrual hygiene education.
• The evaluation design submitted by the independent consultancy proposes a pretest – post-
test evaluation design.
• However, in the course of implementation and evaluation certain constraints emerged
relating to availability of poor data as well as insufficient data. Same number of women and
girls were not being reached out to, making a baseline and endline comparison difficult,
secondary data on this kind of intervention is unavailable to make valid conclusions of the
impact.
There are methods to deal with this problem – using secondary data, referring to project
records, methods of recall, identification of key informants over the project cycle and using
participatory methods. There are issues in baseline reconstruction that have to be kept in mind
such as variations in reliability and memory distortion. The team would reconstruct the baseline
data through recall and use participatory methods. They would look for data at the local
Primary Health Centre. They will validate the data received through triangulation and reach out
to ASHA workers who work in the area.

19. 18
53 PEOPLE REGISTERED
FOR THE WORKSHOP.
NGO
55%
Corporate
13%
Entrepreneurs
19%
Students
13%
Mastek Foundation
Enable Health Society
Paani Foundation
Light of Life Trust
Atma Education
Gray Matters Capital
American India
Foundation
SynergE
Pupdates
The Blue Ribbon
Movement
Azim Premji University
The Better India
Baker Tilly DHC Pvt. Ltd.
CINI - Tata Trust
Participating Organisations

20. KEY MESSAGES OF THE
DAY
caff.co
19
Evaluation should be about improving rather than proving. Impact frameworks
must be built into program design.
The diversity, heterogeneity and complexity of social issues need frameworks
those are adapted/localized to them in order to evaluate.
Always be aware of unintended outcomes of projects. They may not be part of
the plan, but can sometimes give more insight than what is accounted for.
Evaluation must be linked with program. It must be interwoven into the project
cycle by design.
Shift to technology-based data collection methods instead of pen and paper.
That saves time and improves quality of data!

21. WORKSHOP FEEDBACK
stated that workshops
like these are extremely
relevant
89%
EXERCISES
GROUP WORK
LEARNING
PRESENTATIONS
72%
60%
67%
77%
70%
OVERALL
WORKSHOP
caff.co
20

22. WORKSHOP FEEDBACK
EXERCISES
GROUP WORK
LEARNING
PRESENTATIONS
72%
60%
67%
77%
70%
OVERALL
WORKSHOP
caff.co
20

23. GET IN TOUCH!
hello@the4thwheel.com
(+91) 9825885044
www.the4thwheel.com