The benefits and challenges of using Outcome Harvesting to evaluate a short-term intervention are explored using the example of an 18 month social change project supported by the UK Department for International Development in Tanzania. The project was that was highly ambitious: it sought to influence changes in gender attitudes and behaviour of the general public in Tanzania. Challenges included the lack of outcome indications in project document and the lack of knowledge of outcomes among project personnel. Outcome Harvesting was adapted to allow the harvesting of outcomes using focus groups of those the project sought to influence directly. The concept of ‘proto-outcome’ was used for suggestions of attitude changes that may lead ultimately to behaviour changes. Substantiation of outcomes involved not only third parties but direct observation. The resulting descriptions of outcomes and the evaluation findings proved valuable for learning in the organisation, Search for Common Ground.

1. Searching for outcomes in rural Tanzania:
Harvesting directly from those influenced
Implications for evaluating behavioural change
initiatives with communities
John Mauremootoo, Richard Smith, Dunstan Kishekya
Presentation given at the American Evaluation Association on 12th November 2015 in the
session entitled: The strengths and challenges of Outcome Harvesting for Evaluating in
Complex Situations. Experiences from around the World.

2. The Project
A vehicle for
tackling polarizing
social issues

3. The Project
The Team Tanzania (The Team) one
of 15 completed or ongoing
projects implemented in Africa and
Asia by Search for Common
Ground (SFCG) using The Team
concept – The Team is a vehicle for
tackling polarizing societal issues
by stimulating learning in a
persuasive, but non-
confrontational manner.
For Tanzania the issue chosen was
gender equality
Using an ‘edutainment’ approach
(TV and radio soap operas) the
project planned to reach and
influence a wide audience.

4. Goal To contribute
to strengthening the
implementation and
enforcement of
gender-sensitive
legislation in
Tanzania by
influencing changes
in gender attitudes
and behaviour of the
general public

5. Priority
gender equality
themes targeted
in the 18 month
project

6. 1. Inheritance & women’s consideration
in inheritance issues

7. 2. Women’s leadership

8. 3. Gender-based violence

9. 4. Retention of girls in secondary school

10. 5. Rape

11. The Project Theory of Change
Reach
Response
Resonance
Numbers who watch and listen to the show
The attitude and knowledge changes at
the individual and institutional levels
The actions that have been triggered by
the programmes

12. Evaluation Method
Identification, description and interpretation of outcomes through Outcome
Harvesting (using an OM-inspired definition of Outcome as changes in actions,
relationships, policies or practices of one or more social actors influenced by the
intervention).
With especial emphasis on harvesting outcomes of social actors the project had
been seeking to influence by mobile cinema screenings and follow up discussions.

13. Evaluation challenges
The “Blank Slate of:
• Poor understanding of outcome concept among project implementation team
• Little or no knowledge of outcomes among the project implementation team
• No documentation of outcomes in reports
Short duration of assignment given the fact
that all outcome data had to be collected from
scratch

14. Data collection via focus group discussion

15. Focus Group Discussion
Guiding questions
explained issues such as:
• The definition of
outcomes
• The definition of
contribution
• How to determine how
importance of The
Team’s contribution
• Sources of
substantiation
• Similar outcomes
• Negative outcomes
Data collection via focus group discussion

16. “Treatment” “Control”
Harvesting outcomes through focus
group discussions with social actors
the project had been seeking to
influence by mobile cinema
screenings and follow up discussions
Harvesting outcomes from ‘control
groups’ – target group
representatives who had not
participated in the mobile screenings
and discussions

17. Follow up
Follow-up work to finalise and substantiate results descriptions

18. Results: The Team Contribution
54 outcomes and 10 “proto-outcomes”
0 10 20 30 40 50
Useful (25)
Very important (25)
Important (14)
percentage
Strengthened and / or brought forward
changes that may have happened to
some extent anyway
Either essential to the change or greatly
accelerated it
Strengthened and / or brought forward
changes that may have happened to some
extent anyway
Helped realise a change that may not otherwise have happened or
would have happened very much more slowly

19. Results: Outcomes directly related to
priority issues

20. Inheritance & women’s
consideration in inheritance issues
Results relating to shared ownership of goods
and property

21. Women’s leadership
Women taking more responsibility in work,
home and in political life

22. Gender-based violence
Men stopping beating their wives and others
encouraging men to stop beating their wives

23. Retention of girls in secondary school
Parents prioritising girls education and others
encouraging parents to prioritise girls education

24. Rape
No results provided

25. Results: Other Outcomes
Financial implications of gender
equality
Participatory decision making

26. Implications for evaluating behavioural
change initiatives with communities

27. 1. Outcome Mapping Concepts
OM concepts of outcomes as behavioural change and contribution and the
differences between knowledge, attitude and behaviour were easy and
intuitive to grasp for the participating communities.

28. 2. Knowledge & Attitude Changes
Claimed attitude changes even without behavioural change are important and can be
readily assessed using OH
These “proto-outcomes” helped:
• To understand the claimed process of change even where there have been no
tangible changes in behaviour
• To compare the quality and process of change between “treatment” and “control”
• Inform the change agent

29. 3. Utility of Control Groups
In situations where there is a well-defined theory of change the control
group provides a counter-factual

30. 4. Focus Group Format
The focus group format is effective for discussing less sensitive topics but may not be
the best way of evaluating results concerning sensitive issues and totally inappropriate
for the most sensitive such as rape

31. 5. The value of Follow up
The importance of an iterative follow-up process to enrich the result
description and enhance its credibility.

32. 6. The Timing of the Evaluation
a) It was surprising there were outcomes from a very ambitious project when its 18
months had not yet finished,
b) It was a snap shot & provided credible evidence of change
c) Follow up was needed, e.g. after say 2 years
d) It ideally should really have been a formative evaluation.

33. Now What?
If we consider the
evaluation as an
intervention then both its
findings and the process
need to be built on to help
maximise positive change.
This was not formally done
for a variety of reasons
which in our opinion was a
lost opportunity as is often
the case for evaluations.

34. • Ansila Marandu (evaluation assistant)
• Paul Glick (Search for Common Ground)
• Cornelia Wamba (Search for Common Ground)
• Stella Msami (Search for Common Ground)
• VanessaCorlazzoli (Search for Common Ground)
• The Mvomero Organizations Coalition
• Women and Girls Fight illiteracy and Poverty
Organisation Save the Children Tanzania
• Kate Dyer (AcT and KPMG)
• Layla Ghaid (AcT and KPMG)
Acknowledgements

35. Images
Screenshots from The Team Tanzania on YouTube

36. Images
Stock Images from Shutterstock © All rights reserved

37. Images
Public domain

38. All Other Images
John Mauremootoo (Creative Commons Attribution: CC-BY)