By A/Prof Juliet Willetts, Bruce Bailey, Dr Paul Crawford for the the Monitoring Sustainable WASH Service Delivery Symposium, 9 - 11 April 2013, Addis Ababa, Ethiopia

1. THINK
CHANGE
DO
REFLECTIONS ON MONITORING A LARGE-
SCALE CIVIL SOCIETY WASH INITIATIVE
Lessons for sector monitoring and potential
contributions from NGOs
Tim Brennan and John McKibbin
A/Prof Juliet Willetts, Bruce Bailey, Dr Paul Crawford
IRC Symposium 2013 Addis Ababa
Griffin nrm

2. Key messages
• Three lessons for sector monitoring from donor
monitoring:
1. Investing in a ‘learning focus’ as part of sector or donor
monitoring is critical
2. Tensions between specificity and flexibility in choosing
indicators and frameworks must be addressed carefully
3. An actor-centred ‘theory of change’ provided a helpful,
readily understandable structure
• A wide range of strategies are available to
NGOs to support sector monitoring (but so far
are little adopted)

3. A large-scale
civil society
WASH fund…
$32.5 million
11 CSOs
21 countries
782,000 people with toilets
45 activities
$200K- >$1.6m 564,000 people with water

4. Our role in the Fund… diverse activities
of the monitoring review panel (MRP)
…
• Engaged for life of the
program
• Designed the performance
framework and reporting
requirements
• Conducted analysis and
synthesis – learning papers
• Undertook monitoring of
Fund activities
• Provided an M&E capacity
building role
• Undertook two independent
evaluations of the Fund

5. Lesson 1 from donor monitoring: Invest
in a learning focus
• ‘Developmental evaluation’ Developmental
approach – what did we do?: evaluation adapts
evaluation (or
• Collected ‘real-time data’ to inform on-
monitoring) to
going decision making and adaptations “emergent and
• Included descriptive free-form case dynamic realities
studies as well as indicators in complex
environments”
• Used evaluation questions (not just
(Patton, 2011,
indicators) p1).
• Facilitated, structured learning ‘events’
• On-going one-on-one feedback to NGOs
• Learning ‘products’ that synthesise
lessons across contexts

6. Lesson 2 from donor monitoring:
Balance specificity and flexibility
• Why is specificity needed in sector
monitoring?
• Why is flexibility needed in sector
monitoring?
• How can these needs be balanced?

7. Flexible
guidance on
what to
monitor or
report
19 ‘result
areas’ 7 mandatory
indicators
We achieved balance by defining some indicators very tightly,
and making space in other ways for more diverse, flexible inputs

8. Lesson 3 from donor monitoring: Actor-
centred theory of change was helpful
• Monitoring is usually about measuring
‘change’
• ‘Theories of change’ simplify reality to
make it easier to measure, communicate
and understand how we think change
happens
• We can then also test our assumptions
about how change happens

9. Theory of change behind
the performance 
Poor communities
arrangements & households

Institutional &
community partners
CORE OUTCOMES:
Outcome 1:
increased access to
 ENABLING sanitation
CSO
OUTCOMES: Outcome 2:
Delivery Team
Outcome 4: improved increased access to
WASH governance water
Outcome 5: Outcome 3:
DELIVERABLES: strengthened local improved hygiene
capacity behaviour
E.g. training,
mentoring, advocacy, Outcome 6: improved
technical advice, gender equality
campaigns etc. Outcome 7: improved
WASH evidence base

10. NGO roles to support sector
monitoring: a ‘strategy map’
Strategy Causal Persuasive Supportive
I-1 I-2 I-3
Focused
on a
particular
individual
or group
E-1 E-2 E-3
Focused
on the
enabling
environ-
ment
Strategy map was developed by IDRC as a

11. Strategy Causal Persuasive Supportive
I-1 I-2 I-3
Focused Direct role in Providing awareness Providing frequent,
on a
particular
monitoring own raising, education or sustained, on-going
individual direct specific training to mentoring and support;
or group implementation community members or multipurpose
activities. or other partners. capacity building; or
developing support
structures, committees
& networks.
E-1 E-2 E-3
Focused Engaging in Dissemination of Building partnerships,
on the
enabling
policy dialogue, information to broad providing collective
environ- directly causing audience; support and promoting
ment changes in persuasive networking and
incentives, rules; environment for a coordination; also
playing an specific behaviour or supporting higher levels
advocacy role. attitude; conducting of government or local
workshops/conf’s. research networks.

12. Examples of NGO roles and their
benefit to sector monitoring
• Direct monitoring role (I-1
strategy map)
“one NGO in Bangladesh showed
that government (Department of
Public Health Engineering) data
for sanitation coverage of 38-58%
in three ‘upazilas’ in 2009 was
actually at 5-28% when the NGO
conducted their baseline study in
2010, with these large differences
gaining attention from relevant
government staff”

13. Examples of NGO roles and their
benefit to sector monitoring
• Systematic
strategies to build
local government
expertise in
monitoring (I-3)
“one NGO working in
Indonesia and
Bangladesh directly
supported existing
government
monitoring systems
for sanitation by
working in a direct
support role for local
government”

14. Examples of NGO roles and their
benefit to sector monitoring
• Document and share own learning/innovations
and promote their uptake(E-2)
“An NGO working in Bhutan achieved adoption of elements of
their project monitoring framework which used a qualitative
information scale for examining the hygienic status of
toilets, and also includes a scale for examining handwashing
with soap.
Their ideas were taken up and integrated into government
monitoring systems, including Ministry of Health’s Annual
Survey in February/March each year, and inclusion of
indicators to sanitary access and hygienic toilet usage in the
next Five Year Development Plan from mid-2013 as Key
sector Result Areas” (Halcrow, pers comm. 2013)”

15. So what? What needs to change for
NGOs to better support sector
monitoring?
Government staff should:
 • Consider and communicate the range of roles which
NGOs might play in support of national initiatives
Program designers (both NGOs and
donors) should:
 • Make efforts to adopt strategies explored in this
presentation, particularly considering strategies not yet
in use
M&E specialists:
 • Should align monitoring and evaluation systems of
WASH projects to sector monitoring, instigate the
sharing of information with government

16. Key messages
• Three lessons for sector monitoring from donor
monitoring:
1. Investing in a ‘learning focus’ as part of sector or donor
monitoring is critical
2. Tensions between specificity and flexibility in choosing
indicators and frameworks must be addressed carefully
3. An actor-centred ‘theory of change’ provided a helpful,
readily understandable structure
• A wide range of strategies are available to
NGOs to support sector monitoring (but so far
are little adopted)

17. Thank you
Contacts: Juliet.Willetts@uts.edu.au, pcrawford@aid-it.com,
bbailey@griffin-nrm.com.au
Griffin nrm