Slides of Edwin Pynegar's presentation in the symposium "Estimating social and environmental impacts of conservation and sustainable development interventions and policies" at the International Congress for Conservation Biology, Kuala Lumpur, 23rd July 2019.

1. Confessions of an RCT: how evaluation
theory met complex realities in one of the
first large-scale Randomized Control Trials
of a conservation intervention
Edwin Pynegar, Nigel Asquith, Emma Wiik,
James Gibbons, Tara Grillos, Tito Vidaurre and
Julia P. G. Jones
International Congress for Conservation Biology
Kuala Lumpur, 23rd July 2019
@EdwinPynegar

2. Impact evaluation in conservation:
Randomized Control Trials (RCTs)
Randomization units
Treatment
group
Control
group
Treatment
group post-RCT
Control group
post-RCT
Comparisons of outcomes of
interest: calculation of
intervention effectiveness
Random allocation
Timescale of RCT
Implementation
of treatment
Esther Duflo (1972-)
James Lind (1716-1794)

3.  Randomized Control Trial
(RCT) of Watershared, a
Payments for Ecosystem
Services-like forest
conservation program in
Latin America.
 129 communities: 65
allocated at random to
treatment group, 64 to
control group.
 Área Natural de Manejo
Integrado Río Grande y
Valles Cruceños: 7340 sq
km protected area in the
Bolivian Andes.

5. Watershared
Landowner NGO/local authority
zero
deforestation
Delivery of
incentives (goods)
at regular
intervals on
compliance with
the agreement
zero cattle
presence (for
higher-level
agreement)
People who ‘produce’ water share in the benefits
People who benefit from water share the benefits
Three outcome variables of interest:
water quality, deforestation rate, environmental values

6. The Watershared RCT
Unbalanced groups at baseline, so more difficult to interpret results… but
no difference in rate of change between treatment and control groups.
Pynegar, E.L., et al. 2018. The effectiveness of Payments for Ecosystem
Services at delivering improvements in water quality: lessons for experiments
at the landscape scale. PeerJ 6: e5753.

7. Effect of site features which predict 2015 E. coli concentration
in riparian area
Actions associated with Watershared – keeping cows away from water
intakes – significantly improved water quality.
Pynegar, E.L., et al. 2018. The effectiveness of Payments for Ecosystem
Services at delivering improvements in water quality: lessons for experiments
at the landscape scale. PeerJ 6: e5753.
The Watershared RCT

8. The Watershared RCT
No difference in deforestation rates between
treatment and control communities…
Wiik, E., et al. 2019. Experimental evaluation of the impact of a payment for
environmental services program on deforestation. Conservation Science and
Practice: e8.
We’d need MORE conservation
areas to see an effect — difficult
when agreements are voluntary?

9. People are more likely to say conservation should be taught to children
Grillos, T., et al. 2019. In-kind conservation payments crowd in environmental
values and increase support for government intervention: A randomized trial in
Bolivia. Ecological Economics. Accepted.
The Watershared RCT

10. Watershared encouraged pro-environmental values…
… but as implemented didn’t achieve changes in
ecological indicators at the community scale.
This doesn’t just teach us about how to design PES-
like programs, it also shows us a lot about how to
design conservation RCTs.

11. Landowners choose whether and where
to enrol land in conservation…
Low, variable uptake: need to trial intervention before RCT?
Wiik, E., et al. 2019. Experimental evaluation of the impact of a payment for
environmental services program on deforestation. Conservation Science and
Practice: e8.

12. …and they often did it in the “wrong”
places for our experiment!
Endogenous behavioural effects within communities are unavoidable.
We also encountered copying / “social spillover” in a control
community — great for the NGO, terrible for the evaluation!
Pynegar, E.L., et al. 2019. What role should Randomised Control Trials play in
providing the evidence base underpinning conservation? Oryx. Accepted.
Water intake, control communityProportion of land in each community put under
conservation

13. Which randomization unit to choose?
• 129 communities
• BUT: watersheds don’t align with community
boundaries → spillover
• Landowners may be members of one community
but own land in neighbouring communities →
spillover
• Mapping community boundaries is DIFFICULT!
• AND: more than 1 household per unit → “cluster”
trial
It’s probably very challenging to run a high-quality RCT evaluating
different kinds of outcome.
Pynegar, E.L., et al. 2019. What role should Randomised Control Trials play
in providing the evidence base underpinning conservation? Oryx. Accepted.

14. RCTs take a long time… is this why there
are so few examples in conservation?
2009 Conceptualization and design
2010 Randomization
2010-11 Baseline data collection
2011 First offers of Watershared
2015-16 Endline data collection
2015-??? Analysis
2017-??? Publication of results
2019 We’re presenting to you 10 years later!
RCT implementers have to be both visionary
and committed for the long haul

15. What did we do with what we learned?
A NEW RCT, IN A DIFFERENT
PART OF BOLIVIA……
Randomization units
Treatment
group A
Control
group
Random
allocation
Treatment
group B
A: Communities can choose one of four
types of project as a conservation incentive
B: Communities can choose whatever they
want (but not cash!)
So: we’re now using RCTs to ask deeper
questions about project design

16. What did we do with what we learned?
……WITH A NEW DESIGN OF THE
WATERSHARED PROGRAM
- Whole-watershed conservation agreements
- With whole communities (but depends on land tenure!)
- More funding for critical areas (watersheds)

17. In conclusion…
• Watershared changed environmental values; did not
improve water quality or slow deforestation at the
community scale, but can at the local scale.
• Effective incentive-based conservation programs:
targeting, action-outcome relationships, appropriate
incentive levels, limitations due to social context?
• RCTs are a powerful tool for impact evaluation which
should probably be used more – but not straightforward!
• Carefully designed RCTs can answer deeper questions
about program design and functioning & contribute to
theory building.

18. Acknowledgements

19. Thank you for listening!
Questions?
edwin.pynegar@gmail.com
@EdwinPynegar

20. Further reading
• Asquith, Nigel. 2016. Watershared: adaptation, mitigation, watershed
protection and economic development in Latin America. Climate &
Development Knowledge Network.
• Grillos, Tara. 2017. Economic vs non-material incentives for participation in an
in-kind payments for ecosystem services program in Bolivia. Ecological
Economics 131: 178-190.
• Bottazzi, Patrick, et al. 2018. Payment for Environmental “Self-Service”:
exploring the links between farmers' motivation and additionality in a
conservation incentive programme in the Bolivian Andes. Ecological Economics
150: 11-23.
• Pynegar, Edwin L., et al. 2018. The effectiveness of Payments for Ecosystem
Services at delivering improvements in water quality: lessons for experiments
at the landscape scale. PeerJ 6 (2018): e5753.
• Wiik, Emma, et al. 2019. Experimental evaluation of the impact of a payment
for environmental services program on deforestation. Conservation Science and
Practice (2019): e8.
• Pynegar, Edwin L., et al. 2019. What role should Randomised Control Trials play
in providing the evidence base underpinning conservation? Oryx, Accepted.
(Preprint: https://peerj.com/preprints/26929/)
• Grillos, Tara, et al. 2019. In-kind conservation payments crowd in
environmental values and increase support for government intervention: A
randomized trial in Bolivia. Ecological Economics. Accepted.