1.
Stefano D’Errico
July 2017
1
Author name
Date
Stefano D’Errico
July 2017
Clearing the fog of
impact claims:
contribution tracing to
assess research
influence
Stefano D’Errico, IIED MEL lead
RC-MEL, Association of Commonwealth Universities,
London, 4 July 2017

2.
Stefano D’Errico
July 2017
2
The worst enemy of an
impact evaluator
https://www.youtube.com/watch?v=6xMaR8au-YU

3.
Stefano D’Errico
July 2017
3
IIED theory of change
Power
holders
Have-nots
Power-less
Knowledge
Changes
in
policies
and
practice
Improving capacities
to engage with and
use evidence
Improving capacities to
produce, engage with
and use evidence
Improving
capacities to
participate
Creating spaces to improve
interaction/liaison and
change power dynamics

4.
Stefano D’Errico
July 2017
4
PCLG theory of change
PCLG Networks are
more aware of the
issues, are better
equipped to influence
their own national
policies
Increased evidence
based research on
poverty and
conservation
Networks are better
coordinated, they have
a stronger voice, and
greater confidence,Networks become an
effective resource for
policy and practice
Increased engagement
of government
conservation agencies
New and revised
policies at national
level reflect new
learning

5.
Stefano D’Errico
July 2017
5
The case: increase of the
communities’ share in Bwindi
Bwindi Impenetrable National Park
In October 2014, The Ugandan
Wildlife Authority approves a
historical policy change to increase
the share of the communities from
US$5 to US$10 for each gorilla-
tracking permit fee. Why has that
happened?

6.
Stefano D’Errico
July 2017
6
Reconstructing the pathway
to change
Our initial contribution claim
IIED research and U-PCLG lobbing has triggered the
decision of the Uganda Wildlife Authority (UWA)

7.
Stefano D’Errico
July 2017
7
Reconstructing the pathway
of change
1. Uganda PCLG and their partners conducted research to understand who
continue to use Bwindi’s resources illegally and why, despite many years of
integrated conservation and development intervention
2. The research outcomes highlight the poverty and feeling of unfairness
were the main reasons of the illegal use of Bwindi’s resources
3. U-PCLG together with UWA and other partners decides to advocate for an
increase in the community share of the gorilla permit fee
4. U-PCLG’s member, who at the time was also on the UWA board champions
this change in a policy of the UWA board meeting
5. UWA board agrees to increase the Gorilla permit fee going to the local
community from $5 to $10
6. Outcome: The UWA board eventually took the decision suggested by U-
PCLG
U-PCLG contribution claim at the beginning of the
investigation

8.
Stefano D’Errico
July 2017
8
Reconstructing the pathway
to change
1. The communities around the BIN Park have, for a long time, been dissatisfied with the
amount of revenue shared with them. This dissatisfaction increased with a rise in the
gorilla permit fee by US$100 (from 0.5 to 0.91)
2. The UWA board was already considering a change in the community share of the gorilla
permit fee (from 0.5 to 0.8)
3. U-PCLG, in collaboration with others, had undertaken research on the causes of illegal
activity taking place in the park. Such research has generated new and original
insights that motivated the decision (from 0.5 to 0.69), was tailored to support
advocacy work (from 0.5 to 0.77), and was conducted in a collaborative way to build
trust (from 0.5 to 0.63).
4. U-PCLG submitted a formal request for the specific change in the community share to
the UWA board, which acknowledged receipt and initiated a formal response process
(from 0.5 to 1)
5. A U-PCLG member championed the change within UWA’s formal response process
through her role as a UWA Committee Chairperson (from 0.5 to 0.98)
6. Outcome: The UWA board eventually took the decision suggested by U-PCLG
What is more likely to have happened:
U-PCLG work accelerated the process of change

9.
Stefano D’Errico
July 2017
9
Understanding systemic
impact by
investigating
the pathway to change
how we have reconstructed the pathways to
change by engaging in a dialogue between
theory and evidence

10.
Stefano D’Errico
July 2017
10
Investigating the pathway to
change
What if UWA had
their own internal
processes? This
would exclude our
explanation!
Should we look for
complementary
explanations that
prepared the
ground?
Step 1. Alternative explanations!!!

11.
Stefano D’Errico
July 2017
11
Investigating the pathway to
change
Step 2. Building a timeline
2010
2011: Darwin
approves the Research
to Policy (R2P) Project,
‘building capacity for
conservation through
poverty alleviation’.
2012: The R2P project
begins with IIED, ICL and
IFTC supported by U-
PCLG undertaking
research to understand
who continues to use
Bwindi’s resources
illegally and why.
2010: UWA adopt new
national revenue
sharing guidelines.
2013: Arcus Foundation
approves a further 2 year
grant including continued
support for U-PCLG.
2015
2013 (September): U-PCLG
host a final research
workshop with members and
UWA. The option of
increasing the community
share of the gorilla permit fee
is discussed.
2014 (March): U-PCLG
Chairman, Panta Kasoma, sends
a letter to UWA to request an
increase in the community share
of the Mountain gorilla permit
fee from US $5 to US $ 10.
2014 (September): U-PCLG
member, Nature Uganda host an
event to launch the research
September 2014: U-PCLG
member Gladys Kalema
Zikusoka, advocates for a
change in the community share
as Chair of an UWA Committe
2014 (October): UWA Board
approve a change in the
community share
2015 (July): The change in
the community share of
the gorilla permit fee from
US $5 to US $ 10 takes
effect at the start of
Uganda’s new FY
2015 (February): ACODE
publish a policy brief
explaining why increasing
the community share of
the gorilla permit fee is
vital for UWA to address
local perceptions of
inequality.
2013: UWA Review of
Bwindi’s General
Management Plan. Key
provisions include
implementing the revenue
sharing programme.
2014/2015: UWA begin
consultations to revise the
national revenue sharing
guidelines.
2011: Arcus Foundation
provide a 2 year grant
to support the
establishment of U-
PCLG.
2012: UWA publish
revised national revenue
sharing guidelines.

12.
Stefano D’Errico
July 2017
12
Investigating the pathway to
change
Step 3. Imagining evidence
Minutes of
U-PCLG meetings
and workshops
confirming steps of
the change
Correspondence
between UWA
members and
U-PCLG confirming
existence of lobbying
activities
Logical timing
between IIED
research,
U-PCLG lobbying
and the outcome
U-PCLG member
acknowledging
the importance of
other causes
Documents that
confirm the
existence and
relevance of other
causes
Logical timing
between internal
UWA processes
and the outcome
Minutes of the UWA
board
acknowledging the
research
UWA board
members
acknowledging the
importance of the
research

13.
Stefano D’Errico
July 2017
13
Assessing the strength
of evidence
Step 4. Estimating sensitivity and type I error
for each step of the process

14.
Stefano D’Errico
July 2017
14
Assessing the strength
of evidence
Quotes from Panta Kasumu – U-PCLG chair:
“When they came up with the gorilla levy that was five
dollars, that was good enough at the time, but then the permit
was going up in value and people started agitating for an
increase”.
Step 4. Estimating sensitivity and type I error,
in practice
How likely was
he to say so if
community
pressure was
not there?
“Personally I think when they [UWA] saw this proposal
coming from the ten dollars, and this is just my perspective,
they might have thought well let’s just go for this ten dollars
because these guys will come up with some other demand.”
What were his
motivations and
intents to say
so? This clearly
underplay the
role of U-PCLG
activities

15.
Stefano D’Errico
July 2017
15
Updating confidence
thanks to Bayes!
Bayesian inference is a method
of statistical inference in which Bayes'
theorem is used to update the probability
for a hypothesis as
more evidence or information becomes
available.
Step 5. Bayesian confidence update, estimating
the posteriors

16.
Stefano D’Errico
July 2017
16
Updating confidence
thanks to Bayes!
Step 5. Bayesian confidence updating, estimating
the posteriors
Estimating the posterior
P(T|E) means assessing
the probability of theory
given the observed
evidence

17.
Stefano D’Errico
July 2017
17
But what do the scores
mean?
Practical Certainty 0.99+
Reasonable Certainty 0.95 – 0.99
High Confidence 0.85 – 0.95
Cautious Confidence 0.70 – 0.85
More Confident than not 0.50 – 0.70
No information 0.50
Table 3 – Qualitative rubrics for different quantitative levels of confidence
Source: Befani, B. & G. Stedman-Bryce (2016) "Using Process Tracing and
Bayesian updating to assess confidence in contribution claims in impact
evaluation: introducing “Contribution Tracing” and “Contribution Trials”"
(Working Paper)
Step 5. Bayesian confidence updating:
Posterior rubrics

18.
Stefano D’Errico
July 2017
18
Overall confidence in
contribution claims
1. U-PCLG accelerated the change: 0.75
2. U-PCLG shaped the change: 0.73
Estimating overall confidence, so what should be
our confidence in theory?
The overall confidence value of each contribution claim should be
equal to the weakest step of the causal mechanism. This
forced us to investigate more thoroughly the steps with less evidence; at
the same time it also made us think about the weakest parts of our
theory, and if we didn’t find any evidence after exhaustive searches, we
had to refine the contribution claim. That’s when the learning
happens!

19.
Stefano D’Errico
July 2017
19
Overall confidence
0.75 and 0.73, they
don’t look impressive
though, do they?
Shouldn’t it be at
least 0.95?!
Well this is real-world, real-life man!
you rarely find the evidence you
dream of! Though if we treat these
values as relative scores rather than
as absolute grades, we can say that:
U-PCLG theory is more supported
by evidence than the alternative
explanations!

20.
Stefano D’Errico
July 2017
20
PT and BA pros and cons
Strengths Challenges
Handles very well confirmation bias because
it openly addresses it in both data collection
and analysis. Very high level of transparency
Differentiating between absence of evidence
or evidence of absence can be very tricky,
especially when dealing with policy makers
Iterative process. It enables a dialogue
between theory and evidence which makes
learning happen! It’s revelatory about why
change happens.
Iterative process means it’s difficult to
establish time needed for the investigation!
Also more alternative explanations means
more lines of inquiry, hence more time!
It’s intuitive, the investigation can be
conducted by a non expert with minimal
technical input. Although coaching and
mentoring is needed
It can take time to get used to the ‘revelatory
moments’ of the investigation. Must be
prepared to change the data collection
strategy on the basis of new evidence
Can be used ex post and even on projects
which don’t have M&E data or a theory of
change
Access to official records, memoire, meeting
minutes or correspondence can be very
hard! Likewise policy maker may decide of
not engaging with the evaluation as a
strategy. In that case it’s very difficult to
prove it though!