Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Clearing the fog of impact claims: contribution tracing to assess research influence

211 views

Published on

This presentation on how process tracing with Baysian updating can be used to evaluate how research contributed to increasing the benefits available to local communities at Bwindi Impenetrable National Park in Uganda, was given by Stefano D’Errico, monitoring, evaluation and learning lead at the International Institute for Environment and Development (IIED).

D'Ericco made the presentation at a Association of Commonwealth Universities event in London on 4 July 2017.

The research in Uganda shows that methods of process tracing and Bayesian updating facilitate a dialogue between theory and evidence that allows us to assess our degree of confidence in 'contribution claims' in a transparent and replicable way.

More details: http://pubs.iied.org/17359IIED/

https://www.iied.org/taking-more-rigorous-approach-evaluation

Published in: Environment
  • Be the first to comment

  • Be the first to like this

Clearing the fog of impact claims: contribution tracing to assess research influence

  1. 1. Stefano D’Errico July 2017 1 Author name Date Stefano D’Errico July 2017 Clearing the fog of impact claims: contribution tracing to assess research influence Stefano D’Errico, IIED MEL lead RC-MEL, Association of Commonwealth Universities, London, 4 July 2017
  2. 2. Stefano D’Errico July 2017 2 The worst enemy of an impact evaluator https://www.youtube.com/watch?v=6xMaR8au-YU
  3. 3. Stefano D’Errico July 2017 3 IIED theory of change Power holders Have-nots Power-less Knowledge Changes in policies and practice Improving capacities to engage with and use evidence Improving capacities to produce, engage with and use evidence Improving capacities to participate Creating spaces to improve interaction/liaison and change power dynamics
  4. 4. Stefano D’Errico July 2017 4 PCLG theory of change PCLG Networks are more aware of the issues, are better equipped to influence their own national policies Increased evidence based research on poverty and conservation Networks are better coordinated, they have a stronger voice, and greater confidence,Networks become an effective resource for policy and practice Increased engagement of government conservation agencies New and revised policies at national level reflect new learning
  5. 5. Stefano D’Errico July 2017 5 The case: increase of the communities’ share in Bwindi Bwindi Impenetrable National Park In October 2014, The Ugandan Wildlife Authority approves a historical policy change to increase the share of the communities from US$5 to US$10 for each gorilla- tracking permit fee. Why has that happened?
  6. 6. Stefano D’Errico July 2017 6 Reconstructing the pathway to change Our initial contribution claim IIED research and U-PCLG lobbing has triggered the decision of the Uganda Wildlife Authority (UWA)
  7. 7. Stefano D’Errico July 2017 7 Reconstructing the pathway of change 1. Uganda PCLG and their partners conducted research to understand who continue to use Bwindi’s resources illegally and why, despite many years of integrated conservation and development intervention 2. The research outcomes highlight the poverty and feeling of unfairness were the main reasons of the illegal use of Bwindi’s resources 3. U-PCLG together with UWA and other partners decides to advocate for an increase in the community share of the gorilla permit fee 4. U-PCLG’s member, who at the time was also on the UWA board champions this change in a policy of the UWA board meeting 5. UWA board agrees to increase the Gorilla permit fee going to the local community from $5 to $10 6. Outcome: The UWA board eventually took the decision suggested by U- PCLG U-PCLG contribution claim at the beginning of the investigation
  8. 8. Stefano D’Errico July 2017 8 Reconstructing the pathway to change 1. The communities around the BIN Park have, for a long time, been dissatisfied with the amount of revenue shared with them. This dissatisfaction increased with a rise in the gorilla permit fee by US$100 (from 0.5 to 0.91) 2. The UWA board was already considering a change in the community share of the gorilla permit fee (from 0.5 to 0.8) 3. U-PCLG, in collaboration with others, had undertaken research on the causes of illegal activity taking place in the park. Such research has generated new and original insights that motivated the decision (from 0.5 to 0.69), was tailored to support advocacy work (from 0.5 to 0.77), and was conducted in a collaborative way to build trust (from 0.5 to 0.63). 4. U-PCLG submitted a formal request for the specific change in the community share to the UWA board, which acknowledged receipt and initiated a formal response process (from 0.5 to 1) 5. A U-PCLG member championed the change within UWA’s formal response process through her role as a UWA Committee Chairperson (from 0.5 to 0.98) 6. Outcome: The UWA board eventually took the decision suggested by U-PCLG What is more likely to have happened: U-PCLG work accelerated the process of change
  9. 9. Stefano D’Errico July 2017 9 Understanding systemic impact by investigating the pathway to change how we have reconstructed the pathways to change by engaging in a dialogue between theory and evidence
  10. 10. Stefano D’Errico July 2017 10 Investigating the pathway to change What if UWA had their own internal processes? This would exclude our explanation! Should we look for complementary explanations that prepared the ground? Step 1. Alternative explanations!!!
  11. 11. Stefano D’Errico July 2017 11 Investigating the pathway to change Step 2. Building a timeline 2010 2011: Darwin approves the Research to Policy (R2P) Project, ‘building capacity for conservation through poverty alleviation’. 2012: The R2P project begins with IIED, ICL and IFTC supported by U- PCLG undertaking research to understand who continues to use Bwindi’s resources illegally and why. 2010: UWA adopt new national revenue sharing guidelines. 2013: Arcus Foundation approves a further 2 year grant including continued support for U-PCLG. 2015 2013 (September): U-PCLG host a final research workshop with members and UWA. The option of increasing the community share of the gorilla permit fee is discussed. 2014 (March): U-PCLG Chairman, Panta Kasoma, sends a letter to UWA to request an increase in the community share of the Mountain gorilla permit fee from US $5 to US $ 10. 2014 (September): U-PCLG member, Nature Uganda host an event to launch the research September 2014: U-PCLG member Gladys Kalema Zikusoka, advocates for a change in the community share as Chair of an UWA Committe 2014 (October): UWA Board approve a change in the community share 2015 (July): The change in the community share of the gorilla permit fee from US $5 to US $ 10 takes effect at the start of Uganda’s new FY 2015 (February): ACODE publish a policy brief explaining why increasing the community share of the gorilla permit fee is vital for UWA to address local perceptions of inequality. 2013: UWA Review of Bwindi’s General Management Plan. Key provisions include implementing the revenue sharing programme. 2014/2015: UWA begin consultations to revise the national revenue sharing guidelines. 2011: Arcus Foundation provide a 2 year grant to support the establishment of U- PCLG. 2012: UWA publish revised national revenue sharing guidelines.
  12. 12. Stefano D’Errico July 2017 12 Investigating the pathway to change Step 3. Imagining evidence Minutes of U-PCLG meetings and workshops confirming steps of the change Correspondence between UWA members and U-PCLG confirming existence of lobbying activities Logical timing between IIED research, U-PCLG lobbying and the outcome U-PCLG member acknowledging the importance of other causes Documents that confirm the existence and relevance of other causes Logical timing between internal UWA processes and the outcome Minutes of the UWA board acknowledging the research UWA board members acknowledging the importance of the research
  13. 13. Stefano D’Errico July 2017 13 Assessing the strength of evidence Step 4. Estimating sensitivity and type I error for each step of the process
  14. 14. Stefano D’Errico July 2017 14 Assessing the strength of evidence Quotes from Panta Kasumu – U-PCLG chair: “When they came up with the gorilla levy that was five dollars, that was good enough at the time, but then the permit was going up in value and people started agitating for an increase”. Step 4. Estimating sensitivity and type I error, in practice How likely was he to say so if community pressure was not there? “Personally I think when they [UWA] saw this proposal coming from the ten dollars, and this is just my perspective, they might have thought well let’s just go for this ten dollars because these guys will come up with some other demand.” What were his motivations and intents to say so? This clearly underplay the role of U-PCLG activities
  15. 15. Stefano D’Errico July 2017 15 Updating confidence thanks to Bayes! Bayesian inference is a method of statistical inference in which Bayes' theorem is used to update the probability for a hypothesis as more evidence or information becomes available. Step 5. Bayesian confidence update, estimating the posteriors
  16. 16. Stefano D’Errico July 2017 16 Updating confidence thanks to Bayes! Step 5. Bayesian confidence updating, estimating the posteriors Estimating the posterior P(T|E) means assessing the probability of theory given the observed evidence
  17. 17. Stefano D’Errico July 2017 17 But what do the scores mean? Practical Certainty 0.99+ Reasonable Certainty 0.95 – 0.99 High Confidence 0.85 – 0.95 Cautious Confidence 0.70 – 0.85 More Confident than not 0.50 – 0.70 No information 0.50 Table 3 – Qualitative rubrics for different quantitative levels of confidence Source: Befani, B. & G. Stedman-Bryce (2016) "Using Process Tracing and Bayesian updating to assess confidence in contribution claims in impact evaluation: introducing “Contribution Tracing” and “Contribution Trials”" (Working Paper) Step 5. Bayesian confidence updating: Posterior rubrics
  18. 18. Stefano D’Errico July 2017 18 Overall confidence in contribution claims 1. U-PCLG accelerated the change: 0.75 2. U-PCLG shaped the change: 0.73 Estimating overall confidence, so what should be our confidence in theory? The overall confidence value of each contribution claim should be equal to the weakest step of the causal mechanism. This forced us to investigate more thoroughly the steps with less evidence; at the same time it also made us think about the weakest parts of our theory, and if we didn’t find any evidence after exhaustive searches, we had to refine the contribution claim. That’s when the learning happens!
  19. 19. Stefano D’Errico July 2017 19 Overall confidence 0.75 and 0.73, they don’t look impressive though, do they? Shouldn’t it be at least 0.95?! Well this is real-world, real-life man! you rarely find the evidence you dream of! Though if we treat these values as relative scores rather than as absolute grades, we can say that: U-PCLG theory is more supported by evidence than the alternative explanations!
  20. 20. Stefano D’Errico July 2017 20 PT and BA pros and cons Strengths Challenges Handles very well confirmation bias because it openly addresses it in both data collection and analysis. Very high level of transparency Differentiating between absence of evidence or evidence of absence can be very tricky, especially when dealing with policy makers Iterative process. It enables a dialogue between theory and evidence which makes learning happen! It’s revelatory about why change happens. Iterative process means it’s difficult to establish time needed for the investigation! Also more alternative explanations means more lines of inquiry, hence more time! It’s intuitive, the investigation can be conducted by a non expert with minimal technical input. Although coaching and mentoring is needed It can take time to get used to the ‘revelatory moments’ of the investigation. Must be prepared to change the data collection strategy on the basis of new evidence Can be used ex post and even on projects which don’t have M&E data or a theory of change Access to official records, memoire, meeting minutes or correspondence can be very hard! Likewise policy maker may decide of not engaging with the evaluation as a strategy. In that case it’s very difficult to prove it though!

×