Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Ukes process tracing presentation

2,976 views

Published on

Itad's Melanie Punton's presentation on Process Tracing from the UKES Conference, May 2015.

Published in: Data & Analytics
  • Be the first to comment

  • Be the first to like this

Ukes process tracing presentation

  1. 1. Punton & Welle (2015) Straws-in-the- wind, Hoops and Smoking Guns: What can Process Tracing Offer to Impact Evaluation? http://www.ids.ac.uk/publication/straws-in-the-wind-hoops-and- smoking-guns-what-can-process-tracing-offer-to-impact- evaluation Stedman-Bryce (2013) Health for All: Towards Free Universal Health Care in Ghana http://policy- practice.oxfam.org.uk/publications/effectiveness- review-health-for-all-towards-free-universal-health- care-in-ghana-306376 2
  2. 2. Process Tracing: A qualitative method for assessing causal inference within a single case Sits in the ‘evaluator’s toolbox’ alongside • Theory-based evaluation • Contribution analysis • Realist evaluation 3
  3. 3. Why process tracing? 0 Useful where pathways of change are uncertain 0Tells you WHY change happened 0 Rigorous approach through ex-post design without a control group 0Time intensive 0 Requires the outcome to be known 0 Uncharted territory 4
  4. 4. A brief history of process- tracing 0 A method stemming from the social and political sciences. Major contributors: Alexander George, Andrew Bennett, David Collier 0 Originally used to explain historical events, e.g. the cause of the Cuban Missile Crisis (Allison and Zelikow 1999) 0 A ‘contested method’ 0 Beach and Pederson’s book (2013): detailed theory and step-by-step guidance 0 Limited examples of its application in evaluation NB: See Punton & Welle (2015) for references 5
  5. 5. Counterfactual framework Generative framework Experimental (RCTs) Quasi-experimental Process tracing Realist evaluation Intervention A Outcome B???? Opening up the black box 6
  6. 6. Causal mechanisms Intervention A Outcome B???? Causal mechanism Generative causal framework 7
  7. 7. Causal mechanisms Part3Part2Part1 Intervention A Entity 1 Entity 2 Entity 3 Activity 1 Activity 2 Activity 3 Outcome B 8
  8. 8. The Universal Health Care Campaign in Ghana 0 Collaborative advocacy effort by civil society to promote universal free access to health care 0 Core funding from Oxfam until 2013 0 Lobbying, mobilising, media and research 0 Effectiveness review in 2012-13, using elements of process tracing Stedman-Bryce (2013) Effectiveness Review: Health for All: Towards Free Universal Health Care in Ghana http://policy- practice.oxfam.org.uk/publications/effectiveness-review- health-for-all-towards-free-universal-health-care-in- ghana-306376 9
  9. 9. A (simple) example of a process tracing mechanism Universal Health Care campaign Free universal health care Civil society Conduct coordinated advocacy activities for free universal health care Part 1 Public Part 2 Become aware of the limitations of current health care financing Public Part 3 Demand free universal health care from government actors Govern- ment actors Part 4 Increasingly support free universal health care, based on desire for public support Govern- ment Part 5 Amend policies and processes to move towards free universal health care Activity Entity 10
  10. 10. Differences between a mechanism and a theory of change… 0 No leaps in logic between the parts 0 Every part should be necessary 0 Every part should be observable and empirically measurable 11
  11. 11. Collecting evidence • Examples of coordinated advocacy linked to campaign (e.g. events, platforms, meetings) • Interview data suggesting that the campaign increased knowledge about and interest in healthcare reform among civil society actors • Number of advocacy events held before the campaign started, compared to number held during campaign Hypothesis: civil society conducts coordinated advocacy for free healthcare as a result of the Campaign 12
  12. 12. Collecting evidence • Examples of coordinated advocacy taking place before the campaigns • Interviews suggesting that the campaign made little difference to activities Alternative hypothesis: The advocacy campaign made little difference to the coordination or activities of CSOs 13
  13. 13. “We balance probabilities and choose the most likely. It is the scientific use of the imagination.” – The Hound of the Baskervilles 14
  14. 14. Assessing evidence: four tests • Analogous to a criminal trial • The evidence for each hypothesis is weighed…. • …in order to increase the researcher’s confidence in the hypothesis. • Bayesian probability logic followed to assess the strength of each part of the causal chain. • Evidence examined using concepts of necessity and sufficiency. 15
  15. 15. Assessing evidence: four tests Straw in the wind test Evidence: hotel receipts, suggestive text messages - Neither necessary nor sufficient Hypothesis: John shot Mary Hoop test Evidence: John lacks an alibi - Necessary, but not sufficient Smoking gun test Evidence: John was found holding a gun over Mary’s body - Sufficient, but not necessary Double decisive test Evidence: A tamper- proof CCTV camera recorded the murder - Necessary and sufficient 16
  16. 16. Assessing evidence: four tests Certainty of evidence Uniquenessofevidence Low (evidence is insufficient for h ) High (evidence is sufficient for h ) High (evidenceis necessaryforh) Low (evidenceisnot necessaryforh) Hoop tests Double- decisive Straw inthe wind tests Smoking gun tests 17
  17. 17. Assessing evidence: four tests Straw in the wind test Hypothesis: The Campaign significantly increased the capacity of member CSOs to plan and work together on the issue of free health care Evidence: 1. Campaign members were given the opportunity to discuss and debate a report on the complex National Health Insurance Scheme, in order to increase their understanding on the topic 2. Before the report was published, there was little coordinated advocacy for free universal healthcare Increase the plausibility of the hypothesis but do not prove it or disprove alternative hypotheses. BUT together they provide stronger evidence than they do alone 18
  18. 18. Assessing evidence: four tests Hoop test Hypothesis: The Campaign significantly increased the capacity of member CSOs to plan and work together on the issue of free health care Evidence: Until the campaign started, there was little notable coordinated advocacy on this issue. Not enough on its own to prove the hypothesis, BUT evidence must pass the test in order to keep the hypothesis under consideration 19
  19. 19. Smoking gun test Hypothesis: The National Health Insurance Authority in Ghana revised methodology for calculating membership because of pressure created by the campaign Evidence: Ghana delegation at international meeting stated that the campaign’s report ‘prompted us to revise our figures’ (despite initially publically dismissing the report) Evidence is sufficient to give high confidence that the hypothesis is true. BUT this type of evidence is hard to come by. Assessing evidence: four tests 20
  20. 20. The end goal… 0 Researcher can express level of confidence in each part of the mechanism 0 Mechanism is only as strong as its weakest part 0 If the researcher is confident in the whole mechanism… 0 …the researcher can express confidence that the mechanism holds in that particular case: that A caused B, and why. 21
  21. 21. Why process tracing? 0 Useful where pathways of change are uncertain 0 Rigorous approach through ex-post design without a control group 0Tells you WHY change happened 0Useful in a mixed methods design, or in combination with e.g. Contribution Analysis? 0 Time intensive 0 Requires the outcome to be known 0Uncharted territory 0 Risk of one weak link causing the whole mechanism to be rejected 22
  22. 22. Questions? melanie.punton@itad.com Punton & Welle (2015) Straws-in-the- wind, Hoops and Smoking Guns: What can Process Tracing Offer to Impact Evaluation? http://www.ids.ac.uk/publication/straws-in-the-wind-hoops-and- smoking-guns-what-can-process-tracing-offer-to-impact- evaluation 23

×