Lessons learned in using process tracing for evaluation

Lessons learned in using process tracing
for evaluation
Emily A. Bobrow, PhD, MPH
Data for Impact Webinar
17 October 2019

• Generate strong evidence for program and
policy decision making
• Build individual and organizational capacity
• Enhance the use of data for global health
programs and policies
D4I works to:

• Background on process tracing as an innovative
qualitative method for evaluations
• Two evaluations as examples:
1. Process tracing of causal mechanisms implemented in
the Partnership for HIV-Free Survival (PHFS) in
Uganda
2. Learning agenda study of health information system
(HIS) strengthening in Madagascar
• Conclusions
What this presentation will cover

• The work performed for the two case studies of
process tracing in Uganda and Madagascar was
performed under the MEASURE Evaluation
project, Phase IV.
• MEASURE Evaluation is funded by the United
States Agency for International Development
(USAID).
A word about the work

• Process tracing is a qualitative, case-based
approach used to describe a linear causal chain
with steps from a conceptual model or theory of
change*
• This qualitative method can be used to answer
whether, why, and how an intervention causes a
health outcome
• Process tracing is not common in public health
evaluation
Process tracing
*Better Evaluation. (2016, April 28). Process tracing. Retrieved from
http://betterevaluation.org/evaluation-options/processtracing

Process tracing method
2. Developing testable
hypotheses
3. Identifying evidence
required to test the
hypotheses
1. Developing theory
(causal mechanism)
4. Collecting data
5. Analyzing data and
applying testsA method for assessing
causal inference within a
single case design

Generalized conceptual model or theory
Strategy
#1
Entity
(Stake-
holder)
Strategy
#2
Entity
(Stake-
holder)
Strategy
#3
Entity
(Stake-
holder)
Outcome
Improved
Health
System

• Think about causality—how each step could
cause the next one
• Only include necessary steps
• Make sure that the steps are measurable
• Think ahead to generalizability
oLanguage should be used that is potentially
applicable to other situations and contexts
Strategies/steps to test

By testing a theorized causal mechanism,
process tracing methods allow for within-
case analysis to provide more broadly
generalizable results that can be applied to
programs in various contexts*
Generalizability
*Collier, D. (2011). Understanding process tracing. Political Science and Politics, 44(4):
823–30. Retrieved from https://papers.ssrn.com/sol3/papers.cfm?abstract_id=1856702

hypotheses
hypotheses
(causal mechanism)
4. Collecting data
applying tests
Core hypothesis
Alternative hypothesis
Bonus hypothesis

1. What would we expect to observe if the
hypothesis is true?
• Improvement
• No relevant change
• Worsening
2. Which observations would be very unlikely
unless the hypothesis is true?
• Which observations would practically prove the
hypothesis because they are extremely unlikely
under any other circumstance?
Two guiding questions

Like to
see
Love to
see
Expect to
see
hypotheses
hypotheses
(causal mechanism)
Process
tracing
tests
4. Collecting data
applying tests
Highsensitivity
Evidenceisnecessaryforh
Hoop Double-decisive

•Straw in the wind test
•Hoop test
•Smoking gun test
•Doubly decisive test
Four process tracing tests
Like to
see
Expect to
see
Love to
see
ighsensitivity
videnceisnecessaryforh

• What evidence would you “like to see”?
• Evidence for this test is weak
o Neither necessary nor sufficient to prove the hypothesis
o Low specificity and low sensitivity
• However, it helps move you incrementally toward
greater confidence in the hypothesis when
considered alongside other evidence
Straw in the wind test

• What evidence would you “expect to
see” if the hypothesis were true?
• This evidence is necessary to keep the
hypothesis under consideration
o Low specificity and high sensitivity
• If we don’t see evidence, the hypothesis
can be discarded
Hoop test

• What evidence would you “love to see”?
• This evidence is sufficient to prove the
hypothesis
o High specificity and low sensitivity
• If we see it, we have proven the hypothesis
beyond reasonable doubt
Smoking gun test

• Evidence for this test is rare
• The test is passed when the evidence confirms
the hypothesis and strongly supports causality
o High specificity and high sensitivity
Doubly decisive test
High specificity
Evidence is sufficient for
Lowsensitivity
Evidenceisnotnecessaryforh
Low specificity
Evidence is insufficient for h
Highsensitivity
Straw in the wind Smoking gun

High specificity
Evidence is sufficient for h
Lowsensitivity
Low specificity
Highsensitivity
High specificity
Evidence is sufficient for h
Lowsensitivity
Low specificity
Highsensitivity
Visual of the 4 process tracing tests
h= hypothesis

• Process tracing tests reflect the probability of
observing a particular piece of evidence if the
hypothesis under consideration is true
• Researchers weigh evidence according to
how much the evidence increases the
probability that a hypothesis is true. . .
• . . . or how much not finding the evidence
increases the probability that the hypothesis
is false
Process tracing tests

hypotheses
hypotheses
(causal mechanism)
4. Collecting data
applying tests
A method for assessing
causal inference within
a single case design

In process tracing, the unit of analysis is a case,
which consists of the following:
• Effect under investigation (i.e., observed
outcome)
• Hypothesized cause (i.e., program component
or intervention)
• Hypothesized processes that link the
hypothesized cause and the effect*
Analysis
*Punton, M., & Welle, K. (2015). Applying process tracing in five steps.
Brighton, UK: Institute of Development Studies. Retrieved from
https://www.semanticscholar.org/paper/Applying-Process-Tracing-in-Five-
Steps/c1540ce636740524a07a02a5399a69c0011eca3b

Goal of
process
tracing
To estimate the level of
confidence that a
particular intervention
has caused or
contributed to a
particular outcome in a
particular stepwise,
linear fashion as laid
out in the causal
mechanism

Evaluation 1
Process tracing of
causal mechanisms
implemented in the
Partnership for HIV-Free
Survival (PHFS) in
Uganda

• Innovative project designed to prevent and eliminate
mother-to-child transmission of HIV (PMTCT and eMTCT)
• Brought together proven practices from PMTCT, quality
improvement (QI), nutrition, and community outreach to
improve health outcomes for mothers living with HIV and
their HIV-exposed infants
• Supported by the United States Agency for International
Development and the United States President’s
Emergency Plan for AIDS Relief, PHFS was
active between 2012 and 2016 in six countries in
sub-Saharan Africa: Kenya, Lesotho,
Mozambique, South Africa, Tanzania, and Uganda
PHFS

• PHFS legacy evaluation report
https://www.measureevaluation.org/resources/publications/tr-18-314
• Country-specific briefs
• Outcome evaluation of PHFS in Uganda
• A Practical Way to Eliminate Mother-
to-Child Transmission of HIV: Learning
from the Partnership for HIV-Free
Survival (PHFS)
PHFS evaluations by MEASURE Evaluation

• Our MEASURE Evaluation team designed an
additional evaluation of PHFS in Uganda using
process tracing
• Because of institutional review board delays, we
did not complete the data collection
• We created a guide and
sample protocol as a resource
https://www.measureevaluation.org/resources/publications/ms-19-179
Background

Retrospective theory of change created during the
legacy evaluation of PHFS

Intervention Step 1 Step 2 Step 3 Step 4 Step 5 Outcome
Designated
“clinic” days
for PMTCT
mothers with
HIV-exposed
infants (M-B
pairs) at M-B
care points
Health
facilities
scheduled
designated
clinic days
for PMTCT
M-B pairs
(separate
from clinic
days for HIV-
negative
mothers)
PMTCT M-B
pairs
attended the
clinics on
designated
PMTCT
clinic days
PMTCT
mothers felt
less
stigmatized
for receiving
PMTCT
services
AND formed
informal
support
networks at
PMTCT
clinic days
PMTCT
mothers
were more
satisfied
with their
experiences
at the
health
facilities
M-B pairs
returned for
follow-up
appointments
Increased
retention in
care for
PMTCT
M-B pairs,
compared
with
combined
under-5
clinic days
(April
2013–
August
2015)
Causal mechanism focused on “mother-baby (M-B)
pair clinic days” contributing to increased retention

Data source: Focus group discussions with health
facility staff: midwives and maternal and child
health (MCH) nurses
Core hypothesis: 1.1. Health facility staff
scheduled mothers for their appointments on M-B
pair clinic days
Alternative hypothesis: 1.2. Health facility staff
did not schedule mothers for appointments on M-B
pair clinic days
Step 1. Health facilities scheduled designated
clinic days for PMTCT M-B pairs

Data sources:
1. Quantitative data from outcome evaluation (%
of M-B pairs attending appointments on
scheduled M-B pair clinic days)
2. Health facility staff focus groups
3. Mother focus groups
Core hypothesis: 2.1. M-B pairs attended
designated M-B pair clinic days because their
appointments were scheduled for those days
Step 2. PMTCT M-B pairs attended the clinics
on designated PMTCT clinic days

Alternative hypothesis: 2.2. M-B pairs did not
attend the M-B care points on M-B pair clinic days,
because of time conflicts, personal preference, etc
Bonus hypothesis: 2.3. M-B pairs attended
designated M-B pair clinic days for separate
incentives/ programs that coincided with clinic days
(e.g., nutrition demonstrations, food assistance)
Step 2: PMTCT M-B pairs attended the clinics
on designated PMTCT clinic days

Causal mechanism focused on quality improvement
(QI) supervision and coaching contributing to improved
and sustained QI work
Intervention Step 1 Step 2 Step 3 Step 4 Step 5 Outcome
QI
supervision
and coaching
to health
facilities by
regional and
district QI
coaches
QI
coaches
made
contact
with
assigned
facility-
based
teams
QI coaches
provided initial
and ongoing
supervision,
technical
support, and
motivation to
the facility-
based teams
around key QI
issues
Facility-
based QI
team
members
gained QI
skills, felt
account-
able to the
QI
coaches,
and felt
motivated
to do QI
work
Facility-
based
teams
performed
QI work
Facility-
based QI
teams saw
improve-
ment in
defined
indicators
and patient
outcomes
and felt
motivated
to continue
QI work
Improved
and
sustained
QI work on
PMTCT
over time
throughout
PHFS

Data sources:
1. Focus group with district QI coaches
2. Focus group with regional QI coaches
3. Focus group with facility-based QI team members
4. QI journals (look for frequency of meetings, notation of
change ideas and action plan, tracking of indicators/
projects, overall completeness of QI journals)
Core hypothesis: 4.1. With skills, motivation, and continued
coaching, facility-based teams performed QI work at their
facilities
Alternative hypothesis: 4.2. Despite skills and motivation to
do QI work, facility-based teams did not perform QI work
because of conflicting work priorities
Step 3. Facility-based teams performed QI work

Bonus hypotheses:
• 4.3. QI teams performed QI work because they
feared repercussions of not complying with their
responsibilities
• 4.4. QI teams were able to perform QI work
because they received additional QI resources (i.e.,
better journals, posters, worksheets)
• 4.5 QI teams performed QI work because they
were motivated by the learning sessions (i.e., by
competition or inspiration)
Step 3. Facility-based teams performed QI work

Evaluation 2
Prospective study of health
information system (HIS)
strengthening in Madagascar:
Integrated routine and
surveillance systems with a
focus on malaria

• The health ministry and U.S. government made a
commitment to reduce reporting redundancies through
elimination of vertical systems and/or integration in the
health management information system
• At the same time, a malaria-specific surveillance
system is required for active detection of cases
• MEASURE Evaluation Phase IV work began on this
current mandate in February 2017 with the drafting
and acceptance of the Road Map for the
Subcommittee of the Health Information System (HIS)
• Road Map has nine HIS strengthening strategies
Madagascar context

• Answers key questions about investments in HIS
• Identifies evidence-based packages of HIS
interventions
• Builds the evidence base of what works to
strengthen HIS
MEASURE Evaluation Learning Agenda

Study objectives
1. To document the system strengthening
process, resulting interventions, and efforts
to monitor and assess their implementation
2. To understand and verify the process by
which changes in key HIS interventions
result in changes in HIS performance (data
quality and use) and service delivery
Learning Agenda study of HIS
strengthening in Madagascar

Lessons learned in using process tracing for evaluation

•Implement cascade
trainings for central,
regional, and district staff
on use of the data
collection platform and data
quality assurance tools
•Conduct a national
HIS assessment to
inform planning for
HIS strengthening
•Create a data- quality
assurance protocol
and supervision tools
•Support development of
an HIS strengthening joint
action plan
•Support creation of a
technical working group to
standardize national data
quality assurance practices
•Support the transition of
Access-based health
information software to a web-
based portal that communicates
with DHIS 2*
•Support the DLP to
produce ongoing
monthly malaria
bulletin
Improved stakeholder coordination,
along with the development of
standardized protocols for data
analysis, presentation, and review, will
improve availability of high-quality
data and lead to better programmatic
decisions, especially in the face of
disease outbreaks.
*An electronic platform for the collection and analysis of health data

• Strategy 1. Institutional strengthening of HIS (governance:
structure, standards and procedures, strategic documents /
HIS)
• Strategy 2. Establishment of an effective information
technology (IT) platform for HIS support (availability of IT
equipment, performance of IT tools/software)
• Strategy 3. Development or updating of tools or guides for
the management and use of information (management
tools, management manual, training plan, supervision plan)
• Strategy 4. Development of a data quality assurance
system (monitoring and evaluation, supervision,
verification, quality control, validation and transfer, and
retro-information)
Road Map

• Strategy 5. Enhanced competence of officers responsible
for management, use of data, and use of information at all
levels
• Strategy 6. Creation of a culture of data use for decision
making
• Strategy 7. Creation of a platform for sharing and
dissemination of information (Internet, periodic bulletin,
periodic reviews) with easy access by all users
• Strategy 8. Implementation of the DHIS2 software at the
central level for data warehouses, fed periodically by the
various official databases
• Strategy 9. Mobilization of resources and sustainability
Road Map

1. Systematic tracking of HIS strengthening and
integration activities related to the Road Map
2. Periodic and ongoing focus group discussions
to assess perceptions and implementation of
the road map
3. Qualitative data collection for process tracing
method to describe the causal chain between
the intervention activities and the relationships
with the HIS performance outcomes
Data collection

Causal mechanism (macro-level) focused on
implementation of the Road Map for the HIS sub-
committee to create an efficient, integrated HIS
Intervention 1 2 3 4 5 6 Outcome
Implementation
of the Road Map
for the HIS Sub-
committee
Design and
implement
procedures
and
mechanisms
for
institutional
strength-
ening of HIS
Lead the
process for
development
, updating
and launch
of tools,
guides,
training
plans,
supervision
plans, DQA
system
Develop and
implement
plans to
enhance
competence
of officers
responsible
for manage-
ment, use of
data, and
use of
information
at all levels
Increasingly
engage
officers in
data
demand
and use
Implement
strategies to
create a
culture of
data use for
decision -
making
Craft and
shape use of
communication
platform,
including
availability of
dashboards,
bulletins,
regular data
sharing
meetings, etc.
An efficient,
unique, and
integrated HIS

Data source: Focus groups with implementers of HIS Road Map
Core hypothesis: 1.1. The stakeholder workshop for the
implementation of the HIS Sub-committee galvanized efforts to put
in place procedures and mechanisms for institutional strengthening
of HIS
Alternative hypotheses: 1.2. Establishment of mechanisms for
institutional strengthening of HIS were not related to the
stakeholder workshop: the government of Mozambique was
already carrying out plans
1.3. Establishment of mechanisms for institutional strengthening of
HIS were not related to the stakeholder workshop: implementing
partners have been the primary motivator
Step 1. Design and implement procedures and
mechanisms for institutional strengthening of HIS

Evidence to support the core hypothesis:
“ The workshop really was necessary! Because there were no standard
or procedures, it’s as if everything was done blindly.” (FGD 2)
“DHIS2 implementation was included in the Road Map . . . “ (FGD3)
“The purpose of the Antsirabe workshop was to develop an operational
plan to improve the HIS system.” (FGD 6)
“There were other workshops before Antsirabe . . . in Antsirabe
committees were established and we institutionalized everything.”
Evidence against the core hypothesis:
“ there were a lot of efforts made before this Road Map.” (FGD 3)
“ . . . there are still many departments that do not yet know that these
standards and procedures exist.” (FGD 2)
Step 1. Evidence and test result

• Need to weigh the evidence
• Look back at the hypotheses and the tests
• Core hypothesis: 1.1. The stakeholder workshop for
the implementation of the HIS Sub-committee
galvanized efforts to put in place procedures and
mechanisms for institutional strengthening of HIS
• Tests
• Straw in the wind = Like to see = evidence for this test is weak
• Hoop test = Expect to see = Keeps hypothesis under
consideration
• Smoking gun = Love to see = Sufficient to prove the hypothesis
• Conclusion = Smoking gun
Step 1. Evidence and test result

Conclusions about
the value and
challenges of
process tracing

• Allows in-depth examination of how and why an
intervention influenced the outcome
• Can add a lot of clarity but with extra work for the
initial set-up
• Can improve precision of existing data collection
tools
Overall: Systematic, transparent way to analyze
qualitative data, identify and rule out alternative
explanations, justify conclusions, and show
where data are weakest
Value of process tracing

• Detailed, analytical initial thought and work
required
• Need to avoid developing theory at too “micro” a
level – making it difficult to test hypotheses
• Need to be careful about evidence of absence vs.
absence of evidence for hypotheses
• Important to have an iterative process
Challenges in process tracing

• USAID and PEPFAR for supporting this work
• Colleagues at MEASURE Evaluation, particularly Heidi
Reynolds, for inspiring our team to use process tracing
• Colleagues in Uganda at the Makerere School of Public
Health
• Colleagues in Madagascar, particularly those on the
Research and Evaluation Technical Working Group for the
DEP
• Experts on process tracing: Ir. Cecile Kusters at the
Wageningen Centre for Development Innovation; Melanie
Punton at Itad; and Gavin Stedman-Bryce at Pamoja
Thanks to . . .

• Beach, D. & Pedersen, R. (2013) Process-tracing methods: Foundations and
guidelines. Ann Arbor, MI, USA: University of Michigan Press. Retrieved from
https://www.researchgate.net/publication/287260232_Process-
Tracing_Methods_Foundations_and_Guidelines
• Befani, B. & Mayne, J. (2014). Process tracing and contribution analysis: A
combined approach to generative causal inference for impact evaluation. IDS
Bulletin, 45(6): 17–36. Retrieved from
https://onlinelibrary.wiley.com/doi/abs/10.1111/1759-5436.12110
• Befani, B. & Stedman-Bryce G. (2016). Process tracing and Bayesian updating
for impact evaluation. Evaluation, 23(1): 42–60. Retrieved from
https://journals.sagepub.com/doi/abs/10.1177/1356389016654584
• Better Evaluation. (2016, April 28). Process tracing. Retrieved from
http://betterevaluation.org/evaluation-options/processtracing
• Collier, D. (2011). Understanding process tracing. Political Science and Politics,
44(4): 823–30. Retrieved from
https://papers.ssrn.com/sol3/papers.cfm?abstract_id=1856702
• Punton, M. & Welle, K. (2015). Applying process tracing in five steps. Brighton,
UK: Institute of Development Studies. Retrieved from
https://www.semanticscholar.org/paper/Applying-Process-Tracing-in-Five-
Steps/c1540ce636740524a07a02a5399a69c0011eca3b
References

This presentation was produced with the support of the United States
Agency for International Development (USAID) under the terms of the
Data for Impact (D4I) associate award 7200AA18LA00008, which is
implemented by the Carolina Population Center at the University of
North Carolina at Chapel Hill, in partnership with Palladium
International, LLC; ICF Macro, Inc.; John Snow, Inc.; and Tulane
University. The views expressed in this publication do not necessarily
reflect the views of USAID or the United States government.
www.data4impactproject.org

Lessons learned in using process tracing for evaluation

Lessons learned in using process tracing for evaluation

Recommended

More Related Content

What's hot

What's hot(20)

Similar to Lessons learned in using process tracing for evaluation

Similar to Lessons learned in using process tracing for evaluation(20)

More from MEASURE Evaluation

More from MEASURE Evaluation(20)

Recently uploaded

Recently uploaded(20)

Lessons learned in using process tracing for evaluation

Editor's Notes