Theory-driven evaluation is an important approach in implementation science that focuses on mechanisms of change rather than just outcomes. It involves developing a program theory or theory of change that explains how a strategy is expected to work. A theory-driven evaluation then assesses whether the actual implementation and effects match the initial program theory. It aims to learn whether a strategy works, for whom, in what contexts, and how. This provides valuable information for improving strategies and assessing their transferability.

1. Theory-driven Evaluation in Implementation
Science
Ferdinand C Mukumbang, PhD
Assistant Professor
Department of Global Health

2. The focus of implementation
Science
Evidence-based interventions (EBIs) Implementation strategies
Methods or techniques to enhance the
adoption, implementation, and
sustainability of evidence-based
interventions
Procedures, products, pills, or policies
that have been demonstrated to
improve health behaviors or health
outcomes
Does a diary reminder system
increase fidelity to 16-week
courses of lactolyze?

3. Implementation strategies
Implementation strategy:
Methods or techniques used to enhance the adoption, implementation, and
sustainability of evidence-based interventions
• Example: ART is a health intervention (the “what”), and relevant
implementation strategies may include targeted health worker training or
social marketing to improve the availability of and demand for ART (the
“how”)
• The Expert Recommendations for Implementing Change (ERIC) study
identified 73 discrete implementation strategies

4. Implementation strategies
Waltz (2015) Implementation Science

5. Implementation strategies
• Assess for readiness
• Develop and implement tools
for quality monitoring
• Provide clinical
supervision
• Promote
adaptability
• Develop and promote champions
• Train the trainer
strategy
• Remind clinicians
• Intervene with patient to
increase adherence and
uptake
• Alter consumer fees
• Develop disincentives
• Change service sites
Waltz (2015) Implementation Science

6. Implementation science outcomes
Acceptability
Perception that a treatment, service, practice, or innovation is
agreeable or satisfactory
Adoption
Intention, initial decision, or action to try or employ an evidence-
based practice, from the perspective of provider or organization
Appropriateness
Perceived fit, relevance, or compatibility of the evidence-based
practice for a practice setting, provider, or consumer
Cost
Depends upon the costs of the intervention, the implementation
strategy used, and the location
Feasibility
Extent to which an innovation can be successfully used or carried
out
Fidelity
Degree to which an intervention was implemented as it was
prescribed in the protocol or as it was intended by the developers
Penetration Integration of a practice within a service setting and its subsystems
Sustainability
Extent to which an innovation is maintained or institutionalized
within a service setting’s ongoing, stable operations

7. Common types of evaluations
Implementation strategy design
and implementation

8. Common types of evaluations

9. Process evaluation
• To inform changes or improvements in the strategy’s operations
Goal
Format
• How is the program being implemented?
• How do program beneficiaries describe their program experiences?
• What resources are being described as needed for implementing the
program?
Possible Research Questions

10. Outcome-based evaluation
•Did the strategy increase knowledge or awareness?
•Did the strategy change attitudes or beliefs?
• To identify the results or effects of an implementation strategy
Possible Research Questions
Format
Goal
Implementation
outcome
Health outcome of
interest
Outcomes
Implementatio
n strategy

11. Outcome-based evaluation
•Did the strategy achieve its intended outcomes and objectives?
•Did the strategy produce sustained and meaningful change among
participants?
• To determine whether the strategy is making a positive difference in the lives of
its beneficiaries
Possible Research Questions
Format
Goal
Implementation
outcome
Health
outcome of
interest
Outcomes
Implementatio
n strategy

12. Appendix
What then, is the role of theory in
evaluation?
Are theories being considered important in
Implementation science?

13. The application of theory in IS
Nilsen, P. Making sense of implementation theories, models and frameworks. Implementation Sci 10, 53
(2015). https://doi.org/10.1186/s13012-015-0242-0
Theoretical Domains
Framework
Active
Implementation
Frameworks
Knowledge-to-
Action
Model
Quality
Implementation
Framework
Theory of
Diffusion
Social networks
theories
Implementation
Climate
Normalization
Process Theory
RE-AIM
framework by
Proctor et al.

14. Calls for theorizing in IS

15. Pawson and Tilley (1997)
“Programs are theory-incarnate”
1. Strategies as solutions
Y
X
2. Theories of change
Inputs… activities… outputs… Immediate outcomes… Intermediate outcomes…final outcomes…
3. Realist theories
Program mechanism
Context 1
Context 2
Context 3

16. Theory driven evaluation
A group of approaches that are driven by theory (and not
method) and that focus on mechanisms.
Aim: To learn ‘whether a strategy works, for whom, in which
contexts and how.’
– Essential information for policymakers and program
managers
– Allows appraisal of the transferability of a strategy
• Different from outcome-based evaluations that only assess
whether a program attained its intended results, not how and in
which conditions

17. Program theory (Theory of change)
• A program theory explains how a project, a program, a policy,
or a strategy is understood to contribute to a chain of results
that produce the intended or actual impacts.
• A set of assumptions underlying a program that explains why
the planned activities should lead to the predefined goals and
objectives.
• Different types of diagrams can represent a theory of
change. These are often referred to as logic models, as they
show the overall logic of how the intervention is understood to
work.

18. Uses of program theory
• Program theory can help to identify the elements of a program
which are essential for its widespread replication
• Program theory is used to identify whether program success
has been achieved despite (or perhaps because of) failure to
implement the program as designed.
• If a program does not achieve its intended outcomes, a
program theory evaluation may be able to identify whether
this is due to implementation failure or unsuitable context.

19. The action model
The assumptions, theories and/or
knowledge that inform (and explain)
the design and implementation of
the intervention, and its outcomes
( // logic model, programme logic)
The causal model
Specifies the underlying causal
mechanisms in terms of
relationships between intervention
and outcome, the influence of
context and intervening factors
The elements of the program theory
Two dimensions

20. The elements of the program theory
Dhillon L, Vaca S. Refining theories of change. J Multidiscip Eval.
2018;14:64–87.

21. Theory of change

22. Theory driven evaluation
M Kabongo, et al. Combining the theory of change and realist evaluation approaches to elicit an initial program theory of the
MomConnect program in South Africa. BMC Med Res Methodol 20, 282 (2020). https://doi.org/10.1186/s12874-020-01164-y

23. M Kabongo, et al. Combining the theory of change and realist evaluation approaches to elicit an initial program theory of the
MomConnect program in South Africa. BMC Med Res Methodol 20, 282 (2020). https://doi.org/10.1186/s12874-020-01164-y
Realist evaluation program theory

24. • Strong advocacy for the use of theory in Implementation
sciences, a spillover from the field of evaluation
• Interest in ‘mechanisms’ underlying change
• By identifying contextual influences and articulating
the mechanisms of implementation, theories can be
invaluable for explaining the roles of implementation
strategies and predicting how implementation may
unfold.
Mechanism-based theorizing

25. How do we develop a theory-driven
evaluation design?
Initial program theory Refined program theory
What do the designers expect to be done?
Initial action model
How do the program designers expect the
intervention to work?
Initial change model
What was actually done and to which effect?
Refined action model
How did the actual strategy work?
Refined change model

26. Theory driven research question
Three year - project funded by the EU in urban settings in Mopti
(Mali), Maroua (Cameroon) and Ouagadougou (Burkina Faso)
Aim
To improve the continuum of care in adolescent sexual and
reproductive health (ASRH)
Strategy
Creating networks between service providers of the public and
private sectors and between health and social services
Evaluation question
How does creating networks between different service providers
improve the continuum in ASRH care?

27. Steps to theory-driven evaluation
• Step 1: Assessing the scope of the evaluation and the
appropriateness of TDE
• Step 2: Critical reconstruction of the initial program theory
• Step 3: Choice of data collection methods and development
of tools
• Step 4: Assessing the initial action model: Evaluating the
relevance of strategy design and degree of implementation
• Step 5: Assessing the initial causal model: Establishing the
causal mechanisms and contextual factors and their
interactions
• Step 6: Translating findings into the refined program theory

28. Strengths of Theory-Driven Evaluation
• They can be undertaken when other approaches (e.g.,
experimental designs) cannot be used.
• They allow evaluators and managers to tell a contribution
story that makes sense to those involved.
• They open the black box of the intervention, allowing
evaluators to arrive at findings on why the strategies are
working or are not working.
• They allow conclusions to be drawn on the cause-effect
elements of a strategy.
• They can help leverage existing data to a greater extent and
help focus new data collection.

29. Weaknesses of Theory-Driven Approaches
to Evaluation
• They do not necessarily provide a quantitative measure of
the size of the contribution an intervention is making.
• Developing a theory of change can be difficult because it
involves synthesizing a range of views and information
sources and obtaining stakeholders’ agreement.
• More than one program theory may emerge. If multiple
program theories emerge and are firmly held, they may have
to be tested against the evidence to see which theory best
reflects reality.

30. Appendix
Questions?