This document discusses evaluating complex interventions. It begins by defining evaluation and complex interventions, noting their uncertainty and non-linear relationships. It then outlines several approaches to evaluating complex interventions, including logic analysis, realistic evaluation, and contribution analysis. These approaches emphasize understanding the intervention's theory of change, context, and embracing uncertainty through multiple perspectives. The document concludes by noting these approaches all focus on a program's theory, context, and ensuring a valid and rigorous research process.

1. How to evaluate complex
interventions
Astrid Brousselle, Ph.D.
Canada research Chair in Evaluation and Health System Improvement (crc-easy.ca)
Full professor, community health sciences departement
Researcher, Centre de recherche de l’Hôpital Charles-LeMoyne
Université de Sherbrooke
Toronto, 2015, Program Science Symposium

2. What is evaluation?
- Determining the merit, worth and value (judgement,
several criteria)
- Of an intervention, or of its components (resources,
process, results)
- Systematic inquiry
- Final objective: improvement of the intervention, social
betterment, accountability, social agenda, action
Merriam-Webster: “to judge the value or
condition of (someone or something) in a
careful and thoughtful way”

3. Normative evaluation Evaluative research
Assessment of
the results
Assessment of
the process
Strategic analysis
Effect analysis
Economic eval.
Production analysis
Implementation
analysis
Logic analysis
SEVERAL QUESTIONS
Assessment of
the structure
The problem
Objectives
Activities/service
s
Resources
Effects
Context

4. Lots of models

5. from Shadish Cook
and Leviton (1991)
PracticePractice
Values
KnowledgeUtilization
Social programming Problem

6. Pawson &
Tilley
Source: Alkin MC and Christie CA (2004) An evaluation theory tree. In: Alkin MC (ed.) Evaluation Roots. Tracing Theorists’
Views and Influences. Thousand Oaks, CA: SAGE, 12–65. ADAPTED from Shadish Cook and Leviton (1991)

7. The utilization
paradise
The
lobbying
zone
The
knowledge-
driven swamp
Evaluation models and
use
CONTANDRIOPOULOS, D., BROUSSELLE, A. (2012) “Evaluation models and evaluation use?” Evaluation: The International Journal of
Theory Research and Practice, vol. 18: 61-77.

8. Complex Intervention
Complex interventions are characterized by :
• Uncertainty, unpredictability
• Interdependence among a large number of actors
• Adapting and evolving actors
• Emergent outcomes created by the connections or
relationships in the system
• Non-linearity between inputs and outputs
(Shiell, Hawe, & Gold, 2008, cited in Zimmerman et al. 2012 )

9. Analysing complex
interventions
Reduction trap / complexity trap
Embracing complexity:
• Accepting paradoxes (not contradictions) and dealing with
them
• Analyzing the relations, the “in between”
• Accepting uncertainty
• Using multiple perspectives
(Zimmerman et al. 2012 )
A statement that
appears to be
self-contradictory
or silly but may
include a latent
truth.
A statement that
appears to be
self-contradictory
or silly but may
include a latent
truth.

10. Evaluation answers to
interventions’ complexity
Logic analysis
Realistic evaluation
Contribution analysis
Economic value of programs

11. Logic analysis
Source: Rey et al. 2012
Crucial characteristics
of the intervention
Effects
Desired effects
Other effects
Critical conditions
Direct logic analysis
Alternatives
(interventions)
Effects
Other effects
Critical conditions
Reverse logic analysis
Desired effects

12. Realistic evaluation
“What works, for
whom, under which
circumstances?”
Theory
Hypotheses
Observations
Program
specification
Mechanisms
Contexts
Outcomes

13. Contribution analysis
Source: Mayne, 2012

14. Economic value
TCHOUAKET, E., BROUSSELLE, A., FANSI, A., DIONNE, P-A., BERTRAND, E., FORTIN, C. (2013) The economic value of Quebec's water
fluoridation program, Journal of Public Health, 26 juin 2013, open access: springerlink.com

15. What these approaches share
Centrality of the “theory of change”
Importance of the context
Integration of various perspectives
Plausible interpretation (uncertainty)
Valid and rigorous research process
Creativity
http://www.jr-art.net

16. References
Alkin MC and Christie CA (2004) An evaluation theory tree. In: Alkin MC (ed.) Evaluation Roots. Tracing Theorists’ Views and
Influences. Thousand Oaks, CA: SAGE, 12–65.
Champagne F., Contandriopoulos A-P, Brousselle A., Hartz Z., Denis J-L. (2011-2e
édition) « L'évaluation dans le domaine de la
santé : concepts et méthodes », dans BROUSSELLE A., Champagne F., Contandriopoulos A P., Hartz Z. (Eds) L'Évaluation :‑
Concepts et méthodes, Presses de l’Université de Montréal : 49-70.
Contandriopoulos, D., Brousselle, A. (2012) “Evaluation models and evaluation use?” Evaluation: The International Journal
of Theory Research and Practice, vol. 18: 61-77.
Mayne J. Contribution analysis: Coming of age? Evaluation 2012; 18(3): 270-280.
Dubois N, Lloyd S, Houle J, Mercier C, Brousselle A, Rey L. Discussion: Practice-based evaluation as a response to address
intervention complexity. Canadian Journal of Program Evaluation 2012; 26(3): 105-113.
Pawson, R., & Tilley, N. (2005). Realistic evaluation. In S. Mathison (Ed.), Encyclopedia of evaluation (pp. 362-367). Thousand Oaks,
CA: Sage Publications.
Rey L., Brousselle , A., Dedobbeleer, N. (2012) «Logic analysis: testing program theory to better evaluate complex interventions»,
dans Houle, J., Dubois, N., Lloyd, S., Mercier C., Hartz, Z., Brousselle, A. (Ed.) (2012) « L'évaluation des interventions complexes »,
numéro spécial, Revue Canadienne d'Évaluation de Programme/ Canadian Journal of Program Evaluation, Volume 26(3): 61-89.
Shadish W.R., Cook T.D., Leviton L.C. (1991) «Good Theory for Social Program Evaluation», in Shadish W.R., Cook T.D., Leviton
L.C., Foundations of program evaluation. Theories of practice, Sage Publications: 36-67.
Shiell, A., Hawe, P., & Gold, L. (2008). Complex interventions or complex systems? Implications for health economic evaluation.
BMJ, 336(7656), 1281–1283. doi:10.1136/bmj.39569.510521.AD
Tchouaket, E., Brousselle , A., Fansi, A., Dionne, P-A., Bertrand, E., Fortin, C. (2013) The economic value of Quebec's water
fluoridation program, Journal of Public Health, 26 juin 2013, open access: springerlink.com
Zimmerman, B., Dubois, N., Houle, N., Lloyd, S., Mercier, C., Brousselle, A. , Rey, L. (2012) «How does complexity impact
evaluation?: An Introduction to the Special Issue», numéro spécial, Revue Canadienne d'Évaluation de Programme/ Canadian
Journal of Program Evaluation, Volume 26(3):v-xx.
Astrid.brousselle@usherbrooke.ca
crc-easy.ca