Doctors 2.0 & YOU - 2017 - Methodological and ethical issues of connected dev...
Realist Evaluation : some thoughts about the theoretical foundations
1. THEORY AND PRACTICE OF REALIST EVALUATION
Some thoughts about the theoretical foundations
Valéry Ridde, Emilie Robert, Anthony Lacouture
July, 2; 2015
WHO Meeting : Realist evaluation of national strategic planning and policy processes
equitesante.org
6. PUBLIC HEALTH AND COMPLEXITY
« Multiple entangled actions, acting in often
non-linear relationships, organized in a specific
context and understood as open systems,
researcher who observe in fact itself part of it »
Blaise et al. 2010
Equitesante.org
6
7. 7 CHARACTERISTICS OF COMPLEX INTERVENTIONS
Volitions
Equitesante.org
7
Implementation
Contextes
Time
Outcomes
Rivalry
Emergence
Pawson, 2013
8. CRITICAL REALISM
Equitesante.org
8
POSITIVISM CRITICAL REALISM RELATIVISM
QUESTIONS
Is the policy or
intervention
(cost)-effective?
What works for
whom under
which conditions?
How do actors
experiences, power
and understand?
DISCIPLINES
Epidemiology
Welfare
economics
Political science
Policy analysis
Organizational
studies
Socio-anthropology
DESIGN
Deductive
Fixes
Abductive
Pragmatic/MM
Inductive
Flexibles
Gilson, 2012; Robson, 2002
9. CRITICAL REALISM
• Reality exists outside the human
constructions, but it is perceptible only
through our senses
Equitesante.org
9
Maxwell, Pawson, Bhaskar, etc.
10. GENERATIVE CAUSALITY
• We are not looking for immutable law vs
sucessionist notion of causality
• Importance of mechanisms/context vs
intervention
• But regularities (semi-reg : Lawson)
– semi-predictable patterns
– pathway of program functioning
• Patterns = CMO
Equitesante.org
10
(Pawson, 2006)
12. MECHANISM DEFINITION
• is hidden but real, sensitive to the context and
produces outcomes
• is an element of reasoning and reactions of
agents in regard of the resources available in a
given context to bring about changes
• evolves within an open space-time and social
system of relationships
Equitesante.org
12
Lacouture et al, 2015
13. MECHANISM
• Elements of reasoning of
agents and their choices
about how the change
will be achieved through
the implemented
intervention
• Reactions of agents in
response to the resources
provided by the
intervention
13
Equitesante.org
• Pre-existing characteristics
of the individuals, localities,
situations or systems in
which an intervention is
being set up (= CONTEXT)
• Intentional measures,
activities or a rational plan
taken by program
implementers (= STRATEGIES
OF INTERVENTION)
Lacouture et al, 2015
15. NOT A MECHANISM
Equitesante.org
15
Actor’s attitudes : health workers’ motivation
and satisfaction (A1), the provider–patient
relationship (A2), patients’ satisfaction (A3),
patients’ perceptions on health and on financial
access to healthcare (A4), and health workers’
coping strategies (A5).
Ridde et al, 2012
16. MECHANISM EXAMPLE
CMOc of PR synergy related to interpreting and
disseminating research findings (strategy).
•As a result of the long-term collaborative relationships
between academic and community stakeholders (context),
locally informed and accessible scientific study findings were
presented to the community (strategy). The audiences at
the presentations felt comfortable and safe participating in
the context of their community ownership of the project
(mechanism) and made sense of the data in their own
terms. As a result, new explanatory insights into the data
were generated (outcome).
Equitesante.org
16
Jagosh et al, 2014
18. INTERVENTION THEORY
• "A program is a theory and an evaluation is its
test" (Rein, 1981)
• Programs are “theories incarnate”. (Pawson 2012)
• “It is a small theory attempting explanation of
specific treatment processes, not a large theory
of general social phenomena” (Lipsey, 1993)
Equitesante.org
18
20. INITIAL PT IN UGANDA
• Management practices and leadership styles (interven-
tion) that are supportive of autonomy lead to positive work
behaviour of volunteers, such as improved task per-
formance and persistence (outcomes), because they satisfy
the psychological needs of autonomy, competence and
relatedness (mechanism). More specifically, such
management and leadership contribute to a shift from
‘external motivation’ towards ‘internalised motivation’ for
the volunteers whose initial engagement was more driven
by external motives. Internalised motivation leads not only
to positive work behaviour, but also to well- being. Such
management and leadership furthermore allow volunteers
whose initial engagement was mainly driven by internal
motives to remain engaged and perform well.
Equitesante.org
20
Vareilles et al, 2015
21. MIDLE RANGE THEORY
• « Middle-range theory involves abstraction, of
course, but they are close enough to observed
data to be incorporated in propositions that
permit empirical testing. » Merton, 1967
• Level of abstraction = useful + testable/data
Equitesante.org
21
22. INITIALE MIDLE RANGE THEORY
• "Hospital managers of well-performing hospitals
deploy organisational structures that allow
decentralisation and self-managed teams and
stimulate delegation of decision-making, good
flows of information and transparency. Their
HRM bundles combine employment security,
adequate compensation and training. This results
in strong organisational commitment and trust.
Conditions include competent leaders with an
explicit vision, relatively large decision-making
spaces and adequate resources."
Equitesante.org
22
Marchal et al, 2010
23. INITIALE MIDLE RANGE THEORY
• "‘A pluralistic health system harbours a web of accountability relationships
between actors who combine the roles of account-holder and accountor. 
• Public accountability is actualised when actors are answerable to the public
and remedial action is undertaken. Public accountability requires both
answerability and enforceability in order to be actualised. The answerability or
the capability of the DHMT, SRH INGOs and partnerships to inform, evaluate
and report in an open manner requires transparency and clarity on whom
they represent and deliver services to. Answerability is actualised through
practices grounded in compliance and persuasion.
• Enforceability is grounded in the capability of the public to
demand accountability on the one hand and in meta-governance, i.e. the
function, exercised by a state actor(s), of regulating, monitoring and
sanctioning on the public’s behalf, on the other hand.
• Accountability practices operate along four dimensions (social, political,
organisational and the provider dimension). Each dimension has specific
bundles of strategies, practices, relationships and outcomes. Accountability is
embedded in vertical, horizontal and partnership governance arrangements.
• Multi-level governance arrangements weaken public accountability when
there is confusion over roles and responsibilities between governing actors.’"
Equitesante.org
23
Van Belle, 2014
25. Original CMO for the CHW
• Context
– Lack of community awareness regarding safe motherhood,
danger signs during pregnancy, labour, and postpartum
periods and where to go for services.
• Programme response/mechanism
– Training and deploying trusted members of the
community instills ‘knowledge’ of dangersigns during
pregnancy and labour, safe delivery practices, as well as
newborn care. Regular visits from community members
build relationships between women and health workers,
leading to ‘trust’ in the health system.
• Outcome
– More women aware of complications; increased care
seeking.
Equitesante.org
25Adams et al, 2015
28. CMO CONFIGURATION
A community experiences a high level of
unemployment to which an employment
training program is offered in a remote part of
town (context). But the program has low
enrollment and high attrition, and few people
are trained (outcome). The reason is that people
feel disillusioned by the lack of effort by
program planners to ensure adequate public
transportation to the venue (mechanism).
Equitesante.org
28
Jagosh et al, 2013
29. Initiale CMO CONFIGURATION
Participants who are motivated and share a
good rapport’ (context) may ‘share ideas,
experiences and support one another’
(mechanism), which may be associated with a
smoking cessation and improved weight
management (outcome)
Equitesante.org
29
Mackenzieet al, 2009
30. THEORIES AND CMO
volunteer motivation & work behaviour
• Crowding theory
• Intrinsic motivation theory
• self-determination theory
• Gift exchange and high-
performance HRM theory
• Consistency theory
• Organisational
commitment
• Social exchange theorY
participatory research assessment
30
Equitesante.org
Vareilles et al, 2015
Jagosh et al, 2013
• Theory of Capacity Building
• Communicative Action
Theory
• Empowerment/emancipati
on
• Collaborative Management
• Partnership Synergy Theory
31. FROM CMOi TO CMOf
Equitesante.org
31
Pawson et al, 2012
Complex social interventions …...
... are theories or consist of several theories. …
... involve the participation of numerous stakeholders. ..
... consist of a chain of decision processes. ..
... are not linear and are subject to feedback loops. ..
... are entrenched in several social systems and several contexts. …
... are permeable to the influence of other interventions. ….
.. are open learning systems. ...
UN DE DEUX ????
J,aime bien celui de droite car il mets en avant la question, mais on peut adapter
La RE est fondamentalement une theory driven evaluation
La RE est fondamentalement une theory driven evaluation
A mechanism is hidden but real: Existing prior to the intervention, but latent, a mechanism reveals itself during implementation of the intervention within a given context. Sensitive to the context, it produces outcomes.
A mechanism is an element of reasoning and reactions of agent(s) in regard of their ability to use the resources available in a given context to bring about changes : A mechanism results of the interaction between agents, intervention and structures. It reflects the logic of intervention of the various actors involved directly (e.g. stakeholders) or indirectly (e.g. patients) in the intervention.
A mechanism evolves within an open space-time and social system of relationships : A mechanism is dynamic and it may be interacting with other mechanisms (family of mechanisms), with elements of context or with the effects it has itself produced (i.e. positive or negative feedback).
A mechanism is hidden but real: Existing prior to the intervention, but latent, a mechanism reveals itself during implementation of the intervention within a given context. Sensitive to the context, it produces outcomes.
A mechanism is an element of reasoning and reactions of agent(s) in regard of their ability to use the resources available in a given context to bring about changes : A mechanism results of the interaction between agents, intervention and structures. It reflects the logic of intervention of the various actors involved directly (e.g. stakeholders) or indirectly (e.g. patients) in the intervention.
A mechanism evolves within an open space-time and social system of relationships : A mechanism is dynamic and it may be interacting with other mechanisms (family of mechanisms), with elements of context or with the effects it has itself produced (i.e. positive or negative feedback).
La RE est fondamentalement une theory driven evaluation
Théorie de programme : théorie descriptive de la manière dont l'intervention est censée produire ses effets, et qui inclut nécessairement (pour l'évaluation réaliste) le modèle causal. Ce modèle causal s'appuie sur des théories ou des modèles conceptuels qui constituent le point de départ de l'évaluation réaliste. C'est là où il y a confusion avec la TMP, à mon avis. C'est l'étape n°1 de l'évaluation réaliste.
- Théorie de moyenne portée : théorie explicative de la manière dont le contexte influence les mécanismes mis en oeuvre par l'intervention pour produire des effets. C'est le produit final de l'évaluation réaliste.
La RE est fondamentalement une theory driven evaluation
Deux façons de présenter des théories de programme mais cela n,est pas suffisant, l’intérêt de la RE est d’aller en détail, notamment pour les mécanismes et les CMO
ER : l’intérêt de la RE est surtout de faire le lien entre contexte et mécanisme(s) d’une intervention, ce que ne peut pas faire un théorie de programme.
No all good meca = attention !!
DEUX DIAPOS POUR MONTRER COMMENT NE PAS FAIRE
ER : ce n’est pas un bon exemple : à enlever ! Il faut – je crois – éviter de montrer des tableaux de liste parce que cela ne montre pas les liens de causalité, et laisse penser aux gens que c’est suffisant d’en arriver là. En plus la plupart des mécanismes n’en sont pas.
DEUX DIAPOS POUR MONTRER COMMENT NE PAS FAIRE
ER : ce n’est pas un bon exemple : à enlever ! Il faut – je crois – éviter de montrer des tableaux de liste parce que cela ne montre pas les liens de causalité, et laisse penser aux gens que c’est suffisant d’en arriver là. En plus la plupart des mécanismes n’en sont pas.
Je mettrais bien ici un de tes exemples de thèse, car cela sera un seul, aucun risque, et proche de ce que les gens connaissent ?
Je mettrais bien ici un de tes exemples de thèse, car cela sera un seul, aucun risque, et proche de ce que les gens connaissent ?
ER : tu peux mettre celui-là ; Les défaillances dans la mise en œuvre des politiques d’exemption des paiements directs et les dysfonctionnements parfois préexistants du système de santé sapent les relations entre usagers et prestataires, et contribuent à faire émerger un sentiment de défiance des usagers envers les prestataires de soins, voire envers le système de santé. Cela favorise le phénomène de contournement de certaines formations sanitaires ou le choix de prestataires privés ou de soins domestiques.
Comment formuler des CMO
Revoir, adapté ?
ER : je me rappelle pas trop ce que cela veut dire… Pas sûre que ce soit utile.
L,importance du contexte et dit par tous, pas seulement Pawson et son école
ER : pas certaine de ce schéma… Ca laisse penser que le concept de « contexte » est fourre-tout. Or il faut insister sur le fait que n’est contexte qu’un élément qui déclenche un mécanisme pour produire un effet. C’est la clé – à mon avis – de la compréhension de l’approche réaliste. Il faut que le public soit capable de comprendre que l’enchaînement contexte-mécanisme-effet peut se décliner à différents niveaux d’analyse, et différents niveaux d’abstraction.
La définition LA plus importante est celle de mécanisme.
Le fondement Le plus important est le réalisme critique, c’est-à-dire la relation entre contexte-mécanisme-effet et les semi-régularités.
Je crois que l’on devrait commencer par là, avant de parler des théories.
ER : pas certaine que ce soit pertinent.
Défis de disposer d’outcomes
Défis de disposer d’outcomes
Processus de construction des CMO
ER : il s’agit ici du processus de réflexion et d’analyse, pas nécessairement de construction des C-M-O (car cela concerne également la TMP). J’ai changé la diapo de place parce que je pense qu’on devrait présenter le processus avant le niveau d’abstraction.
PT => CMO => MRT =>
ER : à la place d’études empiriques, tu peux avoir théorie du programme, qui est plus collée à la réalité, sauf que la théorie du programme explicite la manière dont les concepteurs ont pensé que fonctionner l’intervention, et ne représente pas nécessairement ce qui se passe dans la réalité. C’est là toute la nuance ! De plus, la théorie du programme n’est pas explicative mais descriptive, et ne peut donc pas mettre en évidence les interactions avec le contexte.
