The document discusses the foundational aspects and methodologies of realist evaluation within health policy and systems research, highlighting the importance of understanding context and mechanisms that influence outcomes. It emphasizes a theory-driven approach to evaluation, encouraging researchers to frame questions around 'how' and 'under what conditions' interventions work rather than simply 'does it work?'. Interactive discussions and group work are encouraged to explore the complexity of social phenomena and the varied impact of health programs.

1. https://twitter.com/KeystoneHPSR
Building the HPSR CommunityBuilding HPSR Capacity
KEYSTONE
Inaugural KEYSTONE Course on Health Policy and Systems Research 2015
Thinking “really” about HPSR
Realist and theory driven approaches in health
policy and systems research

2. Thinking “really” about HPSR
Realist and theory-driven approaches in health policy and systems research

3. Session objectives
At the end of this session, the participant should be able to:
• Understand the philosophical basis and history for the
Re/TD thinking
• Describe the characteristics of a Re/Td and frame HPSR
questions in a Re/Td lens
• Identify typical evaluation scenarios where the Re/Td
approaches could be applied
• Discuss the advantages and challenges in the use of
Re/Td approaches

4. I Setting the scene
Framing how/why questions (20 mins)
p. 35-42 of the paper
Prashanth, Marchal & Criel (2013) Evaluating healthcare interventions:
Answering the “how” question. Indian Anthropologist. 43:1, 35:50

5. Discussion

6. II Programmes/policies as “theories”

7. What is a programme?
• Brainstorm

8. Programmes (policies too?)
• Not just activities, tasks, objectives and goals
• Not driven only by targets, indicators and
common goals
• Perceptions
• Influence
• Ideas & interests
• Tensions, alignments and negotiations

9. So social phenomena

10. Core concepts – I: 4Ws through
interactive discussion
• In the realist view, there are many possible behavioural
choices that people manifest (or not) in specific conditions,
which results in the outcome. An evaluation using the
realist approach thus begins by seeking an explanation for
why the outcome of interest occurs in some places and not
in others,
• Programmes work through people and their choices.
Programmes facilitate agents to make choices and interact
in new ways by providing physical or symbolic resources (
Pawson and Tilley 1997).
What works, for whom, under what conditions and why?

11. Good intentions -> Good programmes?
• A new cadre of community health workers
• Training programmes in management for
district and block health managers
• Communitisation of health services (cf.
NRHM) – instituting ARS committees at
various levels
• Incentivising performance of health workers:
More performance, more pay

12. Group work
Unpack logic/theories underlying programmes
(policies)
– Orientation (5 mins)
– Groupwork (20 mins)
– Plenary (30 mins)
– Summary (10 mins)

13. Core concepts – I: 4Ws through
interactive discussion
• In the realist view, there are many possible behavioural
choices that people manifest (or not) in specific conditions,
which results in the outcome. An evaluation using the
realist approach thus begins by seeking an explanation for
why the outcome of interest occurs in some places and not
in others,
• Programmes work through people and their choices.
Programmes facilitate agents to make choices and interact
in new ways by providing physical or symbolic resources (
Pawson and Tilley 1997).
What works, for whom, under what conditions and why?

14. Group work
In 3 groups, discuss possible broad design/approaches to
evaluating any of the 3 earlier “intentions”? How will you
go about approaching the evaluation? How would you
frame the question within an Re/Td framework?
In the synthesis, focus on:
• Going beyond “Does it work” to “How/why/under
what conditions”
• How to deal with complexity
• How to gain comprehensive insights about the overall
programme functioning from studying particular
implementation settings in which we do the evaluation

15. Good intentions -> Good programmes?
• A new cadre of community health workers
• Training programmes in management for
district and block health managers
• Communitisation of health services (cf.
NRHM) – instituting ARS committees at
various levels
• Incentivising performance of health workers:
More performance, more pay

16. Synthesis
• Going beyond “Does it work” to
“How/why/under what conditions”
• Complexity thinking: How to deal with
complexity? Unintended outcomes, non-
linearity…
• Systemic thinking: How to keep in mind the
wider “system” and its environment, in which
our programme is but a peg (or is it?)

17. IV Thinking like a (realist) evaluator

18. The realist worldview
• The realist evaluation approach engages with complexity by
taking an open systems approach to social systems (Pawson
2002).
• The number of interacting agents, components and forces
that influence people and organisations in a given system is
high, outcomes are sensitive to initial conditions and thus
outcomes are likely to show high variability.
• The realist approach to this complexity is to view reality as
being stratified, with several layers of explanations to be
found for the empirical observations. This provides a
possibility to hypothesise and refine our explanations of
why some phenomena occur (Pawson 2002; Wong et al.
2013).

19. • Realist origin in theory-driven approaches:
History, present and potential applications
• Realist approaches: Philosophical basis,
history and applications

20. Introduction
Theory-driven inquiry
Increasingly used in Health Policy and Systems
Research
– Spill over from field of evaluation
– Interest in ‘mechanisms’
• Promising approaches for problems of a socially
complex nature, especially if theory-building is
important (Gilson 2012)
• But little guidance
20

21. Theory-driven inquiry
A group of approaches that are driven by theory (and not method)
and that focus on mechanisms
Aim
To learn ‘whether an intervention works, for whom, in which
contexts and how’
• = essential information for policymakers and programme
managers
• Allows analytical generalisability / transferability of an
intervention
– Different from black box evaluations that only assess whether a
programme attained its intended results, not how and in which
conditions
21

22. Theory-driven inquiry
3 main schools
• Theories of change
Connell, Kubisch, Schorr & Weiss (1995)
• Theory-driven evaluation
Chen & Rossi (1987)
aka theory-based evaluation, programme theory evaluation, programme theory-driven
evaluation, etc.
• Realist evaluation & Realist synthesis
Pawson & Tilley (1997)
22

23. Theories of change
Developed by the Roundtable on Community Change (Aspen Institute)
to evaluate complex community-based programmes that involve
• many agencies and actors
• several levels and strands of activities
• objectives and strategies that shift in time
• outcomes that are difficult to measure
More pragmatic in approach and oriented towards stimulating
practical change
See: Connell et al., 1995, Weiss, 1995, Fulbright-Anderson et al., 1998
23

24. Theories of change
Seeks to establish the links between
intervention, context and outcome through
development and testing of logic models that
include
– the populations that are targeted
– indicators used to monitor change
– threshold indicators of significant change
– time lines
24

25. • The development of the initial theory of change
– the intervention activities
– the outputs / expected results
– the sequence of results necessary to obtain the
expected results
• Identification of assumptions and risks underlying
the theory of change
• Identification of other (contextual) factors that may
influence the results chain
Result: a theory of change that can be tested
Theories of change
25

26. 26

27. 27

28. MACKENZIE, M. & BLAMEY, A. 2005. The practice and theory. Lessons from the application of a Theories of Change approach. Evaluation, 11, 151-16828

29. 29

30. Reflect
• Does it get us far enough to “how”?
• Is it sufficiently embracing complexity and
dynamism of social systems?
• Does it explore the inherent human
tendencies to differ in responses given
conditions/histories/situations?

31. • Useful in the planning, monitoring and
evaluation phase
• Focus mainly on learning within the project
and monitoring its implementation
• Risk of linear analysis...
– Analysis of the assumptions about the
relevance of the programme should include the
necessary contextual conditions
Theories of change
31

32. Theory-driven inquiry
Core element: the programme theory
Prosaic, everyday theories that are concerned with
how social problems are generated and programmes
function
= beliefs of programme’s actors, ≠ grand
theories
+
Theories, concepts and knowledge from social science literature
e.g. theory of cognitive dissonance, self-fulfilling prophecy, economic
exchange vs social exchange 32

33. Theory-driven inquiry
The PT = a testable hypothesis, basis for testing assumed
causal chains
• Understanding the contribution of an intervention to
the observed results through a ‘process’ interpretation
of causation
• checking each link between intervention and result
• if links can be validated by empirical evidence
 a causal inference can be made
• Identifying and assessing any significant context
factors
• that may be needed for the intervention to work
• that may influence the implementation
• that may shape the result
33

34. Theory-driven evaluation
Chen and Rossi’s response to
Campbell and Stanley
Experimental and Quasi-experimental Designs for
Research (1963)
• Policy and programmes should be
rooted in social research
• Internal validity is most essential
• Evaluation should methodologically
be based on experimental research
designs
Cronbach
Toward reform or program evaluation: aims,
methods and institutional arrangements (1980)
• Evaluation is not pure research, it is
political in nature
• Evaluation should not serve science,
but enlighten decision-makers and
programme stakeholders
• Emphasis on external validity
34

35. Theory-driven evaluation
Chen and Rossi (1987)
• Need for a perspective on evaluation that ensures both types
of validity
• Not methods, but the problem and existing knowledge
should drive the research and evaluation design
• Describing the (implicit) assumptions that steers the choice
and design of a programme or intervention is useful (“It’s all
about the people”)
• allows to understand what is being implemented and why
• = the programme theory
• Evaluation = critically assessing the programme theory
• guided by social science theory
35

36. Chen and Rossi
– Programmes take place in a specific context that will
influence the outcome as well as the implementation
of the planned intervention
– Programmes work through “sets of dynamic
interactions” that lead to the results (Durie and
Wyatt, 2007)
Theory-driven evaluation
36

37. The elements of the programme theory
37

38. The programme theory
2 dimensions
• The action model
– The assumptions, theories and/or knowledge that inform
(and explain) the design and implementation of the
intervention, and its outcomes // logic model,
programme logic
– The causal model
• Specifies the underlying causal mechanisms in terms of relationships
between intervention and outcome, influence of context and
intervening factors
38

39. The action model
Evaluation of the effectiveness and of consistency of implementation
• Programme theory-failure (policy failure) versus implementation failure
39

40. The causal model
Empirical investigation of the micro-steps in the causal chain
• Allows causal inference on basis of interpretation of process of change,
acknowledging the influence of context
40

41. In practice: 7 steps
1. Assessing the appropriateness of TDE
2. Elicit the initial programme theory
3. Study design
4. Data collection
5. Analysis
6. Synthesis
7. Communication with policymakers, managers, stakeholders
Van Belle, S., Marchal, B., Dubourg, D. & Kegels, G. (2010) How to develop a theory-driven evaluation design - Lessons learned from an adolescent sexual
and reproductive health programme in West-Africa. BMC Public Health, 10, 741.
Theory-driven evaluation in practice
41

43. Realist evaluation
Pawson and Tilley (1997)
In order to be useful for
decision makers,
evaluations need to indicate
what works, for whom,
in what circumstances,
in what respects,
over which duration,
and why?
rather than respond to ‘does
it work?
43
Realist evaluation = primary
research
Realist synthesis = secondary
research

44. Shares emphasis on theory with the 2 other schools
of theory-driven inquiry
• Realist evaluation starts with a theory and ends
with a theory
• Theory should in this case be understood as
middle-range theories (Merton 1968)
“theories that lie between the minor but necessary
working hypotheses (…) and the all-inclusive systematic
efforts to develop a unified theory that will explain all
the observed uniformities of social behavior, social
organization and social change”
44
Realist evaluation

45. Middle range theories
• “Middle-range theory involves abstraction, of course, but they are close enough to
observed data to be incorporated in propositions that permit empirical testing.”
(Merton, 1967)
• A MRT = a theory that is at the correct level of abstraction to be ‘useful’ and ‘testable’
Programme theory
• explains how the planners expect the intervention to reach its objective(s)
CMO configuration
• summary of key empirical findings
Realist evaluation

46. Realist evaluation
The CMO configuration as an analytical tool
Actors
46
Initial
situation
Action
Changed
situation
(outcome)
Context
Mechanism

47. The basic task of social inquiry is to explain interesting,
puzzling, socially significant regularities. Explanation
takes the form of positing some underlying mechanism
which generates the regularity and thus consists of
propositions about how the interplay between structure
and agency has constituted the regularity. Within realist
investigation there is also investigation of how the
workings of such mechanisms are contingent and
conditional, and thus only fired in particular local,
historical or institutional contexts.
(p.71) (Pawson & Tilley, 19

48. Change is brought about by actors
Programme just provides resources
People use resources if programme
triggers a ‘mechanism’
= psychological or social
explanations of behaviour
A pay-for-performance scheme
Intervention
Remuneration tied to performance
Outcome
Improved performance
Mechanism
Money is a motivator
People work harder if you pay them in
function of their performance
Realist evaluation
48
Interventions may work differently for
different people
$ incentives work as long as they are
valuable
- Especially if initial pay is very low
- More for some than for others
- Only until a ceiling is met

49. Intervention works in specific conditions
Actions may have unintended effects
$ incentives  good performance
if personnel is also competent, the
working conditions are right, etc.
Realist evaluation
49
P4P
 Crowding out of intrinsic motivation
 Gaming
Effective P4P requires good monitoring
Outcomes are often explained by several
mechanisms simultaneously
Other factors influence performance
Intrinsic motivation, working
conditions, management style, etc.

50. • Analysis
– ‘Retroduction’ (cfr. Sherlock Holmes)
• Theoretical explanation proceeds by ‘DREI’
– Start with description of significant outcomes
– Retroduction to possible causal mechanisms
– Elimination of alternatives
– Identification of the generative mechanism(s)
« The inquiry starts from an explanandum (the observed
outcome) and walks backwards to find the
concatenation of mechanisms that does the explaining »
(Demetriou 2009)
50
Realist evaluation

51. • Patterns (or demi-regularities) of C + M = O may
occur
(i.e. certain people tend to behave in certain ways
under certain situations)
– If so, one possible inference is that the same
mechanism may be causing the outcome
• At the end of the study, the CMOs are compared
and ‘translated’ to a refined MRT (specification)
• Repeated studies lead to accumulation of insights
51
Realist evaluation

52. RE cycle
52
CMO

55. Why these demi-regs?
• Because of the realist evaluation’s particular
and explicit ontological and epistemological
foundations

57. Critical realism to realist evaluation

58. RE is different in its explicit
philosophical foundations
• Roots in (critical) realism
• Adopting a position between the
extremes of positivism and
relativism
Realist ontology
• accepts that there is a reality
independently of the researcher
Weak relativist epistemology
• knowing this reality through
science is unavoidably relative to
the researcher
Causality is generative
58
Realist evaluation
3 layers: Emperical->Actual->Real

59. Generative causality
• Actors have a potential for change by their very nature
– Agency : role of actors in change
• Structural and institutional features exist independently of
the actors (and researchers)
– Actors and programmes are rooted in a stratified social reality
– Result: interplay between individuals and institutions, each with
their own interest and objectives, within a social context
• Causal mechanisms reside in social relations and context as
much as in individuals
59
Realist evaluation

60. 60

61. Generative causality
• Mechanisms are hidden
– Metaphor of the clock and its invisible mechanism
• Not possible to understand how a clock works by
examining the surface — the numbers on the face and
the movement of the hands
• We need to prise the clock open and go beneath the
‘‘surface (observable) appearance’’ and delve into the
‘‘inner (hidden) workings’’ of the ‘‘balanced spring or the
oscillation of caesium atoms’’
(Pawson & Tilley, 1997, p. 407)
61
Realist evaluation

62. Mechanisms are underlying entities, processes or
structures that lead to outcomes in specific conditions
Mechanisms
– play out at the level of individuals, groups, organisations
and society
• Agency and structure
• Mechanisms can be found in psychological, social, cultural,
political and economic theories
– are usually hidden
– are sensitive to variations in context
(Astbury & Leeuw 2010)
62
Realist evaluation

63. Mechanism is not programme activity
Weiss (1997): contraceptive counseling programme
… if counselling is associated with reduction in pregnancy, the cause of change
might seem to be the counselling.
The mechanism is not the counselling
that is the program activity, the program process
The mechanism might be
– the knowledge that participants gain from the counselling
– the overcoming of cultural taboos against family planning
– Increasing confidence and bolster assertiveness in sexual relationships - a shift
in the power relations between men and women
63
Realist evaluation

64. Optional slides – Skip if needed
64
3 types of mechanisms (Hedstrom & Swedberg, 1998)
Situational mechanisms (macro-to-micro)
• Social situations or events shaping desires and believes of
individuals
– Reference group theory
• Reference groups provide benchmarks needed for comparison and
evaluation of group and personal qualities, circumstances, attitudes,
values and behaviors.
• Individuals compare themselves with ref groups of people who occupy
the social role to which the individual aspires

65. 65
3 types of mechanisms (Hedstrom & Swedberg, 1998)
Situational mechanisms (macro-to-micro)
• Belief formation mechanisms
• “The self-fulfilling prophecy is, in the beginning, a false definition
of the situation evoking a new behaviour which makes the original
false conception come 'true'.” (Merton) – ex: run for the bank

66. 66
3 types of mechanisms (Hedstrom & Swedberg, 1998)
• Action formation mechanisms (micro-to-micro)
– Desires, believes and opportunities shaping individual
action
• Cognitive dissonance theory: coping strategies to deal with stress
that results from holding 2 contrasting beliefs
• Motivation theories
• Cognitive psychology

67. 67
3 types of mechanisms (Hedstrom & Swedberg, 1998)
• Transformational mechanisms (micro-to-macro)
– How individuals shape macro-level outcomes through their
actions and relations
• Reciprocity in social relations + derivatives:
– game theory (tit for tat / prisoners’ dilemma, …)
– high commitment management / social exchange / social capital

68. (Hedstrom & Swedberg,
1998)
68

69. Other ways to look at mechanisms
• Bunge
– mechanisms are constituent events
– = sequences of states or strings of events that entail
transfers of energy and have a function in the
system due to their connection to the system’s
parts and structure
– mechanisms present dynamics that help define the
system (processes)
(Demetriou 2009)
Workshop on complexity and theory-driven
inquiry - Umeå University
69

70. Other ways to look at mechanisms
• Bhaskar (1986)
– mechanism is that aspect of the inner and
environmental structure of a thing by virtue of
which the thing has a certain power
– A mechanism operates when triggered and
normally endures longer than any pattern of
events it triggers
70

71. 71
1. Generative causation
Evaluators need to attend to how and why social programs
have the potential to cause change
2. Ontological depth
Evaluators need to go beneath the surface of observable
inputs and outputs of a programme
3. Mechanisms
Evaluators need to focus on how the causal mechanisms
which generate social and behavioral problems are removed
or countered through the alternative causal mechanisms
introduced in a social programme
Realist evaluation

72. 72
4. Contexts
Evaluators need to understand the contexts within which problem
mechanisms are activated and in which program mechanisms can
be successfully fired
5. Outcomes
Evaluators need to understand what are the outcomes of an
initiative and how they are produced
6. CMO configurations
In order to develop transferable lessons from research, evaluators
need to orient their thinking to context-mechanism-outcome
pattern configurations (CMO-configurations)
Realist evaluation

73. 73
7. Teacher-learner processes
In order to construct and test context-
mechanism-outcome pattern explanations,
evaluators need to engage in a teacher-
learner relationship with program policy
makers, practitioners and participants
Realist evaluation

74. V Group work: Unpacking KEYSTONE (20 mins
groupwork + 30 mins plenary)
Participants go into groups and try to
understand the programme theory of
KEYSTONE. They look at available material,
interview key informants and try to illustrate
“How COULD KEYSTONE bring about a change in
the HPSR practice in the country”.
They practice realist thinking and framing
questions in a realist manner.

75. Plenary

76. Design and implementation of a realist evaluation
6 steps:
1. formulation of the initial middle range theory
2. design of the study
3. data collection
4. data analysis
5. synthesis
6. reformulation of refined MRT
76

77. Illustration with case study
Organisational change in response to capacity building
Prashanth, N. S. et. al. (2012). How does capacity
building of health managers work? A realist
evaluation study protocol. BMJ Open, 2(2), e000882.
doi:10.1136/bmjopen-2012-000882

78. PT reformulated
86
Prashanth, N. S. et. al. (2014). Evaluation of capacity-building program
of district health managers in India: a contextualized theoretical
framework. Frontiers in Public Health, 2(July), 89.
doi:10.3389/fpubh.2014.00089

79. Explaining organisational change
• Identified case studies based
on diversity of context
and/or outcome after
scanning context,
mechanism and outcome
elements
• Confronted the refined PT to
these cases
87
Prashanth, N.S, et. al. (2014). Advancing the application of systems
thinking in health: a realist evaluation of a capacity building
programme for district managers in Tumkur, India. Health Research
Policy and Systems, 12(1), 42. doi:10.1186/1478-4505-12-42

80. Cases
• “In my taluka for example, I think we
can make big change. It is not that
everybody in my taluka want to make
changes. Only one-third of them are
motivated to make changes. And that
is enough. I think I can make a lot of
improvement by motivating these
people.”
- a Gubbi taluka helath manager (g1)
• More resources mean more
opportunities to make change. If they
slowly give more and more power to
us at taluka level, we can make many
more improvements. Right now, very
little is possible at taluka level.
• - another taluka health manager
from Gubbi (g2)
What PIP? What decentralisation? I sent so
many requirements for staff and proposals for
improvement. Only thing I got is more work,
less staff and zero solutions. On one hand, I
have to answer the local ZP members’
complaints and on the other hand, I have to
just keep implementing plans and schemes
coming from above. Nothing can be done
without more staff.
- a health manager from CN Halli (cnh1)
88
“Nothing much can be done without
giving powers at taluka level and
PHCs. I cannot even appoint a Group
D staff. Where is decentralisation in
this?”
- a PHC staff from CN Halli taluka
Prashanth, N.S, et. al. (2014). Advancing the application of systems thinking in health: a realist
evaluation of a capacity building programme for district managers in Tumkur, India. Health
Research Policy and Systems, 12(1), 42. doi:10.1186/1478-4505-12-42

81. Final summary
• When to use RE?
• RE intro videos by Ray Pawson
• @realisteval on twitter
• Mendeley reading list on RE maintained by
PNS - http://www.mendeley.com/groups/535071/critical-realism-and-realist-evaluation/
Thanks to Bruno Marchal, Emilie Robert , Bart Criel & Guy Kegals

82. “Nothing pleases people more than to go on
thinking what they have always thought, and at
the same time imagine that they are thinking
something new and daring: it combines the
advantage of security and the delight of
adventure."
- T.S. Eliot

83. Open Access Policy
KEYSTONE commits itself to the principle of open access to knowledge. In keeping with this, we strongly support open access and use of materials
that we created for the course. While some of the material is in fact original, we have drawn from the large body of knowledge already available under
open licenses that promote sharing and dissemination. In keeping with this spirit, we hereby provide all our materials (wherever they are already not
copyrighted elsewhere as indicated) under Creative Commons Attribution-NonCommercial 4.0 International License. To view a copy of this license
visit http://creativecommons.org/licenses/by-nc/4.0/
This work is ‘Open Access,’ published under a creative commons license which means that you are free to copy, distribute, display, and use the
materials as long as you clearly attribute the work to the KEYSTONE course (suggested attribution: Copyright KEYSTONE Health Policy & Systems
Research Initiative, Public Health Foundation of India and KEYSTONE Partners, 2015), that you do not use this work for any commercial gain in any
form and that you in no way alter, transform or build on the work outside of its use in normal academic scholarship without express permission of
the author and the publisher of this volume. Furthermore, for any reuse or distribution, you must make clear to others the license terms of this work.
This means that you can:
read and store this document free of charge
distribute it for personal use free of charge
print sections of the work for personal use
read or use parts or whole of the work in a context where no financial transactions take place
gain financially from the work in anyway
sell the work or seek monies in relation to the distribution of the work
use the work in any commercial activity of any kind
distribute in or through a commercial body (with the exception of academic usage within educational
institutions such as schools and universities
However, you cannot: