Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

KEYSTONE / Module 7 / Slideshow 1 / Realist and theory driven approaches in HPSR


Published on

KEYSTONE HPSR Initiative // Module 7: Realist evaluation // Slideshow 1: Realist and theory driven approaches in HPSR
This is the only slideshow of Module 7: Realist evaluation, of the KEYSTONE Teaching and Learning Resources for Health Policy and Systems Research

To access video sessions and slides for all modules copy and past the following link in your browser:

Module 7: Realist evaluation
Programmes and policies are complex in their design and implementation because of the number of interacting agents, components and forces that influence people and organisations in a given system. In this module through the realist evaluation approach explores why programmes/interventions work for some and not for others and getting to the core issue of trying to understand the conditions under which the interventions works.

There is 1 slideshow in this module.

Module 7: Realist evaluation
Module 7 Slideshow 1: Realist and theory- driven approaches in HPSR

The other modules in this series are:
Module 1: Introducing Health Systems & Health Policy
Module 2: Social justice, equity & gender
Module 3: System complexity
Module 4: Health Policy and Systems Research frameworks
Module 5: Economic analysis
Module 6: Policy analysis
Module 8: Systems thinking
Module 9: Ethnography
Module 10: Implementation research
Module 11: Participatory action research
Module 12: Knowledge translation
Module 13: Research Plan Writing

KEYSTONE is a collective initiative of several Indian health policy and systems research (HPSR) organizations to strengthen national capacity in HPSR towards addressing critical needs of health systems and policy development. KEYSTONE is convened by the Public Health Foundation of India in its role as Nodal Institute of the Alliance for Health Policy and Systems Research (AHPSR).

The inaugural KEYSTONE short course was conducted in New Delhi from 23 February – 5 March 2015. In the process of delivering the inaugural course, a suite of teaching and learning materials were developed under Creative Commons license, and are being made available as open access resources. The KEYSTONE teaching and learning resources include 38 videos and 32 slide presentations organized into 13 modules. These materials cover foundational concepts, common approaches used in HPSR, and guidance for preparing a research plan.

These resources were created and are made available through support and funding from the Alliance for Health Policy & Systems Research (AHPSR), WHO for the KEYSTONE initiative.

Published in: Education
  • Be the first to comment

KEYSTONE / Module 7 / Slideshow 1 / Realist and theory driven approaches in HPSR

  1. 1. Building the HPSR CommunityBuilding HPSR Capacity KEYSTONE Inaugural KEYSTONE Course on Health Policy and Systems Research 2015 Thinking “really” about HPSR Realist and theory driven approaches in health policy and systems research
  2. 2. Thinking “really” about HPSR Realist and theory-driven approaches in health policy and systems research
  3. 3. Session objectives At the end of this session, the participant should be able to: • Understand the philosophical basis and history for the Re/TD thinking • Describe the characteristics of a Re/Td and frame HPSR questions in a Re/Td lens • Identify typical evaluation scenarios where the Re/Td approaches could be applied • Discuss the advantages and challenges in the use of Re/Td approaches
  4. 4. I Setting the scene Framing how/why questions (20 mins) p. 35-42 of the paper Prashanth, Marchal & Criel (2013) Evaluating healthcare interventions: Answering the “how” question. Indian Anthropologist. 43:1, 35:50
  5. 5. Discussion
  6. 6. II Programmes/policies as “theories”
  7. 7. What is a programme? • Brainstorm
  8. 8. Programmes (policies too?) • Not just activities, tasks, objectives and goals • Not driven only by targets, indicators and common goals • Perceptions • Influence • Ideas & interests • Tensions, alignments and negotiations
  9. 9. So social phenomena
  10. 10. Core concepts – I: 4Ws through interactive discussion • In the realist view, there are many possible behavioural choices that people manifest (or not) in specific conditions, which results in the outcome. An evaluation using the realist approach thus begins by seeking an explanation for why the outcome of interest occurs in some places and not in others, • Programmes work through people and their choices. Programmes facilitate agents to make choices and interact in new ways by providing physical or symbolic resources ( Pawson and Tilley 1997). What works, for whom, under what conditions and why?
  11. 11. Good intentions -> Good programmes? • A new cadre of community health workers • Training programmes in management for district and block health managers • Communitisation of health services (cf. NRHM) – instituting ARS committees at various levels • Incentivising performance of health workers: More performance, more pay
  12. 12. Group work Unpack logic/theories underlying programmes (policies) – Orientation (5 mins) – Groupwork (20 mins) – Plenary (30 mins) – Summary (10 mins)
  13. 13. Core concepts – I: 4Ws through interactive discussion • In the realist view, there are many possible behavioural choices that people manifest (or not) in specific conditions, which results in the outcome. An evaluation using the realist approach thus begins by seeking an explanation for why the outcome of interest occurs in some places and not in others, • Programmes work through people and their choices. Programmes facilitate agents to make choices and interact in new ways by providing physical or symbolic resources ( Pawson and Tilley 1997). What works, for whom, under what conditions and why?
  14. 14. Group work In 3 groups, discuss possible broad design/approaches to evaluating any of the 3 earlier “intentions”? How will you go about approaching the evaluation? How would you frame the question within an Re/Td framework? In the synthesis, focus on: • Going beyond “Does it work” to “How/why/under what conditions” • How to deal with complexity • How to gain comprehensive insights about the overall programme functioning from studying particular implementation settings in which we do the evaluation
  15. 15. Good intentions -> Good programmes? • A new cadre of community health workers • Training programmes in management for district and block health managers • Communitisation of health services (cf. NRHM) – instituting ARS committees at various levels • Incentivising performance of health workers: More performance, more pay
  16. 16. Synthesis • Going beyond “Does it work” to “How/why/under what conditions” • Complexity thinking: How to deal with complexity? Unintended outcomes, non- linearity… • Systemic thinking: How to keep in mind the wider “system” and its environment, in which our programme is but a peg (or is it?)
  17. 17. IV Thinking like a (realist) evaluator
  18. 18. The realist worldview • The realist evaluation approach engages with complexity by taking an open systems approach to social systems (Pawson 2002). • The number of interacting agents, components and forces that influence people and organisations in a given system is high, outcomes are sensitive to initial conditions and thus outcomes are likely to show high variability. • The realist approach to this complexity is to view reality as being stratified, with several layers of explanations to be found for the empirical observations. This provides a possibility to hypothesise and refine our explanations of why some phenomena occur (Pawson 2002; Wong et al. 2013).
  19. 19. • Realist origin in theory-driven approaches: History, present and potential applications • Realist approaches: Philosophical basis, history and applications
  20. 20. Introduction Theory-driven inquiry Increasingly used in Health Policy and Systems Research – Spill over from field of evaluation – Interest in ‘mechanisms’ • Promising approaches for problems of a socially complex nature, especially if theory-building is important (Gilson 2012) • But little guidance 20
  21. 21. Theory-driven inquiry A group of approaches that are driven by theory (and not method) and that focus on mechanisms Aim To learn ‘whether an intervention works, for whom, in which contexts and how’ • = essential information for policymakers and programme managers • Allows analytical generalisability / transferability of an intervention – Different from black box evaluations that only assess whether a programme attained its intended results, not how and in which conditions 21
  22. 22. Theory-driven inquiry 3 main schools • Theories of change Connell, Kubisch, Schorr & Weiss (1995) • Theory-driven evaluation Chen & Rossi (1987) aka theory-based evaluation, programme theory evaluation, programme theory-driven evaluation, etc. • Realist evaluation & Realist synthesis Pawson & Tilley (1997) 22
  23. 23. Theories of change Developed by the Roundtable on Community Change (Aspen Institute) to evaluate complex community-based programmes that involve • many agencies and actors • several levels and strands of activities • objectives and strategies that shift in time • outcomes that are difficult to measure More pragmatic in approach and oriented towards stimulating practical change See: Connell et al., 1995, Weiss, 1995, Fulbright-Anderson et al., 1998 23
  24. 24. Theories of change Seeks to establish the links between intervention, context and outcome through development and testing of logic models that include – the populations that are targeted – indicators used to monitor change – threshold indicators of significant change – time lines 24
  25. 25. • The development of the initial theory of change – the intervention activities – the outputs / expected results – the sequence of results necessary to obtain the expected results • Identification of assumptions and risks underlying the theory of change • Identification of other (contextual) factors that may influence the results chain Result: a theory of change that can be tested Theories of change 25
  26. 26. 26
  27. 27. 27
  28. 28. MACKENZIE, M. & BLAMEY, A. 2005. The practice and theory. Lessons from the application of a Theories of Change approach. Evaluation, 11, 151-16828
  29. 29. 29
  30. 30. Reflect • Does it get us far enough to “how”? • Is it sufficiently embracing complexity and dynamism of social systems? • Does it explore the inherent human tendencies to differ in responses given conditions/histories/situations?
  31. 31. • Useful in the planning, monitoring and evaluation phase • Focus mainly on learning within the project and monitoring its implementation • Risk of linear analysis... – Analysis of the assumptions about the relevance of the programme should include the necessary contextual conditions Theories of change 31
  32. 32. Theory-driven inquiry Core element: the programme theory Prosaic, everyday theories that are concerned with how social problems are generated and programmes function = beliefs of programme’s actors, ≠ grand theories + Theories, concepts and knowledge from social science literature e.g. theory of cognitive dissonance, self-fulfilling prophecy, economic exchange vs social exchange 32
  33. 33. Theory-driven inquiry The PT = a testable hypothesis, basis for testing assumed causal chains • Understanding the contribution of an intervention to the observed results through a ‘process’ interpretation of causation • checking each link between intervention and result • if links can be validated by empirical evidence  a causal inference can be made • Identifying and assessing any significant context factors • that may be needed for the intervention to work • that may influence the implementation • that may shape the result 33
  34. 34. Theory-driven evaluation Chen and Rossi’s response to Campbell and Stanley Experimental and Quasi-experimental Designs for Research (1963) • Policy and programmes should be rooted in social research • Internal validity is most essential • Evaluation should methodologically be based on experimental research designs Cronbach Toward reform or program evaluation: aims, methods and institutional arrangements (1980) • Evaluation is not pure research, it is political in nature • Evaluation should not serve science, but enlighten decision-makers and programme stakeholders • Emphasis on external validity 34
  35. 35. Theory-driven evaluation Chen and Rossi (1987) • Need for a perspective on evaluation that ensures both types of validity • Not methods, but the problem and existing knowledge should drive the research and evaluation design • Describing the (implicit) assumptions that steers the choice and design of a programme or intervention is useful (“It’s all about the people”) • allows to understand what is being implemented and why • = the programme theory • Evaluation = critically assessing the programme theory • guided by social science theory 35
  36. 36. Chen and Rossi – Programmes take place in a specific context that will influence the outcome as well as the implementation of the planned intervention – Programmes work through “sets of dynamic interactions” that lead to the results (Durie and Wyatt, 2007) Theory-driven evaluation 36
  37. 37. The elements of the programme theory 37
  38. 38. The programme theory 2 dimensions • The action model – The assumptions, theories and/or knowledge that inform (and explain) the design and implementation of the intervention, and its outcomes // logic model, programme logic – The causal model • Specifies the underlying causal mechanisms in terms of relationships between intervention and outcome, influence of context and intervening factors 38
  39. 39. The action model Evaluation of the effectiveness and of consistency of implementation • Programme theory-failure (policy failure) versus implementation failure 39
  40. 40. The causal model Empirical investigation of the micro-steps in the causal chain • Allows causal inference on basis of interpretation of process of change, acknowledging the influence of context 40
  41. 41. In practice: 7 steps 1. Assessing the appropriateness of TDE 2. Elicit the initial programme theory 3. Study design 4. Data collection 5. Analysis 6. Synthesis 7. Communication with policymakers, managers, stakeholders Van Belle, S., Marchal, B., Dubourg, D. & Kegels, G. (2010) How to develop a theory-driven evaluation design - Lessons learned from an adolescent sexual and reproductive health programme in West-Africa. BMC Public Health, 10, 741. Theory-driven evaluation in practice 41
  42. 42. Realist evaluation Pawson and Tilley (1997) In order to be useful for decision makers, evaluations need to indicate what works, for whom, in what circumstances, in what respects, over which duration, and why? rather than respond to ‘does it work? 43 Realist evaluation = primary research Realist synthesis = secondary research
  43. 43. Shares emphasis on theory with the 2 other schools of theory-driven inquiry • Realist evaluation starts with a theory and ends with a theory • Theory should in this case be understood as middle-range theories (Merton 1968) “theories that lie between the minor but necessary working hypotheses (…) and the all-inclusive systematic efforts to develop a unified theory that will explain all the observed uniformities of social behavior, social organization and social change” 44 Realist evaluation
  44. 44. Middle range theories • “Middle-range theory involves abstraction, of course, but they are close enough to observed data to be incorporated in propositions that permit empirical testing.” (Merton, 1967) • A MRT = a theory that is at the correct level of abstraction to be ‘useful’ and ‘testable’ Programme theory • explains how the planners expect the intervention to reach its objective(s) CMO configuration • summary of key empirical findings Realist evaluation
  45. 45. Realist evaluation The CMO configuration as an analytical tool Actors 46 Initial situation Action Changed situation (outcome) Context Mechanism
  46. 46. The basic task of social inquiry is to explain interesting, puzzling, socially significant regularities. Explanation takes the form of positing some underlying mechanism which generates the regularity and thus consists of propositions about how the interplay between structure and agency has constituted the regularity. Within realist investigation there is also investigation of how the workings of such mechanisms are contingent and conditional, and thus only fired in particular local, historical or institutional contexts. (p.71) (Pawson & Tilley, 19
  47. 47. Change is brought about by actors Programme just provides resources People use resources if programme triggers a ‘mechanism’ = psychological or social explanations of behaviour A pay-for-performance scheme Intervention Remuneration tied to performance Outcome Improved performance Mechanism Money is a motivator People work harder if you pay them in function of their performance Realist evaluation 48 Interventions may work differently for different people $ incentives work as long as they are valuable - Especially if initial pay is very low - More for some than for others - Only until a ceiling is met
  48. 48. Intervention works in specific conditions Actions may have unintended effects $ incentives  good performance if personnel is also competent, the working conditions are right, etc. Realist evaluation 49 P4P  Crowding out of intrinsic motivation  Gaming Effective P4P requires good monitoring Outcomes are often explained by several mechanisms simultaneously Other factors influence performance Intrinsic motivation, working conditions, management style, etc.
  49. 49. • Analysis – ‘Retroduction’ (cfr. Sherlock Holmes) • Theoretical explanation proceeds by ‘DREI’ – Start with description of significant outcomes – Retroduction to possible causal mechanisms – Elimination of alternatives – Identification of the generative mechanism(s) « The inquiry starts from an explanandum (the observed outcome) and walks backwards to find the concatenation of mechanisms that does the explaining » (Demetriou 2009) 50 Realist evaluation
  50. 50. • Patterns (or demi-regularities) of C + M = O may occur (i.e. certain people tend to behave in certain ways under certain situations) – If so, one possible inference is that the same mechanism may be causing the outcome • At the end of the study, the CMOs are compared and ‘translated’ to a refined MRT (specification) • Repeated studies lead to accumulation of insights 51 Realist evaluation
  51. 51. RE cycle 52 CMO
  52. 52. Why these demi-regs? • Because of the realist evaluation’s particular and explicit ontological and epistemological foundations
  53. 53. Critical realism to realist evaluation
  54. 54. RE is different in its explicit philosophical foundations • Roots in (critical) realism • Adopting a position between the extremes of positivism and relativism Realist ontology • accepts that there is a reality independently of the researcher Weak relativist epistemology • knowing this reality through science is unavoidably relative to the researcher Causality is generative 58 Realist evaluation 3 layers: Emperical->Actual->Real
  55. 55. Generative causality • Actors have a potential for change by their very nature – Agency : role of actors in change • Structural and institutional features exist independently of the actors (and researchers) – Actors and programmes are rooted in a stratified social reality – Result: interplay between individuals and institutions, each with their own interest and objectives, within a social context • Causal mechanisms reside in social relations and context as much as in individuals 59 Realist evaluation
  56. 56. 60
  57. 57. Generative causality • Mechanisms are hidden – Metaphor of the clock and its invisible mechanism • Not possible to understand how a clock works by examining the surface — the numbers on the face and the movement of the hands • We need to prise the clock open and go beneath the ‘‘surface (observable) appearance’’ and delve into the ‘‘inner (hidden) workings’’ of the ‘‘balanced spring or the oscillation of caesium atoms’’ (Pawson & Tilley, 1997, p. 407) 61 Realist evaluation
  58. 58. Mechanisms are underlying entities, processes or structures that lead to outcomes in specific conditions Mechanisms – play out at the level of individuals, groups, organisations and society • Agency and structure • Mechanisms can be found in psychological, social, cultural, political and economic theories – are usually hidden – are sensitive to variations in context (Astbury & Leeuw 2010) 62 Realist evaluation
  59. 59. Mechanism is not programme activity Weiss (1997): contraceptive counseling programme … if counselling is associated with reduction in pregnancy, the cause of change might seem to be the counselling. The mechanism is not the counselling that is the program activity, the program process The mechanism might be – the knowledge that participants gain from the counselling – the overcoming of cultural taboos against family planning – Increasing confidence and bolster assertiveness in sexual relationships - a shift in the power relations between men and women 63 Realist evaluation
  60. 60. Optional slides – Skip if needed 64 3 types of mechanisms (Hedstrom & Swedberg, 1998) Situational mechanisms (macro-to-micro) • Social situations or events shaping desires and believes of individuals – Reference group theory • Reference groups provide benchmarks needed for comparison and evaluation of group and personal qualities, circumstances, attitudes, values and behaviors. • Individuals compare themselves with ref groups of people who occupy the social role to which the individual aspires
  61. 61. 65 3 types of mechanisms (Hedstrom & Swedberg, 1998) Situational mechanisms (macro-to-micro) • Belief formation mechanisms • “The self-fulfilling prophecy is, in the beginning, a false definition of the situation evoking a new behaviour which makes the original false conception come 'true'.” (Merton) – ex: run for the bank
  62. 62. 66 3 types of mechanisms (Hedstrom & Swedberg, 1998) • Action formation mechanisms (micro-to-micro) – Desires, believes and opportunities shaping individual action • Cognitive dissonance theory: coping strategies to deal with stress that results from holding 2 contrasting beliefs • Motivation theories • Cognitive psychology
  63. 63. 67 3 types of mechanisms (Hedstrom & Swedberg, 1998) • Transformational mechanisms (micro-to-macro) – How individuals shape macro-level outcomes through their actions and relations • Reciprocity in social relations + derivatives: – game theory (tit for tat / prisoners’ dilemma, …) – high commitment management / social exchange / social capital
  64. 64. (Hedstrom & Swedberg, 1998) 68
  65. 65. Other ways to look at mechanisms • Bunge – mechanisms are constituent events – = sequences of states or strings of events that entail transfers of energy and have a function in the system due to their connection to the system’s parts and structure – mechanisms present dynamics that help define the system (processes) (Demetriou 2009) Workshop on complexity and theory-driven inquiry - Umeå University 69
  66. 66. Other ways to look at mechanisms • Bhaskar (1986) – mechanism is that aspect of the inner and environmental structure of a thing by virtue of which the thing has a certain power – A mechanism operates when triggered and normally endures longer than any pattern of events it triggers 70
  67. 67. 71 1. Generative causation Evaluators need to attend to how and why social programs have the potential to cause change 2. Ontological depth Evaluators need to go beneath the surface of observable inputs and outputs of a programme 3. Mechanisms Evaluators need to focus on how the causal mechanisms which generate social and behavioral problems are removed or countered through the alternative causal mechanisms introduced in a social programme Realist evaluation
  68. 68. 72 4. Contexts Evaluators need to understand the contexts within which problem mechanisms are activated and in which program mechanisms can be successfully fired 5. Outcomes Evaluators need to understand what are the outcomes of an initiative and how they are produced 6. CMO configurations In order to develop transferable lessons from research, evaluators need to orient their thinking to context-mechanism-outcome pattern configurations (CMO-configurations) Realist evaluation
  69. 69. 73 7. Teacher-learner processes In order to construct and test context- mechanism-outcome pattern explanations, evaluators need to engage in a teacher- learner relationship with program policy makers, practitioners and participants Realist evaluation
  70. 70. V Group work: Unpacking KEYSTONE (20 mins groupwork + 30 mins plenary) Participants go into groups and try to understand the programme theory of KEYSTONE. They look at available material, interview key informants and try to illustrate “How COULD KEYSTONE bring about a change in the HPSR practice in the country”. They practice realist thinking and framing questions in a realist manner.
  71. 71. Plenary
  72. 72. Design and implementation of a realist evaluation 6 steps: 1. formulation of the initial middle range theory 2. design of the study 3. data collection 4. data analysis 5. synthesis 6. reformulation of refined MRT 76
  73. 73. Illustration with case study Organisational change in response to capacity building Prashanth, N. S. et. al. (2012). How does capacity building of health managers work? A realist evaluation study protocol. BMJ Open, 2(2), e000882. doi:10.1136/bmjopen-2012-000882
  74. 74. PT reformulated 86 Prashanth, N. S. et. al. (2014). Evaluation of capacity-building program of district health managers in India: a contextualized theoretical framework. Frontiers in Public Health, 2(July), 89. doi:10.3389/fpubh.2014.00089
  75. 75. Explaining organisational change • Identified case studies based on diversity of context and/or outcome after scanning context, mechanism and outcome elements • Confronted the refined PT to these cases 87 Prashanth, N.S, et. al. (2014). Advancing the application of systems thinking in health: a realist evaluation of a capacity building programme for district managers in Tumkur, India. Health Research Policy and Systems, 12(1), 42. doi:10.1186/1478-4505-12-42
  76. 76. Cases • “In my taluka for example, I think we can make big change. It is not that everybody in my taluka want to make changes. Only one-third of them are motivated to make changes. And that is enough. I think I can make a lot of improvement by motivating these people.” - a Gubbi taluka helath manager (g1) • More resources mean more opportunities to make change. If they slowly give more and more power to us at taluka level, we can make many more improvements. Right now, very little is possible at taluka level. • - another taluka health manager from Gubbi (g2) What PIP? What decentralisation? I sent so many requirements for staff and proposals for improvement. Only thing I got is more work, less staff and zero solutions. On one hand, I have to answer the local ZP members’ complaints and on the other hand, I have to just keep implementing plans and schemes coming from above. Nothing can be done without more staff. - a health manager from CN Halli (cnh1) 88 “Nothing much can be done without giving powers at taluka level and PHCs. I cannot even appoint a Group D staff. Where is decentralisation in this?” - a PHC staff from CN Halli taluka Prashanth, N.S, et. al. (2014). Advancing the application of systems thinking in health: a realist evaluation of a capacity building programme for district managers in Tumkur, India. Health Research Policy and Systems, 12(1), 42. doi:10.1186/1478-4505-12-42
  77. 77. Final summary • When to use RE? • RE intro videos by Ray Pawson • @realisteval on twitter • Mendeley reading list on RE maintained by PNS - Thanks to Bruno Marchal, Emilie Robert , Bart Criel & Guy Kegals
  78. 78. “Nothing pleases people more than to go on thinking what they have always thought, and at the same time imagine that they are thinking something new and daring: it combines the advantage of security and the delight of adventure." - T.S. Eliot
  79. 79. Open Access Policy KEYSTONE commits itself to the principle of open access to knowledge. In keeping with this, we strongly support open access and use of materials that we created for the course. While some of the material is in fact original, we have drawn from the large body of knowledge already available under open licenses that promote sharing and dissemination. In keeping with this spirit, we hereby provide all our materials (wherever they are already not copyrighted elsewhere as indicated) under Creative Commons Attribution-NonCommercial 4.0 International License. To view a copy of this license visit This work is ‘Open Access,’ published under a creative commons license which means that you are free to copy, distribute, display, and use the materials as long as you clearly attribute the work to the KEYSTONE course (suggested attribution: Copyright KEYSTONE Health Policy & Systems Research Initiative, Public Health Foundation of India and KEYSTONE Partners, 2015), that you do not use this work for any commercial gain in any form and that you in no way alter, transform or build on the work outside of its use in normal academic scholarship without express permission of the author and the publisher of this volume. Furthermore, for any reuse or distribution, you must make clear to others the license terms of this work. This means that you can: read and store this document free of charge distribute it for personal use free of charge print sections of the work for personal use read or use parts or whole of the work in a context where no financial transactions take place gain financially from the work in anyway sell the work or seek monies in relation to the distribution of the work use the work in any commercial activity of any kind distribute in or through a commercial body (with the exception of academic usage within educational institutions such as schools and universities However, you cannot: