Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

PhD private defence: Realist evaluation of a capacity building programme for health managers in Tumkur, India


Published on

My PhD private defence on realist evaluation of health managers capacity building programme examining scope for organisational change in public health services in a district setting in southern India. A less technical version from the public defence is here:
More details at

Published in: Healthcare
  • Be the first to comment

  • Be the first to like this

PhD private defence: Realist evaluation of a capacity building programme for health managers in Tumkur, India

  1. 1. How do capacity building programmes work in local health systems? A realist evaluation of a local health system strengthening intervention in Tumkur, India Prashanth Nuggehalli Srinivas Private defense UCL February 20, 2015 Promoter: Jean Macq Co-promoter: Bart Criel
  2. 2. Outline • Part 1: The big picture – Strengthening health systems in India • Part 2: Local health systems and organisational change • Part 3: Study setting and intervention • Part 4: Methodology and study design • Part 5: Analysis and results • Part 6: Discussion, relevance and lessons learned 2 Background Methods Results & Discussion
  3. 3. The big picture Strengthening health systems in India 3 Part 1
  4. 4. Indian health system is pluralistic & evolving 4 Van Damme et al 2010 Part 1 Background
  5. 5. Regional improvements, but disparities remain 5 “Accelerated progress to reduce mortality during the neonatal period and at ages 1–59 months is needed in most Indian districts.” - Usha Ram et. al. 2013 Part 1 Background
  6. 6. Regional/sub-regional (district-taluka) disparities Role of (poor) management? For example, in 2006 Immunisation coverage – 91% in Kodagu district and 70% in Raichur. 114 “backward” talukas, nearly half in “forward” districts “Systemic failure” as a cause? (George, 2007&2009), Sen (2006) 6 Part 1 Background
  7. 7. Human resources for health • Good health workforce – Available & competent • Improved organisational outcomes through ‘good HRM’ – Lifelong learning and supportive (yet firm) supervision – Competent and responsive managers who are able to manage resources and plan health care services Part 1 Background
  8. 8. The National (Rural) Health Mission 8 Part 1 Background
  9. 9. • Poor planning and management contribute to disparate health outcomes in Indian districts • Structural reforms such as the NRHM need well-performing district & sub-district local health systems Prashanth N S (2013) BMJ Rapid Response 9
  10. 10. Understanding local health system performance 10
  11. 11. Local health systems • More than a sum of the services; acquire a specific local character in view of their internal characteristics and the context • Interface between top-down policies and bottom-up demands • Locus for conceptualising organisational change through building capacity and improving performance Van Olmen et. al. 2012 11 Part 2 Background
  12. 12. Capacity and performance • Multi-dimensional nature of capacity & performance – individual, organisational, environmental • Capacity & capacity building closely related to performance, but may not automatically improve performance – various dynamic interactions between internal and external factors influence performance 12 Part 2 Background Brown, L., LaFond, A., & Macintyre, K. (2001). Measuring capacity building. Chapel Hill: MEASURE Evaluation.
  13. 13. Socio-cultural Effort Time Culture-oriented change & “new way of doing things” (shifting norms, powers, values) 13 Technical Programmatic Quick(er) Tangible Task-oriented nature of change (changing procedures and activities) Potter & Brough 2004 Organisational change Part 2
  14. 14. Capacity-building as an HRM intervention • Implemented with the objective of knowledge or skills transfer through training programmes • Frequent calls for greater capacity-building in literature and some studies on effectiveness, but: – How do these programmes work at the “systems” level? – Under what circumstances do these lead to behavioural change and improved performance of the organisations? – What are the contextual elements that promote (or hinder) such change? 14 Part 2 Background
  15. 15. Rationale for the study • The literature gap – Review of 28 European Commission-funded projects shows need for systemic capacity building & research; and scanty literature on how it works (Potter & Borough 2005) • The evidence gap – poor evidence for structuring capacity-building interventions (Rowe et al 2005) • The methodological gap – evaluation of complex HRM interventions • Timeliness & relevance – National Rural Health Mission 15 Part 2 Background
  16. 16. Asking the how question in healthcare evaluations in India • Review of health programme evaluation in India • Little inter-disciplinarity • Heavy tilt towards “did programme work” and comparing coverage and effectiveness • Two case studies – maternity benefit scheme & health insurance for people below poverty line Prashanth, N. S., Marchal, B., & Criel, B. (2013). Evaluating Healthcare Interventions: Answering the “How” Question. Indian Anthropologist, 43(1), 35–50. 16 Part 2
  17. 17. Study setting and intervention 17
  18. 18. 18 Public health services organisation in India Part 3 Background
  19. 19. 19 Part 3 Background Delphi study by IPH, Bangalore on poor performance of district health services (2007)
  20. 20. Study setting 20 Part 3 Background
  21. 21. 21
  22. 22. 22 Mentoring Contact classes & Assignment Health managers (medical & non- medical) at District level – DHO, DS, Programme officers, DPM, nursing managers and senior admin staff Health team at taluka level – THO, AMO, BPM, AAO PHC Medical Officers, PRI members 2-3 days per month, residential contact classes At least 5 mentoring days 1 assignment/month
  23. 23. Methodology and study design 23
  24. 24. Methodological considerations • Programme theory and assumptions were not explicitly formulated • From effectiveness to mechanisms of change in organisations • Mixed methods study 24 Part 4 Methods
  25. 25. Scope for realist evaluation Prashanth, N. S., Marchal, B., & Criel, B. (2013). Evaluating Healthcare Interventions: Answering the “How” Question. Indian Anthropologist, 43(1), 35–50. 25 Part 4 Methods
  26. 26. Realist approach Mechanism: what is it about an intervention which may lead it to have a particular outcome in a given context? Context: what conditions are needed for an intervention to trigger mechanisms to produce particular outcomes patterns? Outcomes pattern: what are the practical effects produced by causal mechanisms being triggered in a given context? Pawson & Tilley, 199726 Part 4 Methods
  27. 27. The realist cycle 27 Part 4 Methods
  28. 28. Three cycles • Eliciting the PT • Macro/meso level contextual conditions • Contrasting cases within Tumkur 28 Part 4 Methods
  29. 29. Data collection • Field notes of observations during classroom teaching, mentoring visits, district and taluka review meetings and supervision visits • Interview with participants (7+7+8), supervisors (2), state-level bureaucrats (2) and implementers (2) in three episodes: early intervention, mid and post • Secondary data: annual plans, district-planning guidelines from state and central government, programme documents of the NRHM 29 Part 4 Methods
  30. 30. Survey – Attitude towards decentralised planning and training programmes – Organisational commitment (Mayer & Allen) – Self-efficacy (Bandura) – Supervision (Oldham & Cummings as adapted from Michigan OA package) – Respondents (Tumkur and a comparator district): 65+27 30 Part 4 Methods
  31. 31. 31 Part 4
  32. 32. Eliciting the programme theory (PT) • Described the process of refining PT – Understanding the intervention (IPT) – Review of literature to identify mechanisms reported – Identify relevant contextual factors – Refine PT – Formulate change scenarios (C-M-O) 32 Part 4 Methods
  33. 33. Key IPT assumption Supporting theory Key contextual factor Plausible mechanism identifiable from IPT and theory Outcome of interest Contact classes’ work through improving knowledge and/or skills, which are eventually applied. This results in improved performance Outcomes of training programmes accrue through four hierarchical levels: reaction (to training programme), learning, behaviour and impact (Kirkpatrick and Kirkpatrick 1998) Team dynamics (nature of team and relationship s) affects the individual with intention for positive change Motivation of the participant towards positive organisational change - a “can-do” attitude in the IPT Intention to make positive changes Context-mechanism-outcome 33
  34. 34. Key IPT assumption Supporting theory Key contextual factor Plausible mechanism identifiable from IPT and theory Outcome of interest Mentoring participants at workplace facilitates application of knowledge and skills Workplace environment in healthcare organisations has been identified as an important element that explains application of learning from training programmes in some settings, while not in others (Clarke 2005). Nature of supervision and district’s openness to “allow” change Nature of commitment to organisation Identify/seek opportunities to make positive change in the organisation’s performance Decentralised action plans and decision-making at district and lower levels. State and higher levels’ openness to change proposals Self-efficacy Improved annual action plans – Better situation analysis, problem identification, allocation and utilisation of resources 34 Context-mechanism-outcome
  35. 35. Key IPT assumption Supporting theory Key contextual factor Plausible mechanism identifiable from IPT and theory Outcome of interest A capacitate d health manager can become an agent of positive organisatio nal change High commitment managemen t literature shows the potential for change by committed staff in settings where resources could be mobilised (Marchal, Dedzo, and Kegels 2010a). Change proposals by districts are in line with state (or central) vision as well as address local needs. (Allocation and strategic alignment with external environment per Sicotte et al.’s conceptual framework)(Sico tte et al. 1998) Claiming and utilising decision spaces; organisational commitment and self- efficacy in negotiating with superiors and community leaders Taluka and districts plan improves. They identify more needs, mobilise more resources from state and utilise it better (Efficiency – both allocative and technical – improves) 35 Context-mechanism-outcome
  36. 36. Analysis and results 36
  37. 37. Elicited PT - 1 Contact classes could work through commitment and efficacy of health managers who bear an intention to make positive change by providing them resources in the form of knowledge and/or skills; they are likely to apply these knowledge and skills in talukas where local team environment supports such change and the change agenda aligns with the local PRI and district/state expectations 37 Results & Discussion Part 5
  38. 38. Mentored participants are more likely to seek opportunities to improve their local health systems to make positive change in the organisation’s performance wherever there is no hindrance (or there is an alignment) to such moves either from above or from the PRI/community structures Elicited PT - 2 38 Results & Discussion Part 5
  39. 39. Local health systems could be improved in decentralising health systems if teams have the ability to negotiate with various actors about their change proposals and if they claim decision-spaces for preparation and implementation of action plans and local decision-making at district and lower levels; if the capacity building programme could work at multiple levels to ensure better alignments between opposing elements across various actors and levels in the health system. Elicited PT - 3 39 Results & Discussion Part 5
  40. 40. 40 Results & Discussion Part 5
  41. 41. Macro/meso contextual factors 41 Part 5
  42. 42. Perceptions were aligned 42 Part 5
  43. 43. Receptive to technical guidance 43 Part 5
  44. 44. But structural problems… 44 The NRHM appointed “managers” were contractual appointees within teams with very senior clinically trained doctor-managers Results & Discussion Part 5
  45. 45. “What is the use of putting my time into the PIP, if they will change it anyway at the state (level)?” a district level health manager “They seemed to make more noise than usual” a senior state-level official 45 “At village level they do not really know much planning. They are actually not bothered about plans and all.” a taluka health manager “What do they know? After all, many of them are uneducated? What is the need for them to oversee our decisions?” a taluka health manager “BPMs should provide data as and when required and prepare good reports. They are too young and cannot understand the health department’s work.” a taluka health manager Part 5 Perceptions across the health bureaucracy
  46. 46. 46 Narrow perceived decision-spaces …in spite of NRHM’s on-paper decentralised planning and management since 2005 Part 5
  47. 47. 47 Results & Discussion Part 5
  48. 48. Case analysis The hypothetical CMO frames offer a context- sensitive, theory-informed lens to analyse the intervention – In purposively chosen talukas with and without a positive outcome (relate-able to the intervention), what were the differing contexts? – What were the differences in the nature of commitment of the individuals in these contrasting talukas? – ..… 48 Results & Discussion Part 5
  49. 49. Explaining organisational change • Identified case studies based on diversity of context and/or outcome after scanning context, mechanism and outcome elements • Confronted the reformulated PT and first round of CMO- based change scenarios to these cases 49 Results & Discussion Part 5
  50. 50. Case selection • a mix of individual, organisational and contextual factors – intervention exposure – socio-economic development index of taluka – mentoring interest & supervision received – stability of team – proxy measures of outcomes logically related to improvements in the talukas. 50 Results & Discussion Part 5
  51. 51. Degree of classroo m participat ion (attendan ce and classroo m activity) (0-1.0) Degree of mentorin g received (0-1.0) Rete ntion of ment or inter est by taluk a High - Mod erate -Low Organisa tional commitm ent Affective commitm ent(AC), normativ e commitm ent (NC) & continua nce commitm ent (CC) (0-5) Self- effica cy (0- 100) Support ive degree of supervi sion supervi sion (1-5; 1 being most support ive and 5 being most authorit ative) Perce ntage of ever- traine d memb ers who expres sed intenti on to make chang e Stabi lity of team – turnv over (Hig h- Mod erate - low) Devel opme nt index Net chang e in percen tage budge t utiliza tion (2008- 2012) Net change in proport ion of LSCS among total deliveri es (2008- 2012) Ne t cha ng e in stil lbi th rat e (20 08 20 12) Gubbi 0.7 0.7 High AC 2.66 NC 2.47 CC 2.42 68 2.5 50 Mod erate 0.95 2 1 -16 Tumkur 0.7 0.7 Mod erate AC 2.85 NC 2.46 CC 2.69 68 2.6 75 Low 1.21 6 1.5 -8 CN Halli 0.6 0.5 Mod erate AC 2.75 NC 2.29 CC 2.71 70 2.2 100 High 1.02 4 0.1 0 Turuvekere 0.6 0.4 Low AC 2.81 NC 2.80 CC 2.47 68 2.4 83 High 1.06 5 5.8 -4 Tiptur 0.5 0.5 Mod erate AC 2.25 NC 2.33 CC 3.17 86 2.5 75 Low 1.25 -4 12.6 -1 Koratagere 0.4 0.5 Low AC 2.87 NC 2.73 CC 3.07 71 2.3 20 Mod erate 0.89 3 1.8 -3 Madhugiri 0.5 0.5 Low AC 2.50 NC 2.03 CC 2.50 83 2.4 40 High 0.82 4 1.3 -1 Pavagada 0.6 0.5 Mod erate AC 2.50 NC 2.05 CC 2.28 79 2.3 0 High 0.78 6 0 1 Kunigal 0.6 0.5 High AC 2.12 NC 2.59 CC 2.83 83 2.2 75 Mod erate 0.96 2 4.9 -4 Sira 0.7 0.9 High AC 1.80 NC 2.00 CC 2.67 68 2.2 100 Mod erate 0.81 6 8.3 2 51
  52. 52. Committed and mentored teams with low- moderate intention to make change “In my taluka for example, I think we can make big change. It is not that everybody in my taluka want to make changes. Only one-third of them are motivated to make changes. And that is enough. I think I can make a lot of improvement by motivating these people.” - a Gubbi taluka helath manager “More resources mean more opportunities to make change. If they slowly give more and more power to us at taluka level, we can make many more improvements. Right now, very little is possible at taluka level. “ - another taluka health manager from Gubbi (g2) 52 Results & Discussion Part 5 Committed health management teams could utilise new opportunities for organisational improvement presented by decentralising health systems wherever their change agenda aligns with the expectations of higher levels of the bureaucracy.
  53. 53. 53 What PIP? What decentralisation? I sent so many requirements for staff and proposals for improvement. Only thing I got is more work, less staff and zero solutions. On one hand, I have to answer the local ZP members’ complaints and on the other hand, I have to just keep implementing plans and schemes coming from above. Nothing can be done without more staff. - a health manager from CN Halli (cnh1) We felt that we have to do it. So many mothers were just being referred to Tumkur. The delivery load is high and for several months, we had only one obstetrician, but somehow we managed. I know how the pressure is at the distict hospital, so having LSCS facility at Sira decreases the burden at the district hospital. It’s not easy, but somehow it is happening. - a Sira health manager (s1) “Nothing much can be done without giving powers at taluka level and PHCs. I cannot even appoint a Group D staff. Where is decentralisation in this?” - a PHC staff from CN Halli taluka Tapping commitment for organisational change could be frustrating in low-resource local health systems where health managers working in poorly resourced talukas, in spite of their improved management capacities and intentions to make change, could get frustrated by the lack of facilitating action from above. Poorly resourced teams with varying commitment levels/types & high intention for change
  54. 54. Discussion, relevance and lessons learned 54 Results & Discussion Results & Discussion Part 6
  55. 55. Synthesis 55 Results & Discussion Part 6
  56. 56. Lessons learned – capacity building • Capacity building programmes seek to influence health manager decisions and choices: capacity to manage alignments matter, not only determinants • Pushing public health service organisations towards change in decentralising health organisations: need to engage with multiple levels in the bureaucracy • Capacity building strategies need to invest more in local goal-setting and negotiation and coping skills of health managers, and not entirely focus on knowledge/skill transfer • Capacity building programmes could seek to become the context for change through facilitating a desire for change (or harness pre-existing feeling of unhappiness with the current status) 56 Results & Discussion Part 6
  57. 57. • Application of insights from organisational sciences and social sciences in health systems strengthening • Using PT refinement and realist evaluation as an operational tool for implementation • Need for more case diversity and further iterations could improve the final refined theory Lessons learned - methods 57 Results & Discussion Part 6
  58. 58. Relevance • Building a human resources management strategy for improved district health system functioning in Karnataka • Improving the design of existing state and district level capacity building efforts • Teaching material for teaching organisational change approaches within health servcies and for teaching theory- driven and realist approaches for evaluating healthcare interentions 58 Results & Discussion Part 6
  59. 59. Thank you 59
  60. 60. Explaining organisational change 60
  61. 61. Dimensions of organisational commitment measures (AC, NC & CC – Meyer & Allen) by taluka 61 Results & Discussion