My PhD private defence on realist evaluation of health managers capacity building programme examining scope for organisational change in public health services in a district setting in southern India. A less technical version from the public defence is here: http://www.slideshare.net/PrashanthSrinivas/public-defence-realist-evaluation-of-capacity-building-programme-of-health-managers-in-tumkur-india More details at http://www.daktre.com/2015/05/studying-organisational-change-in-indian-district-health-systems

1. How do capacity building programmes work
in local health systems?
A realist evaluation of a local health system strengthening intervention
in Tumkur, India
Prashanth Nuggehalli Srinivas
Private defense UCL
February 20, 2015
Promoter: Jean Macq
Co-promoter: Bart Criel

2. Outline
• Part 1: The big picture – Strengthening health
systems in India
• Part 2: Local health systems and
organisational change
• Part 3: Study setting and intervention
• Part 4: Methodology and study design
• Part 5: Analysis and results
• Part 6: Discussion, relevance and lessons
learned
2
Background
Methods
Results &
Discussion

3. The big picture
Strengthening health systems in India
3
Part 1

4. Indian health system is pluralistic & evolving
4
Van Damme et al 2010
Part 1
Background

5. Regional improvements, but disparities remain
5
“Accelerated progress to reduce mortality during the neonatal
period and at ages 1–59 months is needed in most Indian
districts.”
- Usha Ram et. al. 2013
Part 1
Background

6. Regional/sub-regional (district-taluka) disparities
Role of (poor) management?
For example, in 2006
Immunisation coverage
– 91% in Kodagu district
and 70% in Raichur.
114 “backward” talukas,
nearly half in “forward”
districts
“Systemic failure” as a
cause?
(George, 2007&2009),
Sen (2006)
6
Part 1
Background

7. Human resources for health
• Good health workforce –
Available & competent
• Improved organisational
outcomes through ‘good HRM’
– Lifelong learning and supportive
(yet firm) supervision
– Competent and responsive
managers who are able to manage
resources and plan health care
services
Part 1
Background

8. The National (Rural) Health Mission
8
Part 1
Background

9. • Poor planning and management
contribute to disparate health
outcomes in Indian districts
• Structural reforms such as the
NRHM need well-performing
district & sub-district local health
systems
Prashanth N S (2013) BMJ Rapid Response
http://www.bmj.com/content/347/bmj.f5621/rr/662992 9

10. Understanding local health system performance
10

11. Local health systems
• More than a sum of the services;
acquire a specific local character
in view of their internal
characteristics and the context
• Interface between top-down
policies and bottom-up
demands
• Locus for conceptualising
organisational change through
building capacity and improving
performance
Van Olmen et. al. 2012
11
Part 2
Background

12. Capacity and performance
• Multi-dimensional nature
of capacity & performance
– individual, organisational,
environmental
• Capacity & capacity
building closely related to
performance, but may not
automatically improve
performance – various
dynamic interactions
between internal and
external factors influence
performance 12
Part 2
Background
Brown, L., LaFond, A., & Macintyre, K. (2001). Measuring capacity
building. Chapel Hill: MEASURE Evaluation.

13. Socio-cultural
Effort
Time
Culture-oriented
change & “new way of
doing things” (shifting
norms, powers,
values)
13
Technical
Programmatic
Quick(er)
Tangible
Task-oriented nature
of change (changing
procedures and
activities)
Potter & Brough 2004
Organisational change
Part 2

14. Capacity-building as an HRM
intervention
• Implemented with the objective of knowledge or
skills transfer through training programmes
• Frequent calls for greater capacity-building in
literature and some studies on effectiveness, but:
– How do these programmes work at the “systems” level?
– Under what circumstances do these lead to behavioural
change and improved performance of the organisations?
– What are the contextual elements that promote (or
hinder) such change?
14
Part 2
Background

15. Rationale for the study
• The literature gap – Review of 28 European
Commission-funded projects shows need for
systemic capacity building & research; and scanty
literature on how it works (Potter & Borough 2005)
• The evidence gap – poor evidence for structuring
capacity-building interventions (Rowe et al 2005)
• The methodological gap – evaluation of complex
HRM interventions
• Timeliness & relevance – National Rural Health
Mission
15
Part 2
Background

16. Asking the how question in healthcare evaluations in India
• Review of health programme
evaluation in India
• Little inter-disciplinarity
• Heavy tilt towards “did
programme work” and comparing
coverage and effectiveness
• Two case studies – maternity
benefit scheme & health
insurance for people below
poverty line Prashanth, N. S., Marchal, B., & Criel, B. (2013). Evaluating
Healthcare Interventions: Answering the “How” Question.
Indian Anthropologist, 43(1), 35–50.
16
Part 2

17. Study setting and intervention
17

18. 18
Public health services
organisation in India
Part 3
Background

19. 19
Part 3
Background
Delphi study by IPH,
Bangalore on poor
performance of
district health
services (2007)

20. Study setting
20
Part 3
Background

21. 21

22. 22
Mentoring
Contact classes
& Assignment
Health managers (medical & non-
medical) at District level – DHO, DS,
Programme officers, DPM, nursing
managers and senior admin staff
Health team at taluka level – THO,
AMO, BPM, AAO
PHC Medical Officers, PRI members
2-3 days per month,
residential contact
classes
At least 5 mentoring
days
1 assignment/month

23. Methodology and study design
23

24. Methodological considerations
• Programme theory and assumptions were
not explicitly formulated
• From effectiveness to mechanisms of
change in organisations
• Mixed methods study
24
Part 4
Methods

25. Scope for realist evaluation
Prashanth, N. S., Marchal, B., & Criel, B. (2013). Evaluating
Healthcare Interventions: Answering the “How” Question.
Indian Anthropologist, 43(1), 35–50.
25
Part 4
Methods

26. Realist approach
Mechanism: what is it about an
intervention which may lead it to
have a particular outcome in a
given context?
Context: what conditions are
needed for an intervention to
trigger mechanisms to produce
particular outcomes patterns?
Outcomes pattern: what are the
practical effects produced by
causal mechanisms being
triggered in a given context?
Pawson & Tilley, 199726
Part 4
Methods

27. The realist cycle
27
Part 4
Methods

28. Three cycles
• Eliciting the PT
• Macro/meso level
contextual
conditions
• Contrasting cases
within Tumkur
28
Part 4
Methods

29. Data collection
• Field notes of observations during classroom
teaching, mentoring visits, district and taluka
review meetings and supervision visits
• Interview with participants (7+7+8),
supervisors (2), state-level bureaucrats (2) and
implementers (2) in three episodes: early
intervention, mid and post
• Secondary data: annual plans, district-planning
guidelines from state and central government,
programme documents of the NRHM
29
Part 4
Methods

30. Survey
– Attitude towards decentralised planning and
training programmes
– Organisational commitment (Mayer & Allen)
– Self-efficacy (Bandura)
– Supervision (Oldham & Cummings as adapted
from Michigan OA package)
– Respondents (Tumkur and a comparator district):
65+27
30
Part 4
Methods

31. 31
Part 4

32. Eliciting the programme theory (PT)
• Described the process of
refining PT
– Understanding the
intervention (IPT)
– Review of literature to
identify mechanisms
reported
– Identify relevant contextual
factors
– Refine PT
– Formulate change scenarios
(C-M-O)
32
Part 4
Methods

33. Key IPT
assumption
Supporting
theory
Key
contextual
factor
Plausible
mechanism
identifiable
from IPT and
theory
Outcome
of interest
Contact
classes’
work
through
improving
knowledge
and/or skills,
which are
eventually
applied. This
results in
improved
performance
Outcomes of
training
programmes
accrue through
four
hierarchical
levels: reaction
(to training
programme),
learning,
behaviour and
impact
(Kirkpatrick
and Kirkpatrick
1998)
Team
dynamics
(nature of
team and
relationship
s) affects
the
individual
with
intention
for positive
change
Motivation of
the
participant
towards
positive
organisational
change - a
“can-do”
attitude in
the IPT
Intention
to make
positive
changes
Context-mechanism-outcome 33

34. Key IPT
assumption
Supporting
theory
Key contextual
factor
Plausible
mechanism
identifiable
from IPT
and theory
Outcome of
interest
Mentoring
participants
at workplace
facilitates
application
of
knowledge
and skills
Workplace
environment in
healthcare
organisations has
been identified
as an important
element that
explains
application of
learning from
training
programmes in
some settings,
while not in
others (Clarke
2005).
Nature of
supervision and
district’s
openness to
“allow” change
Nature of
commitment
to
organisation
Identify/seek
opportunities
to make
positive
change in the
organisation’s
performance
Decentralised
action plans and
decision-making
at district and
lower levels.
State and higher
levels’ openness
to change
proposals
Self-efficacy Improved
annual action
plans – Better
situation
analysis,
problem
identification,
allocation
and
utilisation of
resources 34
Context-mechanism-outcome

35. Key IPT
assumption
Supporting
theory
Key contextual
factor
Plausible
mechanism
identifiable from
IPT and theory
Outcome of
interest
A
capacitate
d health
manager
can
become an
agent of
positive
organisatio
nal change
High
commitment
managemen
t literature
shows the
potential for
change by
committed
staff in
settings
where
resources
could be
mobilised
(Marchal,
Dedzo, and
Kegels
2010a).
Change
proposals by
districts are in
line with state
(or central)
vision as well as
address local
needs.
(Allocation and
strategic
alignment with
external
environment
per Sicotte et
al.’s conceptual
framework)(Sico
tte et al. 1998)
Claiming and
utilising
decision
spaces;
organisational
commitment
and self-
efficacy in
negotiating
with superiors
and
community
leaders
Taluka and
districts plan
improves.
They identify
more needs,
mobilise
more
resources
from state
and utilise it
better
(Efficiency –
both
allocative and
technical –
improves)
35
Context-mechanism-outcome

36. Analysis and results
36

37. Elicited PT - 1
Contact classes could work through
commitment and efficacy of health
managers who bear an intention to make
positive change by providing them
resources in the form of knowledge and/or
skills; they are likely to apply these
knowledge and skills in talukas where local
team environment supports such change
and the change agenda aligns with the local
PRI and district/state expectations
37
Results &
Discussion
Part 5

38. Mentored participants are more likely to seek
opportunities to improve their local health
systems to make positive change in the
organisation’s performance wherever there is no
hindrance (or there is an alignment) to such
moves either from above or from the
PRI/community structures
Elicited PT - 2
38
Results &
Discussion
Part 5

39. Local health systems could be improved in
decentralising health systems if teams have the
ability to negotiate with various actors about their
change proposals and if they claim decision-spaces
for preparation and implementation of action
plans and local decision-making at district and
lower levels; if the capacity building programme
could work at multiple levels to ensure better
alignments between opposing elements across
various actors and levels in the health system.
Elicited PT - 3
39
Results &
Discussion
Part 5

40. 40
Results &
Discussion
Part 5

41. Macro/meso contextual factors
41
Part 5

42. Perceptions were aligned
42
Part 5

43. Receptive to technical guidance
43
Part 5

44. But structural problems…
44
The NRHM appointed “managers” were contractual appointees
within teams with very senior clinically trained doctor-managers
Results &
Discussion
Part 5

45. “What is the use of putting my time into
the PIP, if they will change it anyway at the
state (level)?”
a district level health manager
“They seemed to make more noise than
usual”
a senior state-level official
45
“At village level they do not really know
much planning. They are actually not
bothered about plans and all.”
a taluka health manager
“What do they know? After all, many of
them are uneducated? What is the need
for them to oversee our decisions?”
a taluka health manager
“BPMs should provide data as and when
required and prepare good reports. They
are too young and cannot understand the
health department’s work.”
a taluka health manager
Part 5
Perceptions across the health bureaucracy

46. 46
Narrow perceived decision-spaces
…in spite of NRHM’s on-paper decentralised planning and management since 2005
Part 5

47. 47
Results &
Discussion
Part 5

48. Case analysis
The hypothetical CMO frames offer a context-
sensitive, theory-informed lens to analyse the
intervention
– In purposively chosen talukas with and without a
positive outcome (relate-able to the
intervention), what were the differing contexts?
– What were the differences in the nature of
commitment of the individuals in these
contrasting talukas?
– ..…
48
Results &
Discussion
Part 5

49. Explaining organisational change
• Identified case studies based
on diversity of context
and/or outcome after
scanning context,
mechanism and outcome
elements
• Confronted the reformulated
PT and first round of CMO-
based change scenarios to
these cases
49
Results &
Discussion
Part 5

50. Case selection
• a mix of individual, organisational and
contextual factors
– intervention exposure
– socio-economic development index of taluka
– mentoring interest & supervision received
– stability of team
– proxy measures of outcomes logically related
to improvements in the talukas.
50
Results &
Discussion
Part 5

51. Degree
of
classroo
m
participat
ion
(attendan
ce and
classroo
m
activity)
(0-1.0)
Degree
of
mentorin
g
received
(0-1.0)
Rete
ntion
of
ment
or
inter
est
by
taluk
a
High
-
Mod
erate
-Low
Organisa
tional
commitm
ent
Affective
commitm
ent(AC),
normativ
e
commitm
ent (NC)
&
continua
nce
commitm
ent (CC)
(0-5)
Self-
effica
cy
(0-
100)
Support
ive
degree
of
supervi
sion
supervi
sion
(1-5; 1
being
most
support
ive and
5 being
most
authorit
ative)
Perce
ntage
of
ever-
traine
d
memb
ers
who
expres
sed
intenti
on to
make
chang
e
Stabi
lity
of
team
–
turnv
over
(Hig
h-
Mod
erate
-
low)
Devel
opme
nt
index
Net
chang
e in
percen
tage
budge
t
utiliza
tion
(2008-
2012)
Net
change
in
proport
ion of
LSCS
among
total
deliveri
es
(2008-
2012)
Ne
t
cha
ng
e
in
stil
lbi
th
rat
e
(20
08
20
12)
Gubbi 0.7 0.7 High AC 2.66
NC 2.47
CC 2.42
68 2.5 50 Mod
erate
0.95 2 1 -16
Tumkur 0.7 0.7 Mod
erate
AC 2.85
NC 2.46
CC 2.69
68 2.6 75 Low 1.21 6 1.5 -8
CN Halli 0.6 0.5 Mod
erate
AC 2.75
NC 2.29
CC 2.71
70 2.2 100 High 1.02 4 0.1 0
Turuvekere 0.6 0.4 Low AC 2.81
NC 2.80
CC 2.47
68 2.4 83 High 1.06 5 5.8 -4
Tiptur 0.5 0.5 Mod
erate
AC 2.25
NC 2.33
CC 3.17
86 2.5 75 Low 1.25 -4 12.6 -1
Koratagere 0.4 0.5 Low AC 2.87
NC 2.73
CC 3.07
71 2.3 20 Mod
erate
0.89 3 1.8 -3
Madhugiri 0.5 0.5 Low AC 2.50
NC 2.03
CC 2.50
83 2.4 40 High 0.82 4 1.3 -1
Pavagada 0.6 0.5 Mod
erate
AC 2.50
NC 2.05
CC 2.28
79 2.3 0 High 0.78 6 0 1
Kunigal 0.6 0.5 High AC 2.12
NC 2.59
CC 2.83
83 2.2 75 Mod
erate
0.96 2 4.9 -4
Sira 0.7 0.9 High AC 1.80
NC 2.00
CC 2.67
68 2.2 100 Mod
erate
0.81 6 8.3 2
51

52. Committed and mentored teams with low-
moderate intention to make change
“In my taluka for example, I think we can
make big change. It is not that everybody
in my taluka want to make changes. Only
one-third of them are motivated to make
changes. And that is enough. I think I can
make a lot of improvement by motivating
these people.”
- a Gubbi taluka helath manager
“More resources mean more
opportunities to make change. If they
slowly give more and more power to us
at taluka level, we can make many more
improvements. Right now, very little is
possible at taluka level. “
- another taluka health manager from
Gubbi (g2)
52
Results &
Discussion
Part 5
Committed health management teams could
utilise new opportunities for organisational
improvement presented by decentralising
health systems wherever their change agenda
aligns with the expectations of higher levels of
the bureaucracy.

53. 53
What PIP? What decentralisation? I sent
so many requirements for staff and
proposals for improvement. Only thing I
got is more work, less staff and zero
solutions. On one hand, I have to answer
the local ZP members’ complaints and on
the other hand, I have to just keep
implementing plans and schemes coming
from above. Nothing can be done without
more staff.
- a health manager from CN Halli (cnh1)
We felt that we have to do it. So many
mothers were just being referred to Tumkur.
The delivery load is high and for several
months, we had only one obstetrician, but
somehow we managed. I know how the
pressure is at the distict hospital, so having
LSCS facility at Sira decreases the burden at
the district hospital. It’s not easy, but
somehow it is happening.
- a Sira health manager (s1)
“Nothing much can be done without
giving powers at taluka level and
PHCs. I cannot even appoint a Group
D staff. Where is decentralisation in
this?”
- a PHC staff from CN Halli taluka
Tapping commitment for organisational change
could be frustrating in low-resource local
health systems where health managers
working in poorly resourced talukas, in spite of
their improved management capacities and
intentions to make change, could get
frustrated by the lack of facilitating action from
above.
Poorly resourced teams with varying commitment
levels/types & high intention for change

54. Discussion, relevance and lessons learned
54
Results &
Discussion
Results &
Discussion
Part 6

55. Synthesis
55
Results &
Discussion
Part 6

56. Lessons learned – capacity building
• Capacity building programmes seek to influence health
manager decisions and choices: capacity to manage
alignments matter, not only determinants
• Pushing public health service organisations towards change
in decentralising health organisations: need to engage with
multiple levels in the bureaucracy
• Capacity building strategies need to invest more in local
goal-setting and negotiation and coping skills of health
managers, and not entirely focus on knowledge/skill
transfer
• Capacity building programmes could seek to become the
context for change through facilitating a desire for change
(or harness pre-existing feeling of unhappiness with the
current status) 56
Results &
Discussion
Part 6

57. • Application of insights from organisational sciences
and social sciences in health systems strengthening
• Using PT refinement and realist evaluation as an
operational tool for implementation
• Need for more case diversity and further iterations
could improve the final refined theory
Lessons learned - methods
57
Results &
Discussion
Part 6

58. Relevance
• Building a human resources
management strategy for
improved district health system
functioning in Karnataka
• Improving the design of existing
state and district level capacity
building efforts
• Teaching material for teaching
organisational change
approaches within health
servcies and for teaching theory-
driven and realist approaches for
evaluating healthcare
interentions
58
Results &
Discussion
Part 6

59. Thank you
59

60. Explaining organisational change
60

61. Dimensions of organisational commitment measures (AC, NC & CC – Meyer & Allen) by taluka
61
Results &
Discussion