Feb 3, 2016 Dr. John Ovretveit, Director of Research and Professor of Health Innovation and Evaluation at the Karolinska Institutet, presented as part of a seminar series on UCLA CTSI Dissemination, Improvement and Implementation Research.

1. Faster Improvement with
Adaptive Implementation
Research
1
John Øvretveit,
Director of Research, Professor of Health Innovation and
Evaluation, Karolinska Institutet, Stockholm, Sweden
2/3/2016

2. Help with these challenges
 Changing clinical practice and service delivery
 eg use this test in these situations not these, establish team-based service
 Effective methods for change under-used
 by improvement projects
 Frustration of practical improvers
 research not answering their questions
 how do we implement this? costs and savings? conditions we need to get the same
results?
 Researcher’s challenges
 reducing time and cost;
 Time and funding to publish
 Role for working on implementation – for trial or spread – time and funding
 documenting changes;
 attributing outcomes (esp which part most impactful);
 generalising; take-up of our research
22/3/2016

3. Does not help with …
 Changing academic research criteria
 Not improver’s questions
 Do not reward “3R research” relevant, responsive,
rigorous
 Rather “3P review criteria” for proposals, publications
and promotion, using RCT, ideally
 Settings where implementation is not
possible
 but will know why
3

4. Relevance to researchers
..if evaluating an intervention
 (e.g. new treatment or service delivery model)
.. knowledge of non-experimental methods
 which data to gather to document and evaluate
..designing effective intervention
(eg protocol for establishing an intervention to test)
 Issues in translating research into practice and
scale-up – researcher’s role?
42/3/2016

5. It was effective there… but different here…?
52/3/2016
We don’t have a
problem here
Our patients/service
is different
We have other changes
and priorities

6. Session covers
 Case example - problems
 Proven solutions – why not implemented?
 “Fit” of Solution in the “Context”
 Does the adaption work?
Adaptive implementation research
 Research to answer improver’s questions
 Implications for you
62/3/2016

7. What I mean by…
“Intervention” covers both
a)Do hand hygiene consistently (before/after change)
b) Training and feedback (actions to enable change)
Words
a) the “new better way” - hand hygiene
 eg wash hands between patients; use bar code reader to reduce medication errors; rapid response
team to prevent avoidable deterioration on nursing units. (The Improvement-change)
b) What we do directly to enable “take up” by staff of the
new better way
(“implementation” methods - training, feedback, rewards, punishment for not using, etc.)
72/3/2016

8. .
82/3/2016
B
Better hand
hygiene
A
Poor hand
hygiene
Intervention Plan
Wash hands between
patients
Reduce barriers
Feedback
Implementation actions (phases)
1Education
2Gell dispensers
3Patient expectations
4Feedback compliance (infection
rates)
How?
Surrounding “context” helps and hinders
Intervention
concept
Wash hands between
patients
Reduce barriers
Feedback

9. What I mean by…
 Conditions or context for the intervention:
= influences which indirectly help or hinder the
intervention
 high workload affects hand hygiene,
 disruption to supplies,
 IT system,
 how the hospital or physicians are paid
92/3/2016

10. 2/3/2016 10
Mary: 84 yrs. Obstructive airways (COPD),
heart disease, mild depression
Stable at home on meds,
very independent
Unpaid motivational coach
and security-guard - “Matty”

11. Healthcare experience
 Emergency hospital admission
 Delays, and meds changed
 HAI
 Sent home – no comms or support
 Readmission
 Before/after
1
12/3/2016

12. Mary - six weeks later
 After hospitalisation
 Avoidable cost 4600$
1
22/3/2016

13. Name a proven interventions for any problems?
Changes of meds/wrong meds?
Hospital acquired infection?
After care transition?
 Medication reconcilliation
 Hand hygiene - bundle of interventions
 After-care - Coleman CTI – Transition coach and
education
1
32/3/2016

14. Your experience?
We have implemented solutions to prevent this
- Yes, No, sometimes
 We have effective Med Rec - Medications changes
unlikely
 HAI from Hand Hygiene non-compliance unlikely
 After care information to PCP ensured within 2
days of discharge
 Effective after-care support for older multiple
morbidity patients
1
42/3/2016

15. Why not implemented? - your vote
 No problem (one-off event)
 Not aware of these proven effective solutions
 Know how to implement - don’t have time or
resources to implement
 To implement effectively - don’t have knowledge
and skills
 Know solutions - uncertain if we would get same
results - would it work here?
 Don’t have research to show how to implement in
different settings
1
5

16. Johns Quest…
 Why not implemented?
 What would help fast & widespread
implementation?
 How can research best help?
1
62/3/2016
Conclusions so far…
…in 2 slides

17. 2/3/2016 17
Success depends on…
Seed Gardener/planting & nurture Soil / climate
Idea 10%
Adaption/Implementation 30%
Personalities 20%
The 10;20;30;40 change success rule
Soil receptive – staff
readiness
& wider Climate 40%

18. No intervention survives first contact with context
 .
1
82/3/2016
Implemented as planned?
Intervention plan

19. Adapt to survive
 Adapt the improvement change tested
elsewhere
 E.g. turn at risk patients every 4hrs not 2hrs
(Pressure ulcer intervention)
 Adapt the context
 Increase staffing by 0.5 FTE
 Both - Adapt the change & context
 Need “Goldilocks Improvement-Fit, for
Take Up” GIFTU
1
9

20. Context – innovation “Fit”
2
02/3/2016

21. Fidelity and adaptive implementation
 .
 t
2
12/3/2016
Adapt to “Fit”, to setting and subjects
Adjust over time – dynamic>>>>>
To do this-Resources, data and skills

22. Research questions…
 Is the adaptation more, or less effective?
 How can we find effective adaptations in
different settings?
 How can we do lower-cost faster research
 on questions which will help take up
2
22/3/2016

23. Research “Case” example
Improving physicians hand hygiene compliance
– most effective intervention?
Pilot study, 8 hospitals using QI methods
Identified 24 causes.
Different in each hospital
Interventions tailored to local cause
2nd study – web based tool
Collect data about your causes (from 24) and used these
interventions to fix your local causes
 8 Pilots compliance 48% / 81% ;
 Vs 174 organisations 58% / 84% (769
2
3

24. Variation reported in a UK study
 .
2
42/3/2016

25. One lesson for research from case?
 Providers supplied data
 Method enables easy local adaptation
 But adaptations made are unknown
 Effectiveness on average
 not the only knowledge needed by implementers
 Need data to explain variations
 Not implemented
 Context hinders
 Adapt successfully or unsuccessfully 2
52/3/2016

26. More informative research
Experimental Trials
 Can and should we explain high and low outcomes/
performers?
 Can we do subgroup analysis?
 Focus on non-experimental observational
studies
2
62/3/2016

27. Do you have any of these “Research
challenges”? Documenting?
 Use taxonomies of implementation methods
 Use adaptation typology
 Attributing?
 Use Implementation outcome model
 Use Logic Model/programme theory
 Generalising? – guidance
 Find which adaptations in which context get which results
 Use? (take up by practitioners)
= need Pre-study planning, to get data for,
2
7

28. Exercise
You are the head of a clinic:
 To enable physicians to follow antibiotic
prescribing guidelines…
 Which behavioral change techniques would
be most effective?
 Would you use the same in FQC PHC and
Regan paediatrics outpatients?
2
82/3/2016

29. Selection from 93 behavior change techniques (Michie2013)
1. Instruction on how to perform a behavior
2. Punishment
3. Anticipation of future rewards or removal of punishment
4. Practice with Graded tasks
5. Mental rehearsal of successful performance
6. Examine Anticipated regret
7. Feedback on behavior
8. Other(s) monitoring with awareness
9. Social comparison
10. Behavioral contract
11. Social support
12. Identification of self as role model
2
92/3/2016

30. See also Powell 2015: 73 implementation
strategies
Mixes direct interventions and indirect context
changes:
 Education
 Audit and provide feedback
 Create or change credentialing and/or licensure
standards
 Develop disincentives
 Use capitated payments
Do we need to distinguish? 3
02/3/2016

31. Document
Adaptions
1Who made the
modification?
2What was modified?
(Stirman 2009)
3
12/3/2016

32. Documenting critical context factors -
framworksCFIR (Damschroder 2009)
PHARIS (Rycroft-Malone 2002)
ORCA readiness - based on PHARIS
MUSIQ (Kaplan et al 2010) - QI
French et al 2009 (review of context measures
for evidence-based practice (EBP))
3
22/3/2016

33. Recommended practical tools
ORCA (or CFIR http://www.cfirguide.org/)
3
3

34. change readiness (HRET)
4 areas: the innovation, target audience, the organisation,
the environment,
3
4

35. Attributing – use causal chain frameworks
Theory informed Case evaluation
 .
 .
 .
3
52/3/2016

36. Programme theory models, or logic models
3
62/3/2016

37. Attributing – use causal chain frameworks
3
72/3/2016

38. Attributing – check implementation, before
later outcomes(Proctor 2012)
3
82/3/2016
Johns quest – enlightenment!

39. Summary
Challenges both implementing, and researching
implementation
One approach = Adaptive implementation research
Plan data to gather to
 Document adaptation and context
 Assess implementation outcomes
 Attribute outcomes through causal chain map
3
92/3/2016

40. Implications for you
 Trial - Experimental evaluation – who doing
this?
 Do we need to identify and explain “outliers”?
 Do we have data to do this?
 Non-experimental/observational - – who ?
(inc. natural experiments) - ob
 Plan and use existing data
 Ask a cross-section of “informed observers”
 Find out implementers needs for information
effectively to take up improvements
 Responsible for implementation during or
4
02/3/2016

41. Implications for you
 Responsibilities for implementation, during or
after your research?
“Another essential component of implementation
research involves the enhancement of readiness
through the creation of effective climate and culture
in an organization or community”
(Implementation research glossary, Brownson et al 2012)
4
12/3/2016

42. Questions to you
 Do you study implementation?
 What do you need to do better in
your implementation research?
 Anything a surprise?
 Examples or experience you have
about this? 4
22/3/2016

43. Questions posed
 Why not implemented?
 What would help fast & widespread
implementation?
 How can research best help?
 how best can implementation researchers help
practitioners to implement proven improvements
locally?
 Is it really their role to do so?
4
32/3/2016

44. .
 .
 .
 .
4
42/3/2016

45. What do you think?
Anything a surprise?
Examples or experience you have
about this?
4
52/3/2016

46. 6)
 .
4
62/3/2016

47. .
 .
 .
 .
4
72/3/2016

48. .
 .
 .
 .
4
82/3/2016

49. What do you think?
Anything a surprise?
Examples or experience you have
about this?
4
92/3/2016

50. .
Conclusions
through
Discussion 5
02/3/2016

51. Which was most surprising,interesting or useful – vote
1) .
5
1

52. .
Surprises?
Most useful?
Might not be true for us?
5
22/3/2016

53. Action:Pre-mortem
Crystal ball shows 1 year ahead
no results from the project
1)What are the most likely causes?
2)How could we have known before?
3)Possible actions we can take now?
5
3

54. Resources
5
42/3/2016

55. Frameworks for studying context – list (pictures
later) See references
 Clinicians implementing clinical research:
 PARHIS Guidance (Stetler et al 2011) and related:
 Context Assessment Index (CAI) (McCormack et al 2008) – the best
validated
 Alberta Context Tool (ACT) (Estabrooks et al 2008).
 CFIR Damschroder et al 2009, and French et al 2009
 Quality Improvement Projects : MUSIQ (Kaplan
et al 2012) or French et al 2009)
 See also ORCA readiness for change assessment
(Helfrich et al 2009).
5
52/3/2016

56. BUT, how many services can do this…
“evaluating the program and continually adapting
the program on the basis of evaluation results and
changes to the context of your organization
(e.g., changes in staff, changes to the community,
changes in the population served)
can help ensure that the program remains relevant
and addresses any potential challenges that occur
over time”
“Evaluation”? = lunchtime review <<<>>>RCT
5
6

57. Strategy Steps over time
Feb 1)Form project team
March 2)Gather initial data
April 3) etc.
57
Supports
 Systems for data
 Facilitators
2/3/2016
“3S framework” to describe “Implementation approach
Structure responsibilities; accountability reporting
Example – QI breakthrough collaborative

58. Recommended practical tools
Brach et al 2008: Will it work here?
5
8

59. French et al 2009 synthesis of 30 instruments measuring context
5
9

60. Context for EBP (PARiHS) Rycroft
 .C
6
0

61. 2/3/2016 61
Barriers analysis – the beginning of “context discussion”

62. Example 1 – adoption by clinicians of recommendation
for more effective treatment of depression
Medications & CBT for moderate depression.
 Implementation strategy: Simple guidelines, training, feedback
on outcomes, access to experts.
 Study found
 patient outcomes 10% improved.
 Guess % physicians following guidelines?
 40% not following guidelines
 “Implementation fidelity not met”
 practice changed in some but not fully, and in many, not at all
 Phase 2 found barriers to change
 consultation time extra, and opinion leader “lukewarm”
6
22/3/2016

63. 2)Barriers to take up
 the lack of information technology systems;
 physician culture,
 beliefs and habits;
 development & function of guideline
 (Kenefick, et al 2008)
6
32/3/2016

64. Example studies of implementation strategies
Specialist-nurse led clinics for implementing lipid control cost-
effective (Mason 2005)
Multifaceted strategy for CBT cost-effective (Scheeres 2008 & gen
Mortimer 2013)
Financial incentives for implementing ACE inhibitor & other Qndic
cost-effective (Walker 2009)
Audit and feedback for implementing intensified control of blood
glucose is cost-effective (Hoomans 2009)
No co-payments for implementing preventive medication is cost-
effective (Choudry 2011)
Structured patient education with group for implementing self-
management is not cost-effective (Gillespie 2014)
6
4

65. Types of fidelity
1Copy the proven intervention
 Treatment, practice, service delivery model
 Whatever it takes to reproduce this in every day life and operations
2Copy the implementation approach
 To enable patients to take the intevention, use exacty the same reminder
system they found was effective for enabling patient uptake
3Copy both
4Copy the logic of the intervention
65
The letter kills but the spirit gives life

66. 4)Copy the logic of the intervention – the spirit
The effective ingredients to enable practitioners to
follow hand hygiene were:
 Motivation (e.g. patient talks about MRSA)
 Ability (Gell dispensers everywhere, agreement
excuse for late/take longer)
 Triggers (reminders)
 Rewards (performance feedback, etc.)
You make the mix which fits your service
Is that adaption or fidelity to logic or both?
6
6

67. 5)Fidelity to guidance for adaption
for targeting or tailoring
 Following the guidelines for adapting treatments
to older patients with multiple morbidity
6
7

68. Resources
Tools
Web sites
6
82/3/2016

69. References
6
92/3/2016

70. References
7
02/3/2016

71. DETAILS
7
12/3/2016

72. Practice-customer’s questions:
Is the implementation approach effective for
enabling providers to take up the new way?
In setting where we can rigorously evaluate it?
In a range of typical settings
Does the “new way” then result in better patient and
cost outcomes?
What conditions do we need to implement it?
Which adaptations are possible?
How much does it cost and how long before results?7
2

73. 2) Action evaluation
 Researchers share tasks with implementers
in all stages of research
 Researchers present their data to
implementers
 Implementers adjust intervention (or not)
 +ve gain insights and data
 -ve explain effects of researchers (additional
intervention)
 -ve researchers not available in other typical
settings (generalisation)
7
32/3/2016

74. Coleman care transitions model = people
leaving hospital - support for self care
7
42/3/2016
1)Education 2)Coach support at
home

75. RCT evaluated – proven effective
 Research funded
version
 Intervention
specified in
protocol
 Implementation
not described
7
52/3/2016

76. Groups on Coleman dissemination
evaluation
7
6

77. Evaluation type 1: Fidelity to planned practice
 Coleman 2013 core elements to get
outcomes
7
72/3/2016

78. Groups on Coleman dissemination
evaluation Did this study evaluate implementation?
 What data did they collect? How?
 What was “the intervention” and what was
“the implementation” activities, structure and
support?
 How do we know if their conclusions follow
from the data or a personal opinion?
7
8

79. Did subjects “take up” the change as
intended?
Evaluation type 2: for adaptive
implementation
How well did they adapt
intervention during
implementation better to achieve
implementation outcomes
Role of evaluation = a) Help
adaption, b) feedback on success
7
9

80. Other studies
Descriptive study, by facilitators, of
adaptions
…with unclear evidence of outcomes, but
very useful to practice improvement
- More funding for and research of this type?
8
02/3/2016

81. Partnership research with SLL
8
1
Improvement
projects

82. John Øvretveit – Medical Management Center Karolinska
Institutet
Applied healthcare research in a medical
university
Grown since 2002 to 87 staff and 35 PHD
researchers
Example projects
Funding models for Value based purchasing
Implementation of improvements for chronic care
Integrated care
Most mixed methods, many non-experimental
Details: http://ki.se/en/lime/medical-management-centre
8
22/3/2016

83. Culture and translation
.
8
32/3/2016
The strengths of Ca and US
§ diversity,
§ innovation,
§ entrepreneurship,
§ openness,
§ can come together rapidly and attack a
problem,
§ massive resources and talent
How are you going to create into the future the
new California public health? 21/3/16
“No pants metro day”

84. 8
4

85. .
 .
8
52/3/2016

86. 2/3/2016
86
Conclusions
1. Surprises…
2. Useful…
3. Not mentioned …Look this up…