PHREE 2014 Spring Conference Integration and innovation – meeting the challenges of evaluation in the new system

1. © Nuffield Trust08 May 2014
Evaluating service innovations for
older people
Integration and innovation – meeting the challenges of evaluation in
the new system
Martin Bardsley
Nuffield Trust

2. © Nuffield Trust
• Promote independent analysis
and informed debate on
healthcare policy across the UK
• Charitable organization founded
in 1940
• Formerly a grant-giving
organization
• Since 2008 we have been
conducting in-house research
and policy analysis
• Significant interest in uses of
predictive risk techniques
The Nuffield Trust
William Morris
1st Viscount Nuffield
(1877 -1963)

3. © Nuffield Trust
Predictive risk
modelling
Resource
allocation
Descriptive
studies Evaluations
Integrated
care
pilots
nuffield trust
Nuffield Trust Research team – data linkage projects
Risk
sharing
for CCGs
nuffield trust
Combined
predictive
model
nuffield trust
Person
based
resource
allocation
nuffield trust
Social
care at
end of life
nuffield trust
Cancer
and social
care
nuffield trust
Predicting
social
care
costs
nuffield trust
Virtual
Wards
nuffield trust
WSD
nuffield trust
Marie
Curie
Nursing
Service
nuffield trust

4. © Nuffield Trust
Aims
Background
Exploiting routine information
2 case studies of retrospective evaluations
a. Marie Curie Nursing service
b. Partnerships for Older People

5. © Nuffield Trust
Ten-year trend in emergency admissions (46 million admits)
0.00
0.20
0.40
0.60
0.80
1.00
1.20
1.40
2001/02Q1
2001/02Q3
2002/03Q1
2002/03Q3
2003/04Q1
2003/04Q3
2004/05Q1
2004/05Q3
2005/06Q1
2005/06Q3
2006/07Q1
2006/07Q3
2007/08Q1
2007/08Q3
2008/09Q1
2008/09Q3
2009/10Q1
2009/10Q3
2010/11Q1
2010/11Q3
Numberofemergencyadmissions
(millions)
No ACS diagnosis ACS primary diagnosis ACS secondary diagnosis
+35% (40%)
+34%

6. © Nuffield Trust
By ambulatory care sensitive conditions…

7. © Nuffield Trust
Interventions to reduce avoidable admissions
Primary Care ED Depts Hospital Transition
Practice features Assess/obs wards Structured
Discharge
Transition care
management
Medication review GPs in A&E Medication
Review
Rehabilitation
Case
management
Senior Clinician
Review
Specialist Clinics Self management
and education
Telemedicine Coordination EOL
care
Hospital at home
Virtual Wards
see Purdy et al (2012) Interventions to Reduce Unplanned Hospital Admission: A series of systematic
reviews. Bristol University Final Report)

8. © Nuffield Trust
Why the current interest in integrated care?
• Rising levels of chronic disease
• Ageing population
• Increasing levels of hospital admissions and readmissions,
especially among the elderly and vulnerable, and children
• Economic hard times, and unsustainable health and social
care economies
• And too often we still do not get it right in terms of care co-
ordination, care planning, communication with families
• Interest in prevent solutions that reduce the need for hospital
admissions

9. © Nuffield Trust
Integration
Sara Shaw, Rebecca Rosen and Benedict Rumbold What is integrated care? An overview of
integrated care in the NHS Research report. Nuffield Trust June 2011

10. © Nuffield Trust
What information do we have on
whether these are working?……
© Nuffield Trust

11. © Nuffield Trust
Data are everywhere…
GP
Local Authority
Commissioner
A&E
OP
IP
Pharmacy
Community
Health
Services
Up there
Housing
Council
Tax
Council
Social
Services
Social care
provider
Ambulance
ControlNHS Direct
Commissioning data ...

12. © Nuffield Trust
Exploiting person level data
Linking data
a. over time to look at what happens to people – not
just events
b. across care providers to build broader picture
Person level
Capture services provided ->costs; quality
Descriptions of health -> outcomes

13. © Nuffield Trust
Linkage not new
The Oxford Record Linkage Study: A Review of the Method with
some Preliminary Results by E D Acheson DM MRCP and J G
Evans MB MRCP (Nuffield Department of Clinical Medicine, Oxford
University) Proc R Soc Med. 1964 April; 57(4): 269–274.

14. © Nuffield Trust
Tomb raiders?

15. © Nuffield Trust
Information flows

16. Accident and
emergency
350,000 records
Outpatients
1,680,000 records
Inpatients
360,000 records
Social care
240,000 records
Community
matrons
20,000 records
GPs
60 practices
48.5 million records
Relative size of data sets collected
For one PCT area (WSD project)
March 2011

17. © Nuffield Trust
Health and social care timeline – an individual’s
history

18. © Nuffield Trust
Data linkage
Social & secondary care interface

19. © Nuffield Trust
Final year costs: by age
0
2,000
4,000
6,000
8,000
10,000
12,000
14,000
<55 55-64 65-74 75-84 85-94 >=95
Estimatedaveragecostsperdecedent,
£
Age band
Female
All costs
Hospital costs
Social care costs

20. One person hospital cost profile over a year
50+ year old male, total annual cost > £35,000
Outpatients DayCase Elective AE Nonelective
Time (weeks)

21. © Nuffield Trust
Used of linked person level data
Audit and Quality Improvement
Patient safety (e.g. monitoring drug side effects or surgical mortality rates)
Public Health programmes (immunisation; monitoring cancer rates)
Evaluate Services (are they effective and cost effective?)
Planning services (e.g. ICU bed availability; pandemic flu plans; manage
changing patterns of demand)
Manage Performance (e.g. readmission targets; health outcomes indicators)
Resource allocation
Research

22. Why rely on using existing data for research?
Advantage Disadvantage
• Descriptors of events and health
status
• Constrained by the data that are
collected – and quality/consistency
of coding
• Volume of cases versus costs of
data collection
• Handling sensitive personal
information (+/- consent)
• Comprehensive coverage • Coverage of the data – unknown
unknowns
• Enables retrospective studies/ not
time sensitive
• Volume of data – complex
processing

23. © Nuffield Trust
Example (1)
Impact of Marie Curie Nursing Service on place of death &
hospital use at the end of life
http://www.nuffieldtrust.org.uk/publications/marie-curie-
nursing
Chitnis, X. , Georghiou, T., Steventon, A. & Bardsley, M. J. (2013). Effect of a home-based end-of-life nursing service
on hospital use at the end of life and place of death: a study using administrative data and matched controls. BMJ
Supportive & Palliative Care, 1–9. doi:10.1136/bmjspcare-2012-000424
© Nuffield Trust

24. © Nuffield Trust
Methods
• 29,538 people who received MCNS care from January 2009
to November 2011
• Sophisticated matching techniques used to select 29,538
individually matched controls from those who died in
England from January 2009 – November 2011
• Matched on demographic, clinical and prior hospital use
variables
• People started receiving MCNS care on average 8 days
before death

25. © Nuffield Trust
Evaluation: The Marie Curie Nursing Service
Intervention:
• Nursing care support to people at end of life, in their homes
Nuffield commissioned to evaluate impact:
• Are recipients more likely to die at home?
• Reduction in emergency hospital admissions at end of life?
Methods:
• Retrospective matched control study – use of already existing
administrative data

26. © Nuffield Trust
Matched control studies – broad aim
>1M individuals - died Jan 2009 to Nov 2011, did
not receive service
(everyone else)
Aim to find 30,000 individuals who match
almost exactly on a broad range of
characteristics
Use this group as study control group
30,000 individuals - died Jan 2009 to Nov 2011 &
received Marie Curie nursing service before death

27. © Nuffield Trust
Final datasets available for analysis
Nuffield trust
ONS deaths Hospital inpatient, outpatient, AEMC data - desensitised
N = 30,000
• person details
• dates of service
• type of service
Identifiers:
Names, DOB,
Addresses, etc
• dates & place
of death for all
people in
England,
• associated
hospital (HES)
records
Identifiers:
Nuffield Trust
specific HESID

28. © Nuffield Trust
0%
10%
20%
30%
40%
50%
Comorbidities
0%
5%
10%
15%
20%
25%
30%
35%
Cancer diagnoses
Control group – how well matched? Diagnostic history
0%
10%
20%
30%
40%
50%
Comorbidities
0%
5%
10%
15%
20%
25%
30%
35%
Cancer diagnoses
Marie Curie Controls

29. © Nuffield Trust
Results - Proportion of people dying at home
• 77% of MCNS patients died at home but only 35% of controls
• Impact of MCNS care on home deaths greater for those with no
history of cancer then for those with cancer
Figure 2 – Place of death for Marie Curie Nursing Service patients & matched controls

30. © Nuffield Trust
Emergency admissions for cases where nursing started 3-7 days
before death

31. © Nuffield Trust
Emergency admissions for cases where nursing started 8-14 days
before death

32. © Nuffield Trust
Impact of MCNS care on hospital costs
Table 1 – Post index date hospital costs for Marie Curie cases and matched controls
Mean (sd) hospital costs per person
Activity Type Marie Curie cases Matched controls Difference
Emergency admissions £463 (£1,758) £1,293 (£2,531) £830
Elective admissions £106 (£961) £350 (£1,736) £244
Outpatient attendances £33 (£212) £76 (£340) £43
A&E attendances £9 (£34) £31 (£60) £22
All hospital activity £610 (£2,172) £1,750 (£3,377) £1,140
• Significantly greater reduction in costs among those with no
recent history of cancer
• Also cost reduction much greater for those who started
receiving MCNS care earlier (£2,200 for those >2 weeks
before death)

33. © Nuffield Trust
Summary
• Evaluation of large-scale, existing palliative care service using
well-matched controls
• Caveats – not all costs considered; unobserved differences
about MCNS users
• Those who received home-based palliative care:
• Much more likely to die at home
• Lower use of hospital care (particularly unplanned)
• Lower hospital costs
• Impact of MCNS care greater for those without cancer –
surprising finding, although literature limited

34. Example (2)
Evaluation of Community Based Interventions impact on
hospital admissions
Retrospective evaluation using matched controls
Adam Steventon, Martin Bardsley, John Billings, Theo Georghiou and Geraint Lewis An evaluation of the impact of
community-based interventions on hospital use. A case study of eight Partnership for Older People Projects (POPP) .
Nuffield Trust March 2011
© Nuffield Trust

35. © Nuffield Trust
The Partnership for Older People Projects (POPPs)
“We recommend expanding the
Partnerships for Older People
Projects (POPPs) approach to
prevention across all local
authorities and PCTs.”
•£60m investment by DH with aim to:
“shift resources and culture away
from institutional and hospital-
based crisis care”
•146 interventions piloted in 29 sites.
•National evaluation of whole programme
found £1.20 saving in bed days per £1
spent.

36. © Nuffield Trust
From the 146 interventions offered under POPP, we
selected 8 for an in-depth study of hospital use
Support workers for community matrons
Intermediate care service with generic workers
Integrated health and social care teams
Out-of-hours and daytime response service
+ 4 different short term assessment and
signposting services

37. © Nuffield Trust
Our preferred option for this evaluation:
link participants to HES through a trusted third party
March 2011
Collate files and
add NHS
numbers
Derive
HES ID
Collate patient lists
Patient identifiers
(e.g. NHS number)
Trial information (e.g.
start and end date)
Non-patient identifiable keys
(e.g. HES ID, pseudonymised
NHS number)
Participating sites
Information
Centre
Nuffield Trust

38. © Nuffield Trust
Prevalence of health diagnoses categories in intervention
and control groups
0%
10%
20%
30%
40%
50%
60%
Control Intervention

39. © Nuffield Trust
Overcoming regression to the mean using a control
group
March 2011
0.0
0.1
0.2
0.3
-12 -11 -10 -9 -8 -7 -6 -5 -4 -3 -2 -1 1 2 3 4 5 6 7 8 9 10 11 12
Numberofemergencyhospitaladmissions
perheadpermonth
Month
Intervention
Start of intervention

40. © Nuffield Trust
Overcoming regression to the mean using a control
group
March 2011
0.0
0.1
0.2
0.3
-12 -11 -10 -9 -8 -7 -6 -5 -4 -3 -2 -1 1 2 3 4 5 6 7 8 9 10 11 12
Numberofemergencyhospitaladmissions
perheadpermonth
Month
Intervention
Start of intervention

41. © Nuffield Trust
Overcoming regression to the mean using a control
group
March 2011
0.0
0.1
0.2
0.3
-12 -11 -10 -9 -8 -7 -6 -5 -4 -3 -2 -1 1 2 3 4 5 6 7 8 9 10 11 12
Numberofemergencyhospitaladmissions
perheadpermonth
Month
Intervention
Start of intervention

42. © Nuffield Trust
Overcoming regression to the mean using a control
group
March 2011
0.0
0.1
0.2
0.3
-12 -11 -10 -9 -8 -7 -6 -5 -4 -3 -2 -1 1 2 3 4 5 6 7 8 9 10 11 12
Numberofemergencyhospitaladmissions
perheadpermonth
Month
Control Intervention
Start of intervention

43. © Nuffield Trust
Impact of eight different interventions on hospital use

44. © Nuffield Trust
Conclusions
• Able to undertake a retrospective evaluation of changes in hospital use
for 8 projects, over 5000 subjects
• Study took less than 3 months once permissions obtained
• Findings suggest that none of these projects were delivering the
anticipated reduction in hospital use
• The approach has limitations eg there is always the risk of unmeasured
confounders; end points limited by the data available.
• The ability to track individual histories using existing data sets has great
strengths and wider application

45. © Nuffield Trust
Findings from other studies study
March 2011 © Nuffield Trust

46. © Nuffield Trust
And for 3 virtual wards…

47. © Nuffield Trust
And 11 integrated care pilots
(all pilots combined n=11,296)
• Elective admissions &
outpatient attendances
reduced more quickly for
intervention patients than
matched controls.
• However, emergency
admissions appeared to have
increased more quickly.
Difference in difference analysis
(individual patient level)
Absolute
difference
(per head)
Relative
difference
p-value
Emergency
admissions
0.02 +2 % 0.03
A&E
attendance
-0.01 -1% 0.26
Elective
admissions
-0.04 -4% 0.003
Outpatient
attendance
-0.20 -20% <0.001 *
* Difference also detected at practice level

48. © Nuffield Trust
9 observations
1. Recognise that planning and implementing large scale service changes take time
2. Define the service intervention clearly including what it is meant to achieve and how, and manage
implementation well
3. Be explicit about how the desired outcomes are supposed to arise and use interim markers of
success
4. Consider generalisability and context: they are important
5. If you want to demonstrate statistically significant change, size and time matter
6. Hospital use and costs are not the only impact measures
7. Pay attention to the process of implementation as well as outcome
8. Carefully consider the best models for evaluation
9. Work with what you have: organisation and structural change may not achieve desired outcomes

49. © Nuffield Trust
Summary
• Emergency admissions and urgent care seen as critical drives of need for new
services
• Many different initiatives aimed at integrating across primary/secondary care
divide – often with explicit aims to reduce emergency admissions
• Huge potential in exploiting linked data sets for retrospective evaluation of new
models of care
• Evaluation of many integrated care initiatives suggest reducing emergency
admission is very difficult – though they may have other benefits
• Some evidence that a well established programme for end of life care does reduce
need for hospital care

50. © Nuffield Trust08 May 2014
www.nuffieldtrust.org.uk
Sign-up for our newsletter
www.nuffieldtrust.org.uk/newsletter
Follow us on Twitter:
Twitter.com/NuffieldTrust
© Nuffield Trust
Ian.blunt@nuffieldtrust.org.uk
Adam.steventon@nuffieldtrust.org.uk