This document summarizes NHS England's approach to gathering patient experience and outcome data. It discusses various data collection methods, including national patient surveys, the Friends and Family Test, and Patient Reported Outcome Measures (PROMs). It notes that PROMs data shows patients report significant health improvements after surgeries and there is some variation in outcomes between hospitals. The document also outlines challenges in using this data and opportunities for the future, such as developing new PROMs for additional clinical areas and engaging patients more in collecting and using their own outcome data.

1. www.england.nhs.uk
Dan Wellings,
Head of Insight and
Feedback
NHS approaches to
patient experience
and patient
reported outcome
measures (PROMs)
28th April 2015

2. www.england.nhs.uk
Text Text
World class data and
intelligence
Patient voice
and insight
Customer choice
and control
Transparency
Participation
Working on cornerstones of a
world class patient service
Knowledge
for all
2

3. www.england.nhs.uk
What does insight mean?
3
• Using qualitative and
quantitative data from
users to inform what we do
• Using whatever data
sources we have - not just
surveys but a whole range
of feedback techniques
• Always asking the
question, what do we do
with the data?
• Making a difference with
data
“The most important single
change in the NHS in
response to this report would
be for it to become, more
than ever before, a system
devoted to continual learning
and improvement of patient
care, top to bottom and end
to end”
A promise to learn – a
commitment to act
The Berwick Report

4. www.england.nhs.uk
Patient
experience
data

5. www.england.nhs.uk
National Patient Experience Surveys
5
• Current National Surveys – feedback from c1.5 million people a
year
• GP Patient survey (NHS England, twice a year)
• Community & Mental Health survey (CQC, annual)
• Inpatient survey (CQC, annual)
• A&E survey (CQC, every 3 years)
• Outpatient survey (CQC, every 3 years)
• Maternity survey (CQC, every 3 years)
• National Cancer Patient Experience survey (NHS England, annual)
• VOICES survey of Bereaved People (NHS England, annual)
• Staff survey (NHS England, annual)
• Adult Social Care survey (Department of Health, annual)

6. www.england.nhs.uk
Biggest challenge is measuring
experience of integrated care
6

7. www.england.nhs.uk
The Friends
and Family Test

8. www.england.nhs.uk
Friends and Family Test Overview
‘How likely are you to recommend our <ward / A&E
department> to friends and family if they needed similar care
or treatment?’
1 Extremely Likely
2 Likely
3 Neither Likely nor Unlikely
4 Unlikely
5 Extremely Unlikely
6 Don’t know
Why?

9. www.england.nhs.uk
The Friends and Family Test Review
• Positive findings: FFT is making an impact
• 78% of Trusts reported that FFT had increased the emphasis placed
on patient experience at their Trust
• FFT is being used mainly as a service improvement tool
• Provides feedback to frontline staff, often boosting morale
• Feedback is real-time and granular: it tells staff about what
is happening in their ward
• Identifies problems and encourages action to be taken
• Not used widely for performance management
• Concerns over robustness of data, including comparability
between trusts

10. www.england.nhs.uk
FFT is not a survey
• FFT is a real-time local feedback tool
• It has different characteristics from national surveys
FFT
• Real-time
• Ward-level
• Effective for service improvement
• Can be used as early-warning system
• Not representative, not comparable
National surveys
• Robust, comparable data at Trust level
• Data on a range of issues
• Suitable for performance management
• Not meaningful to front-line staff
• Has not historically changed behaviours
• FFT and National surveys are complementary sources of Insight
• FFT is a formative measure: it provides data to improve services
• National surveys are summative measures: they provide an accurate picture
of relative performance

11. www.england.nhs.uk
FFT has been seen through the wrong lens

12. www.england.nhs.uk
A measure and an intervention –
6.5 million responses so far

13. www.england.nhs.uk
The PROMs
programme

14. www.england.nhs.uk
(Berwick 1997)
“The ultimate measure by which to judge the
quality of medical effort is whether it helps
patients (and their families) as they see it.
Anything done in health care that does not
help a patient or family is, by definition,
waste, whether or not the professions and
their associations traditionally hallow it”

15. www.england.nhs.uk
(Appleby and Devlin 2010)
“The first health system
internationally to measure what
it produces in terms of health.”

16. www.england.nhs.uk
• 4 national PROMs for
surgical conditions
introduced in 2009
• Hip replacement
• Knee replacement
• Groin hernia
• Varicose vein
• 180,000 pre-op
questionnaires per year
• Cost to trusts annually ~
based on approx
£3- £5 / patient
• Cost to develop the
process, tools, pilot and
establish collections, to
date ~ approx £7-8 million
The established PROMs programme - scope

17. www.england.nhs.uk
• Original objectives
• Outcome comparison
between NHS and
independent sector provider;
• Inform patient choice;
• Performance management
of organisations and
clinicians;
• Facilitate service
improvement by clinicians
and managers
• Patient centred care
• Evaluation of practices and
policies
• Established uses
• Outcomes Framework
indicator 3.1
• CQC Intelligence Monitoring
• Hip/ Knee best practice tariff
from April 2014
• CCG Data Packs
• Using PROMs in “Beyond
Compliance” approvals of
new prostheses to market
The established PROMs programme - aims

18. www.england.nhs.uk
Q1
• Pre-operative questionnaire
• Condition-specific + generic instruments
Q2
• Post-operative questionnaire (3 or 6 months)
• Repeats the same instruments
Q2-Q1
• Difference between scores in Q2 and Q1 = health improvement/gain
• Case-mix adjustment
• Outlier identification
The established PROMs programme - process

19. www.england.nhs.uk
Patients report big health improvements
• Patients fill in a pre-op questionnaire at a consultation prior to
surgery and a post-op questionnaire about six months after surgery.
• The difference in Oxford Hip (Knee) scores between the pre-
operative and post-operative scores can be taken as an indication
of the health impact of the surgery - generally very positive.

20. www.england.nhs.uk
There is variation in outcomes, with some
trusts significantly above the England
average, even after we have taken into
account the case-mix
0
5
10
15
20
25
30
0 100 200 300 400 500 600 700 800 900
AdjustedHealthGain
Number of Modelled Records
95% control limits
99.8% control
limits
England
All
Positive Outliers are identified above the
upper 99.8% Confidence Limit
21

21. www.england.nhs.uk
There is variation in outcomes, with some
trusts significantly below the England
average, even after we have taken into
account the case-mix
0
5
10
15
20
25
30
0 100 200 300 400 500 600 700 800 900
AdjustedHealthGain
Number of Modelled Records
95% control limits
99.8% control
limits
England
All
Negative Outliers are identified below the
lower 99.8% Confidence Limit
For Hips and Knees since April 2014 this
may mean not receiving Best Practice Tariff
22

22. www.england.nhs.uk
2010 Health White Paper
“Providers will be paid according to
their performance. Payment should
reflect outcomes, not just activity,
and provide an incentive for better
quality.”

23. www.england.nhs.uk
Best Practice Tariff
• Introduced in 2014/15 for primary hip and knee replacement
• Level of reimbursement conditional on patient outcomes
• Only providers meeting a number of conditions receive full
payments.
• Providers who have significantly poorer outcomes receive an
about 10% smaller payment per procedure.

24. www.england.nhs.uk
Best Practice Tariff (knee replacement
with complications and comorbidities)
5,300
5,400
5,500
5,600
5,700
5,800
5,900
6,000
6,100
6,200
6,300
6,400
(£)
£633 Incentive
Standard
tariff
BPT

25. www.england.nhs.uk
“Little is known about the impact of
PROMs.”
N. Black, BMJ 2013

26. www.england.nhs.uk
Outcomes have improved since 2009/10
Knee outcomes*
0.0%
10.0%
20.0%
30.0%
40.0%
50.0%
60.0%
Excellent results
(hip)
Poor results
(hip)
Hip outcomes*
* Oxford Hip/Knee Scores: Poor result: < 27
Excellent result > 41
0.0%
10.0%
20.0%
30.0%
40.0%
50.0%
60.0%
Poor results
(knee)
Excellent
results (knee)
27

27. www.england.nhs.uk
• Anecdotal evidence of use
of PROMs for service
improvement:
• “Northumbria Hospitals have
changed the type of
prosthesis used after
analysis of the data from
PROMs and the National Joint
Registry showed differential
performance, on the basis of
patient scores on
complications, pain levels,
Oxford hip/knee scores, and
EQ-5D scores”
• Some anecdotal evidence
on usage with patients
• For choice
• For shared decision
making
• Some anecdotal evidence
on how CCGs are using
data for commissioning
Evidence of Impact

28. www.england.nhs.uk
Challenges of using PROMs
• The experiences of professionals with using
information from patient-reported outcome measures
to improve the quality of healthcare: a systematic
review of qualitative research.
• Boyce, Browne, Greenhalgh. BMJ Quality and Safety
(2014)
• •Practical problems (technical support, workload)
• •Negative attitudes (suspicion of managerial objectives,
concerns about impact on patient-clinician relationship)
• •Methodological concerns (validity, interpretability)
• •Doubts about impact (cannot turn the data into practical
solutions).

29. www.england.nhs.uk

30. www.england.nhs.uk
• Develop PROMs tools for shared use by patients and
clinicians
• Support patients in managing their condition and
allow clinicians to monitor their progress remotely
• Personalised goals can ensure that what matters to
the patient is being measured, but also that the care
designed by the clinician reflects these outcomes
Making patients partners in their
care – PCOMs?

31. www.england.nhs.uk
NOW
• Make better use of existing
PROMs data to drive
change, raising awareness
and improving
understanding of what the
data tells us
• Consult on existing
PROMs
• Explore new methods of
collecting PROMs data
including electronic
solutions
FUTURE
• Pilot PROMs in new
clinical areas, where the
evidence is poor
• Develop methods where
patients become partners
in their own outcome data
• Exploring how PROMs
data can be included in
national clinical audits
• Exploring how outcomes
can affect tariff
PROMs and PCOMs – a policy area still in its
relative infancy

32. www.england.nhs.uk
The Insight
Strategy

33. Insight &
Feedback
Less descriptive
LessgeneralisableMoregeneralisable
More descriptive
In depth interviews
Experience Based Co-Design
Focus groups
& patient forums
NHS Trusts
CCGs
Healthwatch
Patient organisations
Surveys
GP Patient Survey
Bereaved Voices
Cancer Patient Experience Survey
NHS Staff Survey
PROMs
Online reviews
& ratings
NHS Choices
Patient Opinion
IWantGreatCare
Care Connect
Patient stories
HealthTalkOnline
Patient Voices
Feedback tools
(kiosks, SMS, apps)
Examples include:
Hospedia
Synapta
Feedback apps
Social media
listening
Salesforce
Polecat
Healthberry
Public
Meetings
NHS Trusts
CCGs
Matrix of insight methods from Measuring patient experience: evidence scan Health Foundation, June 2013
Friends & Family Test
Complaints

34. www.england.nhs.uk
Objective of Insight Strategy
• To establish a clear strategic vision for how
the different elements of insight & feedback
can complement each other most effectively,
to be greater than the sum of their individual
parts in giving us a clear picture about what
patients and the public think of their NHS
services

35. www.england.nhs.uk
Overall themes

36. www.england.nhs.uk
Designing the appropriate methodology
for the intended purpose
37
• Performance management
• Service improvement
• Patient involvement
• Service re-design
• Commissioning services
• Patient Choice
• Pay for performance
• Regulation

37. www.england.nhs.uk
Methodologies – pros and cons
RobustnessUsability

38. www.england.nhs.uk
Balance between local and national
collections - ownership
39

39. www.england.nhs.uk
Is data being presented back
appropriately to each audience?
40
Patients and the
public
Frontline staff
Managers and
commissioners
Academics

40. www.england.nhs.uk
Conclusions and a few thoughts on
what we might all look to do
41
• Be very clear on what data is collected for and do not overload
a collection
• Think early on about how data will be translated into change
• Think clearly about audience(s)
• Involve patients as much as possible in the design
• Train people to use the data and show how change can be
made
• Can we reward appropriate usage of insight?
• Numbers without meaning can be dangerous – signal not noise
• Reflect on ownership - engagement

41. www.england.nhs.uk
Means nothing if nothing is done