Dr. Felix Greaves presents his work using patients' online descriptions of their care to understand healthcare quality in the NHS

1. The Tripadvisor approach for health?
Using patients' online descriptions of their care to
understand healthcare quality
Felix Greaves
Imperial College London
@felixgreaves

2. England is proud of the National Health Service

3. But it is not without problems
Quality can be variable, and we often fail to hear the patient perspective

4. At the same time, we love rating things on the internet

5. We now do it for our experiences of healthcare too

6. The number of patients describing their care online is
increasing
USA
UK
Number of hospital ratings
25000
Gao et al., JMIR, 2012
20000
15000
10000
5000
0
Greaves et al., JMIR, 2012

7. There is resistance from many clinicians to the idea of
being rated online:

8. Arguments for and against online rating
Why it’s bad
Why it’s good
1. Strain doctor-patient
1.Doctors can often be poor
relationships
judges of their patients’
experience
2. Reviews may be malicious
or fake
2.Feedback changes doctors’
performance
3. Selection bias by those
leaving reviews
3.People are using the
Internet to voice opinions,
4. Lack of meaningful data on
so why not capture this
technical quality of health
information in a useful form
care

9. Research Question
Is there a relationship between ratings online
and ‘hard’ measures of quality?

10. The opportunity for a natural experiment
NHS Choices website allows patients to review all NHS services in England

11. Method:
Obtained national review data from the Department of
Health

12. We thought we could compare ratings with some
traditional measures of quality
Vs.

13. Average scores for 10,000 ratings
Mean rating
(out of 5)
Score
Range
The environment where I
was treated was…
3.6
2.6-5.0
I was treated with dignity
and respect by the hospital
staff…
I was involved with
decisions about my care…
The hospital staff worked
well together…
4.0
2.7-5.0
3.8
2.4-5.0
4.1
2.9-5.0
Greaves et al. BMJ Quality and Safety, 2012

14. Average scores for 10,000 ratings
Score
The environment where I
was treated was…
Mean rating
(out of 5)
3.6
Range
2.6-5.0
I was treated with dignity
4.0
2.7-5.0
and respect by The majority of ratings are positive
• the hospital
staff…
• 67% would recommend to a friend
I was involved with
3.8
2.4-5.0
decisions about my care…
The hospital staff worked
4.1
2.9-5.0
well together…
Greaves et al. BMJ Quality and Safety, 2012

15. Ratings compared to patient experience surveys
NHS Choices
Measure
Survey question:
Spearman
Rho
p value
Proportion of
patients
recommending
“Overall, how would you rate the
quality of care you received”
0.40
<0.001
Rating of being
treated with dignity
and respect
“Overall, did you feel you were
treated with dignity and respect
while in hospital?”
0.33
<0.001
Rating of staff
working together
“How well would rate how well the
doctors and nurse worked
together?”
0.32
<0.001
Rating of cleanliness “How clean was the hospital ward or 0.48
room you were in?”
<0.001
Greaves et al. BMJ Quality and Safety, 2012

16. Ratings compared to patient experience surveys
NHS Choices
Measure
Proportion of
patients
recommending
Survey question:
“Overall, how would you rate the
quality of care you received”
Spearman
Rho
0.40
There is a moderate, highly
Rating of being
“Overall, did you feel you were
0.33
significant association between
treated with dignity treated with dignity and respect
ratings online and large
and respect
while in hospital?”
surveys of patient experience
Rating of staff
working together
“How well would rate how well the
doctors and nurse worked
together?”
p value
<0.001
<0.001
0.32
<0.001
Rating of cleanliness “How clean was the hospital ward or 0.48
room you were in?”
<0.001
Greaves et al. BMJ Quality and Safety, 2012

17. Ratings compared to outcomes
NHS Choices Measure
Spearman
Rho
Other variable
Proportion of patients Hospital Standardised Mortality
recommending
Ratio
p value
-0.20
0.01
Proportion of patients Standardised morality rate for high -0.22
recommending
risk conditions
0.01
Proportion of patients Standardised morality rate among -0.00
recommending
surgical inpatients with serious
treatable complications
Proportion of patients Emergency readmission rate
-0.31
recommending
within 28 days
0.99
Patient perception of
cleanliness
<0.001
Rate of MRSA bacteraemia (per
1,000 bed days)
-0.30
<0.001
Greaves et al. Arch Int Med, 2012

18. Ratings compared to outcomes
NHS Choices Measure
Spearman
Rho
Other variable
Proportion of patients Hospital Standardised Mortality
recommending
Ratio
p value
-0.20
0.01
Proportion of patients Standardised morality rate for high -0.22
recommending There is a weak, significant
risk conditions
0.01
association between ratings
Proportion of patients Standardised morality rate among -0.00
recommending online and mortality rates
surgical inpatients with serious
0.99
treatable complications
Proportion of patients Emergency readmission rate
recommending
within 28 days
-0.31
<0.001
-0.30
<0.001
Patient perception of
cleanliness
Rate of MRSA bacteraemia (per
1,000 bed days)
Greaves et al. Arch Int Med, 2012

19. A study in the US found
the same results
comparing Yelp reviews
with the HCAHPS survey

20. A wisdom in the crowd of patients?
Sir Francis Galton won ‘guess-the-weight-of-the-bull’ competitions
by asking lots of local farmers for their best guess, and then taking
the average – he gave us the idea of a ‘wisdom of crowds’

21. Comparison of the NHS Inpatient Survey and ratings on the NHS
Choices website
NHS Inpatient Survey NHS Choices ratings
Mechanism
Paper-based survey
Ratings left on a website
Number of
responses
Selection
69,000 per year
5,000 per year
Self-selecting; patients are
not solicited
Proportion
positive
Random; patients receive a
survey requesting
completion after leaving
hospital
79% rated their overall care
as excellent or very good
Cost
Likely more expensive
Likely less expensive
67% would recommend to a
friend

22. Comparison of the NHS Inpatient Survey and ratings on the NHS
Choices website
NHS Inpatient Survey NHS Choices ratings
Mechanism
Number of
responses
Selection
Proportion
positive
Cost
Paper-based survey
Ratings left on a website
There were 10,000 hospital ratings inper year
69,000 per year
5,000 the
UK over 2 years
Random; patients receive a Self-selecting; patients are
Over the same time
not were
survey requesting period there solicited
29,118,009 hospital
completion after leaving admissions
hospital
79% rated hospital admissions are rated
0.04% of their overall care 67% would recommend to a
as excellent or very good
friend
Likely more expensive
Likely less expensive

23. The stereotypical reviewer?

24. Where are people
rating their care?

25. Associations between whether a practice is rated
with population and practice characteristics
Independent
variable
Z statistic
p value
Practice
population size
IMD score of
patients
Population
density
15.38
<0.001
-7.82
<0.001
6.72
<0.001
Singlehander
Proportion of
population aged
over 65 years
Proportion of
population who
are white
Type of contract
Training practice
-4.50
-3.88
<0.001
<0.001
-1.58
0.11
-0.71
0.35
0.48
0.73

26. Associations between whether a practice is rated
with population and practice characteristics
Independent
variable
Z statistic
p value
15.38
<0.001
Practice
population size
-7.82
<0.001
IMD score of
• Practices serving younger people
patients
<0.001
Population more6.72
likely to be rated
density
are
• Practices serving less deprived people
Singlehander
-4.50
<0.001
-3.88
Proportion are more likely to be rated <0.001
of
population aged
• Practices in urban areas are more
over 65 years
0.11
Proportion likely -1.58be rated
of
to
population who
are white
Type of contract -0.71
0.48
Training practice 0.35
0.73

27. Cloud of patient experience ?

28. A problem
Free Text

29. A new trend: ‘Big Data’ and social media analytics

30. Sentiment analysis

31. Another natural experiment
We can compare patients’ free text descriptions of care with their own quantitative ratings

32. What we did: Machine learning
• You need to teach an algorithm how to recognise particular
words and phrases
• We used all comments and ratings from 3 years as a training
set (13,802 comments)
We tried to predict patients ratings of their care from their
comments in 2010
• Whether the patient would recommend the hospital or not
• Whether the hospital was clean or not
• Whether the patient was treated with dignity or not
Used open source Weka software

33. Sentiment analysis can be tricky

34. Simple punctuation contains little information
(

35. But when paired with other punctuation, it’s very
important
:(

36. And small changes can have big effects on sentiment
:)

37. Some words the algorithm thought were important
Overall
• rude
• excellent
• hours
• pain
• communication
Cleanliness
• dirty
• floor
• filthy
• bed
• blood
Dignity
• rude
• told
• thank
• friendly
• attitude

38. How good is it at predicting ratings?
Question we are predicting
the answer to
Total number Prediction
Kappa P value
of comments accuracy
Would you recommend the
hospital?
6412
88.7
0.75
<0.0001
How clean was the hospital
room or ward that you were in?
6139
81.2
0.40
<0.0001
Overall, did you feel you were
treated with respect and dignity
while you were in the hospital?
6239
83.6
0.56
<0.0001

39. How does this compare to patient surveys?
Patient Survey
Question
Machine learning
prediction
Spearman
rho
Overall, how would you
rate the care you
received?
Whether the patient would
0.46
recommend the hospital
Probability
P<0.001
Overall, did you feel you
were treated with respect Whether the patient was
treated with dignity
and dignity while you
were in the hospital?
0.50
P<0.001
In your opinion, how
clean was the hospital
room or ward that you
were in?
0.37
P<0.001
Standard of cleanliness

40. Limitations
• Selection bias
• Sarcasm / Irony
• Culturally specific
 Phrases like ‘cup of tea’ important,
but effect depends on context

41. The UK quality regulator now
includes online reviews and
comments in its performance
measurement framework

42. The NHS has started publically reporting social media sentiment

43. Conclusions
Online rating and reviewing are on the rise
Evidence of some wisdom in the crowd of patients
Substantial professional resistance to the idea
Selection bias in those commenting online
The ‘cloud of patient experience’ is a potentially valuable
source of information to understand patient views

44. Thanks to my co-authors, and funders
•
•
•
•
•
•
•
•
•
Christopher Millett
Ara Darzi
Dominic King
Henry Lee
Utz Pape
Liam Donaldson
Azeem Majeed
Robert Wachter
Daniel Ramirez-Cano