Mixin Classes in Odoo 17 How to Extend Models Using Mixin Classes
Performance Indicators in the Health Service
1. Outline Introduction Performance Management Data League Tables Issues
Performance Indicators in the Health Service
Paul Hewson1
11th April
1
University of Plymouth, email paul.hewson@plymouth.ac.uk
2. Outline Introduction Performance Management Data League Tables Issues
1 Introduction
2 Performance Management
3 Data
4 League Tables
Assessing uncertainty
Case Mix
Making allowance for the size of an organisation
Funnel Plots
Monitoring changes over time
Methods from Industrial Quality Control
5 Issues
3. Outline Introduction Performance Management Data League Tables Issues
Aims of Management SSUs
To assess the student’s ability to:
Define a health service management problem.
Demonstrate an understanding of the historical and
contemporary background of the problem. How did things
evolve this way? What are the current issues that need
addressing and are driving change?
Define a strategy and research possible solutions.
Propose and justify a particular solution.
Present the problem, possible solutions, and proposed solution
verbally, including answering questions,
Use appropriate visual aids to support the verbal presentation,
4. Outline Introduction Performance Management Data League Tables Issues
Aims of Unit
Appreciation of methods used in the clinical design of
Performance Indicators;
How to interpret performance indicators in the context of
random fluctuation;
How to make allowance for different case mix;
AND thinking about this in a real-world context.
5. Outline Introduction Performance Management Data League Tables Issues
In particular:
Performance Indicators are an increasingly important
management tool in the health service, consider the very well
publicised Healthcare Commission Indicators, QoF indicators
in primary care, as well as recent publication of surgeon
specific mortality rates.
No longer tools imposed from without, in many cases
scientific evidence and practitioner input has been used to
design a suitable set of measures.
6. Outline Introduction Performance Management Data League Tables Issues
So we are interested in:
Best practice in performance indicator design;
How routine clinical information is coded into databases that
ultimately becomes a performance indicator;
how we assess uncertainty;
how we make valid comparison on units (surgeons, hospitals,
areas) which may differ due to context or patient case mix;
How to satisfy the people paying our wages (or their
representatives) that we are delivering continuous
improvement in patient care.
7. Outline Introduction Performance Management Data League Tables Issues
Aims of today
To plan the rest of the unit, contact, logistics, assessment
To consider an overview of the role and practice of
performance management
To present some information on exciting(?) technical aspects
Financial aspects will be considered later. We might also want
to consider data coding in more detail later.
To discuss topics that may be suitable for assessment
8. Outline Introduction Performance Management Data League Tables Issues
Assessment
Assessment will consist of:
A presentation made to a small audience,
Audience may include fellow students,
Presentation will be in a semi-formal environment.
Any volunteers for video-recording?
9. Outline Introduction Performance Management Data League Tables Issues
Assessment
A 20-minute slot should be allowed for each student,
approximately:
(Up to) 15 minutes for the presentation and to answer
questions;
5 minutes for feedback from the audience
In addition:
5 minutes should also be allowed in the programme for set up
of each presentation.
The assessor will need 10 minutes to complete the assessment
form including written feedback.
colorredElectronic presentations will not necessarily score more
than non-electronic ones (OHPs).
10. Outline Introduction Performance Management Data League Tables Issues
Performance Management: One style among many
For the foreseeable future, performance management is here to
stay. But two papers (there are plenty more) remind us that there
are other management styles:
Adab, P., A.M. Rouse, M.A. Mohammed and T. Marhsall
(2002) “Performance league tables: the NHS deserves better”
Brit.Med.J. 324:95-98
Davies, H. and J. Lampel (1998) “Trust in Performance
Indicators” Quality in Health Care 7:159-162
11. Outline Introduction Performance Management Data League Tables Issues
Known Risks with Performance Management
Tunnel vision (ignoring non-measured aspects of a service);
Sub-optimisation (setting modest improvement goals);
Convergence (aiming to match the average);
Gaming (dealing with easiest clients / problems first);
Ossification (avoiding innovation);
Misrepresentation (see National Audit Office (2001)
Inappropriate adjustments to NHS Waiting Lists London:
National Audit Office).
12. Outline Introduction Performance Management Data League Tables Issues
Data Sources
“Public agencies are very keen on amassing statistics -
they collect them, add them, raise them to the nth power,
take the cube root and prepare wonderful diagrams. But
what you must never forget is that every one of those
figures comes in the first instance from the village
watchman, who just puts down what he damn pleases”
Sir Josiah Stamp 1880-1941 (Governor of the Bank of England)
13. Outline Introduction Performance Management Data League Tables Issues
Data Quality
You can find plenty of other examples. For today, consider the
paper by Speigelhalter et al. (2002)2 . Consider in particular:
The number of different sources of data recording the same
events;
The reason for collecting these different data sets;
How useful the data were from any of them.
2
Spiegelhalter, D.J., P.Aylin, N.G.Best, S.J.W. Evans and G.D.Murray
(2002) ‘ Commissioned analysis of surgical performance using routine data:
lessons from the Bristol Enquiry” J.R.Statis.Soc.A 165:191-231
14. Outline Introduction Performance Management Data League Tables Issues
Data Validity: are you measuring what you want to
measure
“Not everything that counts is counted, and not
everything that can be counted counts” Albert Einstein
(approximate quote).
A couple of hospital based clinical examples where this has been
considered includes:
McGlynn, E. and S. Asch (1998) “Developing a clinical
performance measure” American Journal of Preventative
Medicine 14:14-21
Dorsch M., R. Lawrence, R. Sapsford, J. Oldham, D.
Greenwood, B. Jackson, C. Morrell, S. Ball, M. Robinson and
A. Hall (2001) “A simple benchmark for evaluating quality of
care of patients following acute myocardial infarction” Heart
86:150-154
15. Outline Introduction Performance Management Data League Tables Issues
Designing PM systems: technical aspects
A semi-technical discussion has been prepared by a Royal
Statistical Society working party
http://www.rss.org.uk/main.asp?page=1222.
One example considered are simple pass-fail indicators. Where are
these used?
16. Outline Introduction Performance Management Data League Tables Issues
Data Validity
∴ a large part of performance indicators surrounds defining them
sensibly in the first place. Options for developing an indicator
include borrowing a definition from somewhere else:
Miles, H., E.Litton, A. Curran, L.Goldsworthy, P.Sharples and
A. Henderson (2002) “The PATRIARCH study: Using
outcome measures for league tables: Can a North American
prediction of admission score be used in a United Kingdom
children’s emergency department?” Emerg.Med.J. 19:536-538
as well as quite elaborate procedures for developing a clinical
consensus as to what should be measured:
Normand, S-L.T., B. McNeil, L. Peterson and R. Palmer
(1998) “Methodology matters - VIII. Eliciting expert opinion
using the Delphi technique: identifying performance indicators
for cardiovasular disease” International Journal for Quality in
Health Care 10:247-260
17. Outline Introduction Performance Management Data League Tables Issues
Several Famous Problems with league tables
Small changes in performance can lead to very large changes
in rank;
Small organisations more affected than large ones
(randomness);
There is no allowance for “case” mix or the context in which
the organisation operates.
One study will be quoted (there are many which report similar
results) suggesting that between 1.6% and 2.3% of variation in
mortality rate was due to institutional effects: see Merlo, J., P.-O.
Ostegren, K. Broms, A. Bjork-Linne, and H. Liedholm (2001)
“Survival after initial hospitalisation for heart failure: a multilevel
analysis of patients in Swedish acute care hospitals” J. Epidemiol.
Community Health 55:323-329.
18. Outline Introduction Performance Management Data League Tables Issues
Assessing uncertainty
Uncertainty in League Tables
The following slide has been extracted from Marshall and
Spiegelhalter (1998)3 , a paper approaching citation classic
status in the BMJ.
This first chart shows confidence intervals around the raw live
birth rate.
There are arguments that all Performance Indicators should
come with some assessment of the possible uncertainty. What
happens with Healthcare Commission Indicators?
3
Marshall, E. C. and D. J. Spiegelhalter (1998) “Reliability of league tables
of in vitro fertilisation clinics; retrospective analysis of live birth rates.” Br.
Med. J.316:1701-1705
19. Outline Introduction Performance Management Data League Tables Issues
Assessing uncertainty
Uncertainty in League Tables
20. Outline Introduction Performance Management Data League Tables Issues
Case Mix
Case Mix
The following slide has also been extracted from Marshall and
Spiegelhalter (1998).
They have now applied a statistical model which makes some
adjustment for case mix.
Vertical lines indicate median, top quartile and lower quartile
rankings. How many clinics are clearly very ”good” or very
“bad”
Could this be used with individual surgeon indicators?
21. Outline Introduction Performance Management Data League Tables Issues
Case Mix
Allowing for case mix
22. Outline Introduction Performance Management Data League Tables Issues
Funnel Plots
Funnel Plots
Funnel plots are common in meta-analysis.
The following slide has been extracted from Spiegelhalter
(2002) 4 .
Two hospitals appear to have an unusually high readmission
rate following treatment for a stroke
What adjustment has been made for case mix?
4
Spiegelhalter, D. J. (2002) “Funnel plots for institutional comparison
(letters to the editor)” Qual.Saf. Health Care11:390-391
23. Outline Introduction Performance Management Data League Tables Issues
Funnel Plots
Funnel Plots
24. Outline Introduction Performance Management Data League Tables Issues
Monitoring changes over time
Monitoring changes over time
We return to Marshall and Spiegelhalter (1998)
Having adjusted for case mix, we also try to estimate what
changes have happened over time, along with an associated
uncertainty measure
What are the implications for press-releases heralding a 2.1% drop
in crime, 0.3% drop in road accidents . . . (insert clinical example of
your choosing)?
25. Outline Introduction Performance Management Data League Tables Issues
Monitoring changes over time
Changes over time
26. Outline Introduction Performance Management Data League Tables Issues
Methods from Industrial Quality Control
Quality Control Charts
Rather more has been done looking at longer runs of data
An overview of such charts in healthcare is given by Woodall,
20065 .
The basic idea is stop pompous statisticians taking your data
away and creating over elaborate models which nobody else
understands
The hope is that when such charts are designed carefully,
YOU assess whether anything funny is going on.
5
Woodall, W.H., “The Use of Control Charts in Health-Care and
Public-Health Surveillance” Journal of Quality Technology 38:89-104
27. Outline Introduction Performance Management Data League Tables Issues
Methods from Industrial Quality Control
Cusum Charts
28. Outline Introduction Performance Management Data League Tables Issues
Methods from Industrial Quality Control
Cusum Charts
(well, I had to put at least one piece of maths in somewhere)
T
CUSUM = xt − x0
t=1
The aim of the CUSUM chart is to monitor performance
relative to a target x0 .
Level lines are good, downward slopes are bad, crossing the
V-mask is very bad, especially if you had plenty of warning
that this was going to happen.
Consider the following cusum plot from Chang and McLean
(2006)6 for joint replacement wound blisters.
6
Chang, W.R. and I.P McLean ”CUSUM: A tool for early feedback about
performance?” BMC Medical Research Methodology 6:8
29. Outline Introduction Performance Management Data League Tables Issues
Methods from Industrial Quality Control
Cusum Charts
30. Outline Introduction Performance Management Data League Tables Issues
Outstanding Issues
From HM Treasury7 :
“Performance information is a cornerstone of our
commitment to modernise government. It provides some
of the tools needed to bolster improvements in public
sector performance . . . . . . Good quality information also
enables people to participate in government and exert
pressure for continuous improvement. In addition to
empowering citizens, this information equips managers
and staff within the public service to drive improvement.
Performance information is thus a catalyst for innovation,
enterprise and adaptation.”
7
H.M. Treasury (2001) Choosing the right fabric: A framework for
Performance Information London: HM Treasury
31. Outline Introduction Performance Management Data League Tables Issues
Outstanding issues
So the Treasury believe in:
Driving continuous improvement; a management and a
practitioner tool;
Empowering Citizens
But note:
The Treasury do a lot of driving by controlling finance!
32. Outline Introduction Performance Management Data League Tables Issues
Issues to consider
What do Performance indicators do for patient care?
What do Performance indicators do for clinical practice?
What is our public (Patients / Potential Patients / Local
residents) and how are they served by Performance
Information
Financing the NHS
33. Outline Introduction Performance Management Data League Tables Issues
Provider Help with Preparing SSU Assessments
1) I must not read, mark or correct any piece of SSU written
work (or draft) unless it is sent to me by the SSU
administration team for marking.
2) I must not listen to a verbal presentation in advance or correct
slides prior to an assessed presentation.
3) I may answer any specific question posed to me (by students)
regarding SSU assessment preparation.
4) I am encouraged to give general advice and guidance on how
to write a good written assessment or how to deliver a good
presentation (as appropriate) throughout your SSU provision.
5) Following the marking of the assessment I am free to discuss
with the students any aspect of their assessment that I wish
to.
34. Outline Introduction Performance Management Data League Tables Issues
Recap on assessment
The following aspects of your presentation will be explicitly
considered:
1) Knowledge & understanding of the management problem
2) Research of possible solutions
3) Justification of proposed action
4) Use of appropriate visual aids
5) Quality of report
35. Outline Introduction Performance Management Data League Tables Issues
Your task
Find an area of healthcare subject that is or could be
performance managed
Determine how to gather evidence on the clinical and
“statistical” suitability of different ways of managing
performance