Performance Indicators in the Health Service

Outline Introduction Performance Management Data League Tables Issues

Performance Indicators in the Health Service

Paul Hewson1

11th April

1
University of Plymouth, email paul.hewson@plymouth.ac.uk


1 Introduction

2 Performance Management

3 Data

4 League Tables
Assessing uncertainty
Case Mix
Making allowance for the size of an organisation
Funnel Plots
Monitoring changes over time
Methods from Industrial Quality Control

5 Issues


Aims of Management SSUs

To assess the student’s ability to:
Deﬁne a health service management problem.
Demonstrate an understanding of the historical and
contemporary background of the problem. How did things
evolve this way? What are the current issues that need
addressing and are driving change?
Deﬁne a strategy and research possible solutions.
Propose and justify a particular solution.
Present the problem, possible solutions, and proposed solution
verbally, including answering questions,
Use appropriate visual aids to support the verbal presentation,


Aims of Unit

Appreciation of methods used in the clinical design of
Performance Indicators;
How to interpret performance indicators in the context of
random ﬂuctuation;
How to make allowance for diﬀerent case mix;
AND thinking about this in a real-world context.


In particular:

Performance Indicators are an increasingly important
management tool in the health service, consider the very well
publicised Healthcare Commission Indicators, QoF indicators
in primary care, as well as recent publication of surgeon
speciﬁc mortality rates.
No longer tools imposed from without, in many cases
scientiﬁc evidence and practitioner input has been used to
design a suitable set of measures.


So we are interested in:

Best practice in performance indicator design;
How routine clinical information is coded into databases that
ultimately becomes a performance indicator;
how we assess uncertainty;
how we make valid comparison on units (surgeons, hospitals,
areas) which may diﬀer due to context or patient case mix;
How to satisfy the people paying our wages (or their
representatives) that we are delivering continuous
improvement in patient care.


Aims of today

To plan the rest of the unit, contact, logistics, assessment
To consider an overview of the role and practice of
performance management
To present some information on exciting(?) technical aspects
Financial aspects will be considered later. We might also want
to consider data coding in more detail later.
To discuss topics that may be suitable for assessment


Assessment

Assessment will consist of:
A presentation made to a small audience,
Audience may include fellow students,
Presentation will be in a semi-formal environment.
Any volunteers for video-recording?


Assessment

A 20-minute slot should be allowed for each student,
approximately:
(Up to) 15 minutes for the presentation and to answer
questions;
5 minutes for feedback from the audience
In addition:
5 minutes should also be allowed in the programme for set up
of each presentation.
The assessor will need 10 minutes to complete the assessment
form including written feedback.
colorredElectronic presentations will not necessarily score more
than non-electronic ones (OHPs).


Performance Management: One style among many

For the foreseeable future, performance management is here to
stay. But two papers (there are plenty more) remind us that there
are other management styles:
Adab, P., A.M. Rouse, M.A. Mohammed and T. Marhsall
(2002) “Performance league tables: the NHS deserves better”
Brit.Med.J. 324:95-98
Davies, H. and J. Lampel (1998) “Trust in Performance
Indicators” Quality in Health Care 7:159-162


Known Risks with Performance Management

Tunnel vision (ignoring non-measured aspects of a service);
Sub-optimisation (setting modest improvement goals);
Convergence (aiming to match the average);
Gaming (dealing with easiest clients / problems first);
Ossification (avoiding innovation);
Misrepresentation (see National Audit Office (2001)
Inappropriate adjustments to NHS Waiting Lists London:
National Audit Office).


Data Sources

“Public agencies are very keen on amassing statistics -
they collect them, add them, raise them to the nth power,
take the cube root and prepare wonderful diagrams. But
what you must never forget is that every one of those
ﬁgures comes in the ﬁrst instance from the village
watchman, who just puts down what he damn pleases”
Sir Josiah Stamp 1880-1941 (Governor of the Bank of England)


Data Quality

You can find plenty of other examples. For today, consider the
paper by Speigelhalter et al. (2002)2 . Consider in particular:
The number of different sources of data recording the same
events;
The reason for collecting these different data sets;
How useful the data were from any of them.

2
Spiegelhalter, D.J., P.Aylin, N.G.Best, S.J.W. Evans and G.D.Murray
(2002) ‘ Commissioned analysis of surgical performance using routine data:
lessons from the Bristol Enquiry” J.R.Statis.Soc.A 165:191-231


Data Validity: are you measuring what you want to
measure
“Not everything that counts is counted, and not
everything that can be counted counts” Albert Einstein
(approximate quote).

A couple of hospital based clinical examples where this has been
considered includes:
McGlynn, E. and S. Asch (1998) “Developing a clinical
performance measure” American Journal of Preventative
Medicine 14:14-21
Dorsch M., R. Lawrence, R. Sapsford, J. Oldham, D.
Greenwood, B. Jackson, C. Morrell, S. Ball, M. Robinson and
A. Hall (2001) “A simple benchmark for evaluating quality of
care of patients following acute myocardial infarction” Heart
86:150-154


Designing PM systems: technical aspects

A semi-technical discussion has been prepared by a Royal
Statistical Society working party
http://www.rss.org.uk/main.asp?page=1222.
One example considered are simple pass-fail indicators. Where are
these used?


Data Validity
∴ a large part of performance indicators surrounds defining them
sensibly in the first place. Options for developing an indicator
include borrowing a definition from somewhere else:
Miles, H., E.Litton, A. Curran, L.Goldsworthy, P.Sharples and
A. Henderson (2002) “The PATRIARCH study: Using
outcome measures for league tables: Can a North American
prediction of admission score be used in a United Kingdom
children’s emergency department?” Emerg.Med.J. 19:536-538
as well as quite elaborate procedures for developing a clinical
consensus as to what should be measured:
Normand, S-L.T., B. McNeil, L. Peterson and R. Palmer
(1998) “Methodology matters - VIII. Eliciting expert opinion
using the Delphi technique: identifying performance indicators
for cardiovasular disease” International Journal for Quality in
Health Care 10:247-260


Several Famous Problems with league tables

Small changes in performance can lead to very large changes
in rank;
Small organisations more aﬀected than large ones
(randomness);
There is no allowance for “case” mix or the context in which
the organisation operates.
One study will be quoted (there are many which report similar
results) suggesting that between 1.6% and 2.3% of variation in
mortality rate was due to institutional eﬀects: see Merlo, J., P.-O.
Ostegren, K. Broms, A. Bjork-Linne, and H. Liedholm (2001)
“Survival after initial hospitalisation for heart failure: a multilevel
analysis of patients in Swedish acute care hospitals” J. Epidemiol.
Community Health 55:323-329.



Uncertainty in League Tables

The following slide has been extracted from Marshall and
Spiegelhalter (1998)3 , a paper approaching citation classic
status in the BMJ.
This ﬁrst chart shows conﬁdence intervals around the raw live
birth rate.
There are arguments that all Performance Indicators should
come with some assessment of the possible uncertainty. What
happens with Healthcare Commission Indicators?

3
Marshall, E. C. and D. J. Spiegelhalter (1998) “Reliability of league tables
of in vitro fertilisation clinics; retrospective analysis of live birth rates.” Br.
Med. J.316:1701-1705



Uncertainty in League Tables


Case Mix

Case Mix

The following slide has also been extracted from Marshall and
Spiegelhalter (1998).
They have now applied a statistical model which makes some
adjustment for case mix.
Vertical lines indicate median, top quartile and lower quartile
rankings. How many clinics are clearly very ”good” or very
“bad”
Could this be used with individual surgeon indicators?


Case Mix

Allowing for case mix


Funnel Plots

Funnel Plots

Funnel plots are common in meta-analysis.
The following slide has been extracted from Spiegelhalter
(2002) 4 .
Two hospitals appear to have an unusually high readmission
rate following treatment for a stroke
What adjustment has been made for case mix?

4
Spiegelhalter, D. J. (2002) “Funnel plots for institutional comparison
(letters to the editor)” Qual.Saf. Health Care11:390-391


Funnel Plots

Funnel Plots




We return to Marshall and Spiegelhalter (1998)
Having adjusted for case mix, we also try to estimate what
changes have happened over time, along with an associated
uncertainty measure
What are the implications for press-releases heralding a 2.1% drop
in crime, 0.3% drop in road accidents . . . (insert clinical example of
your choosing)?



Changes over time



Quality Control Charts

Rather more has been done looking at longer runs of data
An overview of such charts in healthcare is given by Woodall,
20065 .
The basic idea is stop pompous statisticians taking your data
away and creating over elaborate models which nobody else
understands
The hope is that when such charts are designed carefully,
YOU assess whether anything funny is going on.

5
Woodall, W.H., “The Use of Control Charts in Health-Care and
Public-Health Surveillance” Journal of Quality Technology 38:89-104



Cusum Charts



Cusum Charts

(well, I had to put at least one piece of maths in somewhere)
T
CUSUM = xt − x0
t=1

The aim of the CUSUM chart is to monitor performance
relative to a target x0 .
Level lines are good, downward slopes are bad, crossing the
V-mask is very bad, especially if you had plenty of warning
that this was going to happen.
Consider the following cusum plot from Chang and McLean
(2006)6 for joint replacement wound blisters.
6
Chang, W.R. and I.P McLean ”CUSUM: A tool for early feedback about
performance?” BMC Medical Research Methodology 6:8


Outstanding Issues

From HM Treasury7 :
“Performance information is a cornerstone of our
commitment to modernise government. It provides some
of the tools needed to bolster improvements in public
sector performance . . . . . . Good quality information also
enables people to participate in government and exert
pressure for continuous improvement. In addition to
empowering citizens, this information equips managers
and staﬀ within the public service to drive improvement.
Performance information is thus a catalyst for innovation,
enterprise and adaptation.”

7
H.M. Treasury (2001) Choosing the right fabric: A framework for
Performance Information London: HM Treasury


Outstanding issues

So the Treasury believe in:
Driving continuous improvement; a management and a
practitioner tool;
Empowering Citizens
But note:
The Treasury do a lot of driving by controlling ﬁnance!


Issues to consider

What do Performance indicators do for patient care?
What do Performance indicators do for clinical practice?
What is our public (Patients / Potential Patients / Local
residents) and how are they served by Performance
Information
Financing the NHS


Provider Help with Preparing SSU Assessments

1) I must not read, mark or correct any piece of SSU written
work (or draft) unless it is sent to me by the SSU
administration team for marking.
2) I must not listen to a verbal presentation in advance or correct
slides prior to an assessed presentation.
3) I may answer any speciﬁc question posed to me (by students)
regarding SSU assessment preparation.
4) I am encouraged to give general advice and guidance on how
to write a good written assessment or how to deliver a good
presentation (as appropriate) throughout your SSU provision.
5) Following the marking of the assessment I am free to discuss
with the students any aspect of their assessment that I wish
to.


Recap on assessment

The following aspects of your presentation will be explicitly
considered:
1) Knowledge & understanding of the management problem
2) Research of possible solutions
3) Justiﬁcation of proposed action
4) Use of appropriate visual aids
5) Quality of report


Your task

Find an area of healthcare subject that is or could be
performance managed
Determine how to gather evidence on the clinical and
“statistical” suitability of diﬀerent ways of managing
performance

Performance Indicators in the Health Service

More Related Content

Similar to Performance Indicators in the Health Service

Recently uploaded

Performance Indicators in the Health Service