Your SlideShare is downloading. ×
Performance Indicators in the Health Service
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Performance Indicators in the Health Service

615
views

Published on

Here is an introduction to the SSU

Here is an introduction to the SSU

Published in: Education, Business, Technology

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
615
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
24
Comments
0
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Outline Introduction Performance Management Data League Tables Issues Performance Indicators in the Health Service Paul Hewson1 11th April 1 University of Plymouth, email paul.hewson@plymouth.ac.uk
  • 2. Outline Introduction Performance Management Data League Tables Issues 1 Introduction 2 Performance Management 3 Data 4 League Tables Assessing uncertainty Case Mix Making allowance for the size of an organisation Funnel Plots Monitoring changes over time Methods from Industrial Quality Control 5 Issues
  • 3. Outline Introduction Performance Management Data League Tables IssuesAims of Management SSUs To assess the student’s ability to: Define a health service management problem. Demonstrate an understanding of the historical and contemporary background of the problem. How did things evolve this way? What are the current issues that need addressing and are driving change? Define a strategy and research possible solutions. Propose and justify a particular solution. Present the problem, possible solutions, and proposed solution verbally, including answering questions, Use appropriate visual aids to support the verbal presentation,
  • 4. Outline Introduction Performance Management Data League Tables IssuesAims of Unit Appreciation of methods used in the clinical design of Performance Indicators; How to interpret performance indicators in the context of random fluctuation; How to make allowance for different case mix; AND thinking about this in a real-world context.
  • 5. Outline Introduction Performance Management Data League Tables IssuesIn particular: Performance Indicators are an increasingly important management tool in the health service, consider the very well publicised Healthcare Commission Indicators, QoF indicators in primary care, as well as recent publication of surgeon specific mortality rates. No longer tools imposed from without, in many cases scientific evidence and practitioner input has been used to design a suitable set of measures.
  • 6. Outline Introduction Performance Management Data League Tables IssuesSo we are interested in: Best practice in performance indicator design; How routine clinical information is coded into databases that ultimately becomes a performance indicator; how we assess uncertainty; how we make valid comparison on units (surgeons, hospitals, areas) which may differ due to context or patient case mix; How to satisfy the people paying our wages (or their representatives) that we are delivering continuous improvement in patient care.
  • 7. Outline Introduction Performance Management Data League Tables IssuesAims of today To plan the rest of the unit, contact, logistics, assessment To consider an overview of the role and practice of performance management To present some information on exciting(?) technical aspects Financial aspects will be considered later. We might also want to consider data coding in more detail later. To discuss topics that may be suitable for assessment
  • 8. Outline Introduction Performance Management Data League Tables IssuesAssessment Assessment will consist of: A presentation made to a small audience, Audience may include fellow students, Presentation will be in a semi-formal environment. Any volunteers for video-recording?
  • 9. Outline Introduction Performance Management Data League Tables IssuesAssessment A 20-minute slot should be allowed for each student, approximately: (Up to) 15 minutes for the presentation and to answer questions; 5 minutes for feedback from the audience In addition: 5 minutes should also be allowed in the programme for set up of each presentation. The assessor will need 10 minutes to complete the assessment form including written feedback. colorredElectronic presentations will not necessarily score more than non-electronic ones (OHPs).
  • 10. Outline Introduction Performance Management Data League Tables IssuesPerformance Management: One style among many For the foreseeable future, performance management is here to stay. But two papers (there are plenty more) remind us that there are other management styles: Adab, P., A.M. Rouse, M.A. Mohammed and T. Marhsall (2002) “Performance league tables: the NHS deserves better” Brit.Med.J. 324:95-98 Davies, H. and J. Lampel (1998) “Trust in Performance Indicators” Quality in Health Care 7:159-162
  • 11. Outline Introduction Performance Management Data League Tables IssuesKnown Risks with Performance Management Tunnel vision (ignoring non-measured aspects of a service); Sub-optimisation (setting modest improvement goals); Convergence (aiming to match the average); Gaming (dealing with easiest clients / problems first); Ossification (avoiding innovation); Misrepresentation (see National Audit Office (2001) Inappropriate adjustments to NHS Waiting Lists London: National Audit Office).
  • 12. Outline Introduction Performance Management Data League Tables IssuesData Sources “Public agencies are very keen on amassing statistics - they collect them, add them, raise them to the nth power, take the cube root and prepare wonderful diagrams. But what you must never forget is that every one of those figures comes in the first instance from the village watchman, who just puts down what he damn pleases” Sir Josiah Stamp 1880-1941 (Governor of the Bank of England)
  • 13. Outline Introduction Performance Management Data League Tables IssuesData Quality You can find plenty of other examples. For today, consider the paper by Speigelhalter et al. (2002)2 . Consider in particular: The number of different sources of data recording the same events; The reason for collecting these different data sets; How useful the data were from any of them. 2 Spiegelhalter, D.J., P.Aylin, N.G.Best, S.J.W. Evans and G.D.Murray (2002) ‘ Commissioned analysis of surgical performance using routine data: lessons from the Bristol Enquiry” J.R.Statis.Soc.A 165:191-231
  • 14. Outline Introduction Performance Management Data League Tables IssuesData Validity: are you measuring what you want tomeasure “Not everything that counts is counted, and not everything that can be counted counts” Albert Einstein (approximate quote). A couple of hospital based clinical examples where this has been considered includes: McGlynn, E. and S. Asch (1998) “Developing a clinical performance measure” American Journal of Preventative Medicine 14:14-21 Dorsch M., R. Lawrence, R. Sapsford, J. Oldham, D. Greenwood, B. Jackson, C. Morrell, S. Ball, M. Robinson and A. Hall (2001) “A simple benchmark for evaluating quality of care of patients following acute myocardial infarction” Heart 86:150-154
  • 15. Outline Introduction Performance Management Data League Tables IssuesDesigning PM systems: technical aspects A semi-technical discussion has been prepared by a Royal Statistical Society working party http://www.rss.org.uk/main.asp?page=1222. One example considered are simple pass-fail indicators. Where are these used?
  • 16. Outline Introduction Performance Management Data League Tables IssuesData Validity ∴ a large part of performance indicators surrounds defining them sensibly in the first place. Options for developing an indicator include borrowing a definition from somewhere else: Miles, H., E.Litton, A. Curran, L.Goldsworthy, P.Sharples and A. Henderson (2002) “The PATRIARCH study: Using outcome measures for league tables: Can a North American prediction of admission score be used in a United Kingdom children’s emergency department?” Emerg.Med.J. 19:536-538 as well as quite elaborate procedures for developing a clinical consensus as to what should be measured: Normand, S-L.T., B. McNeil, L. Peterson and R. Palmer (1998) “Methodology matters - VIII. Eliciting expert opinion using the Delphi technique: identifying performance indicators for cardiovasular disease” International Journal for Quality in Health Care 10:247-260
  • 17. Outline Introduction Performance Management Data League Tables IssuesSeveral Famous Problems with league tables Small changes in performance can lead to very large changes in rank; Small organisations more affected than large ones (randomness); There is no allowance for “case” mix or the context in which the organisation operates. One study will be quoted (there are many which report similar results) suggesting that between 1.6% and 2.3% of variation in mortality rate was due to institutional effects: see Merlo, J., P.-O. Ostegren, K. Broms, A. Bjork-Linne, and H. Liedholm (2001) “Survival after initial hospitalisation for heart failure: a multilevel analysis of patients in Swedish acute care hospitals” J. Epidemiol. Community Health 55:323-329.
  • 18. Outline Introduction Performance Management Data League Tables IssuesAssessing uncertaintyUncertainty in League Tables The following slide has been extracted from Marshall and Spiegelhalter (1998)3 , a paper approaching citation classic status in the BMJ. This first chart shows confidence intervals around the raw live birth rate. There are arguments that all Performance Indicators should come with some assessment of the possible uncertainty. What happens with Healthcare Commission Indicators? 3 Marshall, E. C. and D. J. Spiegelhalter (1998) “Reliability of league tables of in vitro fertilisation clinics; retrospective analysis of live birth rates.” Br. Med. J.316:1701-1705
  • 19. Outline Introduction Performance Management Data League Tables IssuesAssessing uncertaintyUncertainty in League Tables
  • 20. Outline Introduction Performance Management Data League Tables IssuesCase MixCase Mix The following slide has also been extracted from Marshall and Spiegelhalter (1998). They have now applied a statistical model which makes some adjustment for case mix. Vertical lines indicate median, top quartile and lower quartile rankings. How many clinics are clearly very ”good” or very “bad” Could this be used with individual surgeon indicators?
  • 21. Outline Introduction Performance Management Data League Tables IssuesCase MixAllowing for case mix
  • 22. Outline Introduction Performance Management Data League Tables IssuesFunnel PlotsFunnel Plots Funnel plots are common in meta-analysis. The following slide has been extracted from Spiegelhalter (2002) 4 . Two hospitals appear to have an unusually high readmission rate following treatment for a stroke What adjustment has been made for case mix? 4 Spiegelhalter, D. J. (2002) “Funnel plots for institutional comparison (letters to the editor)” Qual.Saf. Health Care11:390-391
  • 23. Outline Introduction Performance Management Data League Tables IssuesFunnel PlotsFunnel Plots
  • 24. Outline Introduction Performance Management Data League Tables IssuesMonitoring changes over timeMonitoring changes over time We return to Marshall and Spiegelhalter (1998) Having adjusted for case mix, we also try to estimate what changes have happened over time, along with an associated uncertainty measure What are the implications for press-releases heralding a 2.1% drop in crime, 0.3% drop in road accidents . . . (insert clinical example of your choosing)?
  • 25. Outline Introduction Performance Management Data League Tables IssuesMonitoring changes over timeChanges over time
  • 26. Outline Introduction Performance Management Data League Tables IssuesMethods from Industrial Quality ControlQuality Control Charts Rather more has been done looking at longer runs of data An overview of such charts in healthcare is given by Woodall, 20065 . The basic idea is stop pompous statisticians taking your data away and creating over elaborate models which nobody else understands The hope is that when such charts are designed carefully, YOU assess whether anything funny is going on. 5 Woodall, W.H., “The Use of Control Charts in Health-Care and Public-Health Surveillance” Journal of Quality Technology 38:89-104
  • 27. Outline Introduction Performance Management Data League Tables IssuesMethods from Industrial Quality ControlCusum Charts
  • 28. Outline Introduction Performance Management Data League Tables IssuesMethods from Industrial Quality ControlCusum Charts (well, I had to put at least one piece of maths in somewhere) T CUSUM = xt − x0 t=1 The aim of the CUSUM chart is to monitor performance relative to a target x0 . Level lines are good, downward slopes are bad, crossing the V-mask is very bad, especially if you had plenty of warning that this was going to happen. Consider the following cusum plot from Chang and McLean (2006)6 for joint replacement wound blisters. 6 Chang, W.R. and I.P McLean ”CUSUM: A tool for early feedback about performance?” BMC Medical Research Methodology 6:8
  • 29. Outline Introduction Performance Management Data League Tables IssuesMethods from Industrial Quality ControlCusum Charts
  • 30. Outline Introduction Performance Management Data League Tables IssuesOutstanding Issues From HM Treasury7 : “Performance information is a cornerstone of our commitment to modernise government. It provides some of the tools needed to bolster improvements in public sector performance . . . . . . Good quality information also enables people to participate in government and exert pressure for continuous improvement. In addition to empowering citizens, this information equips managers and staff within the public service to drive improvement. Performance information is thus a catalyst for innovation, enterprise and adaptation.” 7 H.M. Treasury (2001) Choosing the right fabric: A framework for Performance Information London: HM Treasury
  • 31. Outline Introduction Performance Management Data League Tables IssuesOutstanding issues So the Treasury believe in: Driving continuous improvement; a management and a practitioner tool; Empowering Citizens But note: The Treasury do a lot of driving by controlling finance!
  • 32. Outline Introduction Performance Management Data League Tables IssuesIssues to consider What do Performance indicators do for patient care? What do Performance indicators do for clinical practice? What is our public (Patients / Potential Patients / Local residents) and how are they served by Performance Information Financing the NHS
  • 33. Outline Introduction Performance Management Data League Tables IssuesProvider Help with Preparing SSU Assessments 1) I must not read, mark or correct any piece of SSU written work (or draft) unless it is sent to me by the SSU administration team for marking. 2) I must not listen to a verbal presentation in advance or correct slides prior to an assessed presentation. 3) I may answer any specific question posed to me (by students) regarding SSU assessment preparation. 4) I am encouraged to give general advice and guidance on how to write a good written assessment or how to deliver a good presentation (as appropriate) throughout your SSU provision. 5) Following the marking of the assessment I am free to discuss with the students any aspect of their assessment that I wish to.
  • 34. Outline Introduction Performance Management Data League Tables IssuesRecap on assessment The following aspects of your presentation will be explicitly considered: 1) Knowledge & understanding of the management problem 2) Research of possible solutions 3) Justification of proposed action 4) Use of appropriate visual aids 5) Quality of report
  • 35. Outline Introduction Performance Management Data League Tables IssuesYour task Find an area of healthcare subject that is or could be performance managed Determine how to gather evidence on the clinical and “statistical” suitability of different ways of managing performance

×