Measurement for improvement, Patient Safety Team, NHS Improving Quality More at http://www.nhsiq.nhs.uk/improvement-programmes/patient-safety.aspx

1. © NHS Improving Quality 2014© NHS Improving Quality 2014
Measurement for Improvement
Patient Safety Team

2. © NHS Improving Quality 2014
• What are your expectations of
the next three hours?

3. © NHS Improving Quality 2014
Aims and objectives
Aims
• To have a greater knowledge and practical
understanding of Data Collection and Analysis
Objectives
• Understand how to do ‘real’ analysis
• Use data to improve decision-making and identify
real improvements
3

4. © NHS Improving Quality 2014
Introduction
0
50
100
150
200
250
300
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
Distributions – “How long do cars last?”
68.26%
95.44%
99.73%
4

5. © NHS Improving Quality 2014
“Measurement is for improvement not
judgement.”
D. Berwick
Measurement for improvement
5

6. © NHS Improving Quality 2014
“You can’t fatten a cow
by weighing it”
(Palestinian proverb)
Improvement is not about measurement,
but……..
How do we know if a change is an
improvement?
“If you can’t measure it, you can’t improve it”
Measurement for improvement
6

7. © NHS Improving Quality 2014
Model for improvement
7

8. © NHS Improving Quality 2014
Measurement throughout
the project cycle
Project
Identification
Getting a
baseline
Did project
make a
difference
Will project
sustain
Evaluating
worth of the
project
8

9. © NHS Improving Quality 2014
7 Repeat
steps 4-6
Seven steps to measurement
1 Decide
Aim
2 Choose
Measures
3 Define
Measures
4 Collect
Data
5 Analyse
and Present
6 Review
Measures
9

10. © NHS Improving Quality 2014
Step 2 – Choose Measures
10

11. © NHS Improving Quality 2014
Step 2 – What to measure?
Patient experience and
outcomes
Patient
satisfaction
survey
% patients
complication
free in recovery
Pain score
Average time
patient starved
Safety and reliability Clinical incidents Readmissions
Exceptions from
time out
checklist
% correct
equipment to
hand
Efficiency and value Turnaround time
% theatre
utilisation
Cancellations
Delays
(Late starts &
finishes)
Leadership and high
performing teams
Staff survey
Training and
development
Staff turnover Staff absence
11

12. © NHS Improving Quality 2014
Driver Diagrams
Schematic view of a system on the left we depict outcome
As we move right we drill down into the network of causes that drive the
outcome, from ‘primary’ to ‘secondary’ drivers.
Aim: An
improved
system
Primary
Driver
Primary
Driver
Secondary
Driver
Secondary
Driver
Secondary
Driver
Secondary
Driver
Secondary
Driver
12

13. © NHS Improving Quality 2014
Driver Diagrams
On the right we depict ideas for system changes that might ultimately impact the
outcome. Diagram represents our theory about how to modify the system to
change the outcome.
Aim: An
improved
system
Primary
Driver
Primary
Driver
Change
Change
Change
Secondary
Driver
Secondary
Driver
Secondary
Driver
Secondary
Driver
Secondary
Driver
13

14. © NHS Improving Quality 2014
Aim:
2 stones
lighter!
Energy Out
Energy In
Walk daily
commute
Stairs not
lift
Exercise
Reduce
alcohol
intake
Eat Less
Driver Diagrams – weight loss example
14
Pedometer
Gym work
out 3 days
Squash
weekends
No pub
weekdays
Take
packed
lunch
Low fat
meals

15. © NHS Improving Quality 2014
Aim
Defect Free
Surgery
Avoid
Mistakes
Avoid
Complications
Avoid
Delays
Driver Intervention
Conduct team brief
Conduct team debrief
Conduct time out
Produce accurate lists
Implement SSI bundle
Implement VTE bundle
Have correct kit to hand
Ensure staff adequately trained
O1
O2
O3
O1 Overall glitch count
O2 Never events
O3 Number of surgical site infections
P1 % lists with Team Brief
P2 % lists with Team Debrief
P3 % compliance with SSI bundle
P2
P1
P3
Driver Diagrams
clinical example
15

16. © NHS Improving Quality 2014

Exercise: Create your driver
diagram
Thinking about your project
• Start to create a driver diagram and choose measures
• Complete the driver diagram to link
Aims with Measures
You have 20 minutes
16

17. © NHS Improving Quality 2014
Step 2 – Choose Measures
An important note:
As well as clinical and quality measures – you may need
to consider what financial measures are required for
your project?
You may need initial and ongoing funding?
Your success in gaining access to funds will be helped if
you have completed a project financial justification or
return on project investment analysis
Covered in more detail later in the presentation …
17

18. •Measurement
Definitions
18

19. © NHS Improving Quality 2014
Step 3 – Define Measures
• An operational definition is a description, in
quantifiable terms, of what to measure and the
steps to follow to measure it consistently
• Are we measuring the same thing?
19

20. © NHS Improving Quality 2014
Exercise
• M&M Activity

20

21. © NHS Improving Quality 2014
Repeatability
Can you, who created the definition, understand
it and repeat it? Also known as test-retest error,
used as an estimate of short-term
variation
Reproducibility
After repeatability, try seeing if
the definition that you have created
can be reproduced by other people?
Advice on creating
measurement definitions
21

22. © NHS Improving Quality 2014
•The Measures Checklist
•Why important?
•Who owns?
•Definitions?
•Goals?

22

23. © NHS Improving Quality 2014
Step 4 – Collect Data
Practical considerations:
5 W’s and 1H
23

24. © NHS Improving Quality 2014
Step 4 – Collect Data
Decisions, decisions…
• What - All patients or a sample?
• Who – took the data? (what role?)
• When – When was the data taken - real time or
retrospective?
• Where is the data from?
• How – was the data taken - hospital system or
audit?
• Turn the data into a different unit (hours into
days)
24

25. © NHS Improving Quality 2014
Step 5 – Analyse and Present
We will now focus in more detail on methods of
presenting and analysing our chosen measures….
25

26. •Understanding
•and dealing with
• variation in
analysis
26

27. © NHS Improving Quality 2014

The Airplane Game
Using the paper provided – make a paper plane
You have 5 minutes
• When instructed – throw your planes!
• What happened?
• Why are they not flying the same distance?
27
Exercise – The plane game

28. © NHS Improving Quality 2014
Fishbone diagram
Aeroplanes fly
different
distances
Problem
Equipment People Procedures
Measurement
(+Diagnostic)
Communication Materials
Causes
Types of
paper e.g.
card, tracing
paper,
No clear
instructions
provided
Some tables
had scissors,
rulers to help
Skills / ideas
Throwing styles
28

29. © NHS Improving Quality 2014
“We live in a world filled with variation –and yet there is very little
recognition or understanding of variation”
William Scherkenbach
“Data should always be presented in such a way that preserves the
evidence in the data…”
Walter Shewhart
Variation
29
What do people understand by the word variation?

30. © NHS Improving Quality 2014
What does this data tell us?
Patients treated in April
600
550
610
540
560
570
580
590
2008 2009
30

31. © NHS Improving Quality 2014
What does this data tell us?
Patients treated
650
600
550
500
450
400
350
300
April 2008 April 2009
31

32. © NHS Improving Quality 2014
What does this data tell us?
This Month Last Month
Given two different numbers, one will always be bigger than
the other!
SomethingImportant
What action is appropriate?
32

33. © NHS Improving Quality 2014
Common
Cause
Stable in time and therefore
relatively predictable
Paper selection
Persons technique
Design of the plane
Special
Cause
Irregular in time and therefore
unpredictable
Water spill
Fire
Can we classify variation?
33

34. © NHS Improving Quality 2014
It is important to distinguish between the two types of
variation because they require different approaches to deal
with them
Can we classify variation?
“There are different improvement strategies depending
of which type of variation is present (common cause or
special cause), so it is important for a team to know the
difference.”
M.L. George
34

35. © NHS Improving Quality 2014
Step 5 – Analyse and present
Presentation
Dot plot
Individual
value Plot
Interval
Plot
Bar Chart
Run
Chart
SPC
Pareto
Scatter
Plot
Tally Chart
Pie Chart
Radar
Chart
Area
Chart
35
SPC
Run
Chart

36. © NHS Improving Quality 2014
Plotting the dots - example Run Chart
Number of calls to outreach team (weekly)
November 2007 to June 2008
0
NoofCalls
180
160
140
120
100
80
60
40
20
1st Nov 15th Nov 29th Nov 13th Dec 27th Dec 10th Jan 24th Jan 7th Feb 21st Feb 6th Mar 20th Mar 3rd Apr 17th Apr 1st May
Week
Calls per week Median
36

37. © NHS Improving Quality 2014
The Myth of Trends
Upward trend ? Downward trend ?
Downturn ?
Setback ?
Turnaround ?
Rebound?
Static ?
Flatline ?
37

38. © NHS Improving Quality 2014
Time 
Downward trend
Time 
Upward trend
Looking for a trend
7 points all in
upward direction
7 points all in
downward direction
38

39. © NHS Improving Quality 2014
Looking for a trend
7 points above centre line 7 points below centre line
Time 
Below centre
Time 
Above centre
39

40. © NHS Improving Quality 2014
Exercise 7
Creating your own chart
• Using graph paper, ruler and a pencil:
1. Draw and label the axis
2. Plot the dots (daily or weekly data is the best)
3. Work out the median and plot it
4. Add a title (with dates)
5. Add a legend
6. Analyse it!
• You have 20 minutes

40

41. © NHS Improving Quality 2014
100
90
80
70
60
50
40
30
20
10
%
% Compliance with hand hygiene (weekly) April – Sept 2008
Week
1 Apr 15 298 22 6 May13 20 273 Jun10 17 241 Jul8 15 22 295 Aug12 19 262 Sep9 16
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
% Compliance with hand hygiene Medianx
Your chart…
should look like this?
41

42. © NHS Improving Quality 2014
Discussion …
• How could you apply these charts to your projects?

42

43. © NHS Improving Quality 2014

44. •What is SPC?
44

45. © NHS Improving Quality 2014
Statistical Process Control
(SPC) charts…
• …use the pattern of events in the past to predict with some
degree of certainty where future events should fall
• …distinguish between the natural/common cause
• variation and special cause variation
• …enable you to look for problems when they are there, not
when they are not
• …can motivate staff to improve practice thereby reducing
adverse events and minimising variation
45

46. © NHS Improving Quality 2014
Statistical Process Control
(SPC) Charts:
NoofAdmissions
0
200
150
100
50
250
Performance Report – Number of Admissions
Week
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
No Admissions Median Lower Limit (66.5) Upper Limit (222.4)
46

47. © NHS Improving Quality 2014
Two ways to improve a process
If uncontrolled
(special)
variation
Process is unstable and unpredictable
Variation caused by factors outside process
External cause should be identified and
sorted
If controlled
(common)
variation
Process is stable and predictable
Variation is inherent to the process
Therefore the process must be changed
47

48. © NHS Improving Quality 2014
The improvement process
PatientWaitingTime
0
200
150
100
50
250
Performance Report
Week
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
Act Plan
DoStudy
Special causes
present -
unpredictable
Process
predictable (within
control limits)
Process improvement
48

49. © NHS Improving Quality 2014
Three dangers to beware
of …
1. Reacting to special cause variation by changing the
process
2. Ignoring special cause variation by assuming “it’s
part of the process”
3. Comparing more than one process
49

50. •Interpreting
•Charts
50

51. © NHS Improving Quality 2014
LCL
UCL
MEAN
Point above UCL
Special causes
X
X
X
X
X
X
X
X
X
X
Point below LCL
X
X
X
X
X
X
X
X
X
X
Rule 1 - Any point outside one of the control limits
51

52. © NHS Improving Quality 2014
LCL
UCL
MEAN
7 Points above centre line
Special causes
Rule 2 - A run of seven points all above or all below the centre
line, or all increasing or all decreasing
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
7 Points below centre line
52

53. © NHS Improving Quality 2014
LCL
UCL
MEAN
7 Points upward direction
Special causes
Rule 2 - A run of seven points all above or all below the centre
line, or all increasing or all decreasing
7 Points downward direction
X
X
X
X
X X
X
X
X
X X
X
X
X
X
X
X
X
53

54. © NHS Improving Quality 2014
LCL
UCL
MEAN
Cyclic pattern
Special causes
Rule 3 - Any “unusual” pattern or trends within the control limits
Trend pattern
XX
X
X
X
X
X X
X
X
X
X
X
X
X
X
X
X
X
X
X X
X
X
X
X
X
X
X
X
X
X
X
X X
X
X
54

55. © NHS Improving Quality 2014
LCL
UCL
MEAN
Less than 2/3 of all the
points fall in this zone
Special causes
Rule 4
More than 2/3 of all the
points fall in this zone
X
X
X X
X
X
X
X
X
X
X
XX
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
55

56. © NHS Improving Quality 2014
Process out of control
• These rules are important!
• They tell us if the process is stable or unstable
• They tell us if common or special cause
• variation is present
Remember the rules!
• Outside control limits
• Run of 7 or more consecutive points
• Patterns
• Rule of thirds
56

57. © NHS Improving Quality 2014
Exercise 8
Interpreting SPC charts
•Apply the SPC rules to the charts in the handout
• Are there rules present?
• Is the chart in control?

57

58. © NHS Improving Quality 2014
Step 6 – Review Measures
“It is a waste of time collecting and analysing your
data if you don't take action on the results “
58