- Real-time monitoring of healthcare services requires defining both a reporting window and data window to accurately capture demand, activity, and wait times. - Using only a reporting window (e.g. a single month) to request data can result in invalid or misleading performance metrics, as it does not account for patients with long wait times. - Defining a larger data window that includes all patients requested before the end of the reporting window and reported after the start avoids this problem, but requires a counterintuitive data request. - Without properly defining both windows, real-time monitoring can provide an inaccurate picture of service performance and falsely suggest the need for more resources.

1. Real-time monitoring and the data trap
Kate Silvester, Healthcare Systems Engineer, Kate Silvester Ltd
Chair: Andrew Hutchings, Assistant Professor, London School
of Hygiene and Tropical Medicine

2. Kate Silvester Ltd 2
Monitoring service performance
• Safety:
• Right care
• % patients receiving right care = process yield
• Flow
• On time, every time, in full
• ‘Right care but too late’
• Flow governed by the rate limiting step (constraint) of the process
• Cost of the time used to add-value/cost of total time the resources
are available = efficiency
• Quality
• Service experience of patients, carers and staff
• System Productivity
• = yield x efficiency
Simon Dodds MA MS FRCS

3. The Process for turning Data to Information:
Define the
Reporting
Window
Define
the Data
Window
Enter the patient
level
data
(Event start and
finish date and
time)
Question the
Request
Comparative
data cf other
services
Real time
time-series
data for
the service
Return the data
+/- report
Front line
staff
perform
the work
Send the request
to an information
analyst
Delay Redefine
the data
window
Compare data
with
Expectations
and Reality
Adjust
our
thinking
(mental
model)
Make a
decision
+/- a
change
Minimum 6 weeks
with HES data
Kate 3

4. Kate Silvester Ltd 4
Issues with the process for turning
Data to Information
1. Delay in the feedback loop.
• Minimum of 6 weeks if we use HES data
• ‘’driving a car using the rear view mirror’’
2. Error in the method for accessing the data

5. Kate
Silvester Ltd
5
Requesting Data for a Performance
Report
• “I need to write the monthly report for June
2015 to show the demand, activity, waiting
times and numbers waiting for my
endoscopy service”
• In a dynamic system, how do you get an accurate
picture of what is
going on using data for a specific time period (e.g.
month)?

6. 01/06/2015 30/06/2015
Reporting window e.g. June 2015
Define the Reporting Window
Then define
the Data
Window:
Kate
Silvester Ltd
6
All patients
requested after
beginning 01/06/15
and reported
before end
30/06/15
?

7. What would the resulting charts
‘tell us’? •
•
•
•
Waiting list numbers are
increasing Lead times are
getting longer Demand >
Activity
QED: ‘We need more
resources’
800
700
600
500
400
300
200
100
0
Numb
ers
waiti
ng
900
June 2015: EndoscopyWaiting
list (Invalid)
25
20
15
10
5
0
Da
ys
30
Endoscopy:lead time in days for
patientsscoped in
June 2015(invalid)
Clinical and front line staff:
Niggle: ‘Something odd’:
‘We know we have done more than 237
endoscopies in 22 days (10/day =
5/day/endoscopy room… we do double this
number…)
Lead times are under 30 days but we are not meeting our
waiting time targets … for either new or surveillance
patients ……the data must be wrong….’
6
0
5
0
4
0
3
0
2
0
1
0
0
01/06/2015
02/06/2015
03/06/2015
04/06/2015
05/06/2015
06/06/2015
07/06/2015
08/06/2015
09/06/2015
10/06/2015
11/06/2015
12/06/2015
13/06/2015
14/06/2015
15/06/2015
16/06/2015
17/06/2015
18/06/2015
19/06/2015
20/06/2015
21/06/2015
22/06/2015
23/06/2015
24/06/2015
25/06/2015
26/06/2015
27/06/2015
28/06/2015
29/06/2015
30/06/2015
Invalid Vitals Chat for June 2015: Endoscopy
demand, activity
dema
nd
Activi
ty
Kate
Silvester Ltd
7

8. The problem
30/06/1501/06/15
The reporting window
Patient lead times
Request Date
(demand)
Procedure Date
(activity)
Long leadtimes:
How far back should I go to capture all the demand and activity, especially for high risk
patients under surveillance?
Kate
Silvester Ltd
8
Due date
Due date
Due date

9. What is the longest lead time?
01/06/15 30/06/15
The reporting window
Patient lead times
Start of the data window
Kate
Silvester Ltd
9
Now…
Or ‘I need the biggest possible data window’…..

10. Kate
Silvester Ltd
1
0
Now
what do
we get
now?
Same
problem!
For the period of the
reporting window, when
does the valid data start and
end?

11. The trap for
Heffalumps
A.A. Milne
www.bookishnature.wordpress.com
Kate
Silvester Ltd
1
1

12. Recognise the Trap for Heffalumps:
• All patients requested after
beginning 01/06/15
• Reported before end 30/06/15
Kate
Silvester Ltd
1
2

13. 08/03/2015 20/09/2015
01/06/2015
Data Window
Reporting window
e.g. June 2015
31/06/2015
1
3
4
6
Correct way: Counter-intuitive request
All patients requested before
end 30/06/15
6
5
4
2
3
Counterintuitive
requestand reported after
01/06/15
Kate Silvester Ltd
1
3
Patient 3
Patient 4
Patient 5
Patient 6
Patient 2
We want the patients
with
Lead time types
Patient 1

14. Valid Vitals Charts® for June 2015
250
200
150
100
50
0
04/06/2014
11/06/2014
18/06/2014
25/06/2014
Number of
requests,
procedures
and
patients
waiting
Valid Vitals Chart® for for
Endoscopy Demand, Activity and
Waiting list for June 2015
dema
nd
activi
ty
WiP =
cumulative diff
What is the story now?
Demand > Activity, WIP has gone
up slightly
Surveillance Patients scoped
at 1 year
New patients at
6/52 Suspected
cancers at 2/52
Kate
Silvester Ltd
1
4

16. Example of invalid updating of real time data
1
6
Tracking patients with long
Lengths of Stay in hospital
‘My patients are disappearing from the
weekly reports when I know they are still
in hospital! Why?’
Practice Nurse
Incorrect data window.

17. How do we monitor a service in real
time?
• The patient level event data are there for real time monitoring
• Can we teach service managers and clinicians to:
• Define the reporting window,
• Then define the data window,
• Counter-intuitive request
• Write their data queries,
• Update their time series charts in their performance reports,
• Interpret their charts correctly?
Kate
Silvester Ltd
1
7

18. The Process for turning Data to Information:
Define the
Reporting
Window
Define
the
Data
Window
Enter the patient level
data
(Event start and
finish date and
time)
Real time
time-series
data for the
service
Front line
staff
perform
the work
Compare data
with
Expectations and
Reality
Adjust
our
thinking
(mental
model)
Make a
decision
+/- a
change
Kate Silvester
Ltd
18
Question
the
Request
Comparative
data cf other
services
Return the data
+/- report
Send the
request to an
information
analyst
Delay
Redefine
the data
window
Minimum 6
weeks with HES
data

19. Questions, discussion and further information
• Journal in Improvement Science (JOIS)
• http://www.improvementscience.net/jois/
• Free
• Need to register
• A Study of the Relative Value of Different Time-Series Charts for Proactive
Process Monitoring. S Dodds 2012
• Diagnosing the Flow Constraint in an Endoscopy Service. Part 1: Recognising
and Avoiding the Data Query Trap. K
Silvester 2016
• Simon Dodd’s FISH and ISP Training programme
• http://www.improvementscience.net/
• kate@katesilvester.co.uk
Kate
Silvester Ltd
19