Mark Graban SHS 2014: Two Data Points Are Not a Trend: Using SPC to Manage Better
•Download as PPTX, PDF•
2 likes•4,646 views
Healthcare leaders often make bad decisions due to a lack of statistical understanding. This session will remind attendees that simple comparisons of two data points or comparisons to goals and targets can be misleading. Control charts allow us to better validate project success and make better ongoing management decisions. It’s far too easy for improvement facilitators to draw incorrect conclusions about the success of their Lean event or Six Sigma project if they are simply comparing before and after performance. Likewise, healthcare leaders make bad decisions when they are likewise comparing two data points (today versus a previous month or year or today versus a target). Basic Statistical Process Control (SPC) methods, like control charts, are a simple and proven alternative. Key Learning Objectives 1) Understand some of the common pitfalls in the creation and use of performance measures in various healthcare settings 2) See statistical chart analysis methods that allow for the best management decision making, such as knowing if we are improving and if a "bad day" requires investigation or if it is merely "noise" in the system's performance 3) Connect key principles of Lean management and the Deming philosophy into modern KPI and metrics management By the end of this session attendees will 1) Understand the importance of "control charts" for management decision making 2) Be able to create and interpret a basic management control chart 3) Know of other resources for more learning Mark Graban is author of the Shingo-Award winning book "Lean Hospitals: Improving Quality, Patient Safety, and Employee Engagement." Mark is also co-author, with Joe Swartz, of "Healthcare Kaizen: Engaging Front-Line Staff in Sustainable Continuous Improvements" (also a Shingo recipient) and "The Executive Guide to Healthcare Kaizen." He serves as a consultant to healthcare organizations through his company, Constancy, Inc and is also the Chief Improvement Officer of the technology company KaiNexus. Mark has a B.S. in Industrial Engineering from Northwestern University and an M.S. in Mechanical Engineering and an M.B.A. from the Massachusetts Institute of Technology’s Leaders for Global Operations Program. Mark and his wife live in San Antonio, Texas.
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Mark Graban SHS 2014: Two Data Points Are Not a Trend: Using SPC to Manage Better
Similar to Mark Graban SHS 2014: Two Data Points Are Not a Trend: Using SPC to Manage Better (20)
More from Mark Graban
More from Mark Graban (20)
Recently uploaded
Recently uploaded (20)
Mark Graban SHS 2014: Two Data Points Are Not a Trend: Using SPC to Manage Better
- 1. #shs2014 @MarkGraban Two Data Points Are Not a Trend: Using SPC to Manage Better Mark Graban VP of Innovation & Improvement Services, KaiNexus Slides & Audio: www.MarkGraban.com/SHS2014
- 2. Key Management Questions • How are we performing? – Are we getting better or worse? • What action should we take? Some rights reserved by Marco Bellucci
- 3. Or Not Take Action “Management must understand the theory of variation: If you don’t understand variation and how it comes from the system itself, you can only react to every figure. The result is you often overcompensate, when it would have been better to just leave things alone.” W. Edwards Deming
- 4. My Most Favorite Book Ever Donald J. Wheeler, PhD http://www.spcpress.com/ Amazon: http://bit.ly/wheeler-book
- 5. “No data have meaning apart from their context”
- 6. Comparisons of 2 Data Points Fatalities per 100 Million Vehicle Miles Traveled (U.S. & CT) 2 U.S. 1.5 1 CT 0.5 0 1992 1995 1998 2001 2004 2007 2010
- 7. Need to Look for Trends “You don't want to make a big conclusion based on just one year.” – Jonathan Adkins of the Governors Highway Safety Association “Office Space”
- 8. 2-Point Comparisons in Politics
- 9. % improvement vs. prior year
- 10. Did We Improve?
- 11. Run Charts Show More Context
- 12. 2 Data Points Lack Context • Total triage cycle time was reduced by 23 minutes • Total ED-IP cycle time was reduced by 33 minutes • Average LOS decreased from 97 to 61 minutes Source: Case study, “Harris Methodist saves $648,695 through SIPOC process changes”
- 13. What About the Other Data Points? “The average patient satisfaction increased from 87.2 to 89%”
- 14. The Good News… There is a better way: SPC charts
- 15. “X” Control Chart (Chart for Individual Values) Goal = 25 minutes “Every system is perfectly designed to get the results it gets.” (Deming)
- 16. X and MR Chart Combo
- 17. Applications of SPC Charts • • • • • • Monthly employee attrition % Daily % of patients discharged before 11 am Daily lab test turnaround times Weekly patient satisfaction scores Daily % of on-time case starts Daily % of patients who arrive late / no-show
- 18. Small Business Revenue as a Stable Process? X chart MR chart
- 19. Deming’s 7 Concepts of Variation 1. All variation is caused – specific reasons 2. There are 4 types of causes: 1. 2. 3. 4. Common causes Special causes Tampering Structural 3. Managers must distinguish amongst these – Each one requires different managerial actions
- 20. Deming’s 7 Concepts of Variation 4. For special causes, get timely data 5. For common causes, all data are relevant – In-depth knowledge of the process being improved is needed – statistics, flow charts, Pareto, stratification analysis, DOE 6. When all variation is common cause, the system is said to be “stable” and “predictable” 7. SPC limits let a manager predict future performance with some confidence
- 21. Responding to Daily Changes? 180 160 140 120 100 80 60 40 20 0 Daily Length of Stay Average KB KB Kick Butt PT Praise Team PT Are we helping? Is this process stable? 3/1/12 3/2/12 3/3/12 3/4/12 3/5/12 3/6/12 3/7/12 3/8/12 3/9/12 3/10/12 3/11/12 3/12/12
- 22. So we should do nothing? “Don’t just do something, stand there.” -- Deming
- 23. 3/19/2007 3/17/2007 3/15/2007 3/13/2007 3/11/2007 3/9/2007 3/7/2007 20 3/5/2007 60 3/3/2007 3/1/2007 Creating a Control Chart Upper Control Limit 50 40 30 Lower Control Limit 10 0
- 24. “Western Electric” Rules (1956) • 8 consecutive points on same side of mean • 6 consecutive points moving same direction • 14 alternating up/down points in a row • Any single point above or below 3-sigma LCL or UCL – Full rules http://bit.ly/WErules
- 25. Step 1: Initial Data – LeanBlog.org • Generally need 20 data points to calculate control limits
- 26. Step 2: Mean & MRs • Calculate mean of the first 20 points • Calculate the moving range of the first 20 points – Ex: =ABS(E5-E4)
- 27. Step 3: Draw Initial Chart (with Mean line) LeanBlog.org Daily Page Loads 2500 2000 1500 1000 500 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22
- 28. Step 4: Add Control Limits • Calculate “MR-bar” – Average of the 1st 19 MRs • Calculate Control Limits – LCL = Mean – 3*(MR bar)/1.126 – UCL = Mean + 3*(MR bar)/1.126
- 29. Step 5: Review Chart & Limits LeanBlog.org Daily Page Loads 2500 2000 1500 1000 500 Special Cause? 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22
- 30. Step 6: Revise Limits
- 31. Step 7: Evaluate Over Time LeanBlog.org Daily Page Loads 3000 2500 2000 1500 1000 500 0 1 2 3 4 5 6 7 8 9 101112131415161718192021222324252627282930313233343536373839404142434445464748495051
- 32. Step 7: Evaluate Over Time LeanBlog.org Daily Page Loads 3000 2500 2000 1500 1000 500 0 1 2 3 4 5 6 7 8 9 101112131415161718192021222324252627282930313233343536373839404142434445464748495051
- 33. Step 8: Shift the Limits LeanBlog.org Daily Page Loads 3000 2500 2000 1500 1000 500 0 1 2 3 4 5 6 7 8 9 101112131415161718192021222324252627282930313233343536373839404142434445464748495051
- 34. Testing for Process Shifts • If you made a change that you expected to improve the system, use a control chart to test the hypothesis Daily TAT 35 35 30 30 Process Shift 25 25 20 20 15 15 10 10 5 5 52 52 49 49 46 46 43 43 40 40 37 37 34 34 31 31 28 28 25 25 22 22 19 19 16 16 7 7 13 13 4 4 0 10 10 1 1 0
- 36. NOT Understanding Variation Leads To… • Pressuring people to get better results by working harder within the same system • Wasting time looking for explanations of a perceived trend when nothing has changed • Taking other actions when it would have been better to do nothing • Not focusing on systemic improvements
- 37. Isn’t it always the system? It’s (almost) always the system.
- 38. Q&A / Contact Info • President, Constancy, Inc. – www.constancy.us • VP of Professional Services, KaiNexus – www.KaiNexus.com • Founder, LeanBlog.org – mark@leanblog.org • Twitter @MarkGraban • Books: www.MarkGraban.com
- 39. Backup Slides
- 40. The Funnel Experiment • Lloyd Nelson, 1987 – Suspend a funnel on a stand a few inches off the ground – Drop 50 marbles x
- 41. A “Stable” System • Does NOT mean: – Zero variability – System meets customer requirements • It only means: – Causes of variation are basically constant over time
- 42. We Have to Try Harder!!! • 4 different rules for adjusting the funnel No adjustment Adjust relative to last position Adjust relative to center Learn more – online simulator at http://www.symphonytech.com/dfunnel.htm
Editor's Notes
- Don’t turn off your phones and computers…. Tweet and shareI will post a link to an audio recording of my talk at www.MarkGraban.com/SHS2014 by March 1, 2014
- Question from Lloyd S.Nelson, who worked with Deming -- This is a trick question!
- Story from a reader of my blog… a story that illustrates this point perfectly. Management wastes too much time chasing every up and down (or wastes the time of people who are expected to give an “explanation” for each data point. Reacting to every data point usually INCREASES variation in a process and its results.
- This book is so good, you should go online right now, download the Kindle version, leave my talk and spend 50 minutes reading it
- The slide title is a Wheeler quote from the book – data must have context to be meaningful. It must be understandable. It must be more than just a comparison between actual and target.What is a “quality panel”? Keep in mind this is supposed to be a public metric in the LOBBY for visitors to see. What is the scale? What is a good score? What’s a good score compared to others? Why is the target what it is? The target is SUSPICIOUSLY close to the actual. Comparing a number to a target provides very little context… so does comparing a number to last year or last quarter… as we often see in the news.
- Test scores are down, teen smoking rates are up… Big changes aren't necessarily a signal -- Small changes aren't just noise. 2.9% decline might be somewhat trivial (but good), while a 42% increase in CT *might* be statistically significant. Each state’s “JUMP” could be statistical noise. Look at the chart (made from NHTSA data) – are our roads safer (chart) or more dangerous (headline). A 42% increase in CT – what’s going on there?
- Adkins is completely right… we can’t make big conclusions based on two data points. We don’t want to “jump to conclusions” (ala the mat from Office Space)
- Two data points are used all the time in the news, especially politics. At least this context shows a “margin of error.” How is that same concept useful in our use of data at work? It’s weird that the time scale goes right to left. What looks like an increase from 43 to 47% disapproval, could statistically be a drop from 46% to 44%... Therefore, the media and politicians should not overreact to every up and down in the data.
- This is what’s jokingly called an “Executive time series” with just two data points. -- Take into account noise in the system and common cause variation the picture is much less clear
- It’s hard to know from two data points if things are statistically better… Case study link http://www.caldwellbutler.com/index.php/case-studies/harris-methodist-southwest
- Here is the more complete chart from the case study… Consulting firm case study from a hospital… the early and late data point comparison…. Or you can try the linear trend. But by SPC standards, this is a stable process… not statistically valid improvement
- The more complete method should really include TWO paired charts. We’re looking for a signal in the actual data AND unusual variation from period to period (the bottom MR chart)To create the MR chart, we calculate the “MR bar” – the average of the MR data points (the first 19 MRs if we used 20 data points for the limit calculations). MR-bar is the green line on the MR chart on this slide. The UCL (red line) is 2.67 * MR-bar. The LCL is zero (the MR can’t be negative since it’s an absolute value). We apply the same western electric rules to evaluate if the MR chart is in control (like one data point above the UCL, etc.).
- Why is the MR chart helpful? It finds signals that might be missed from just using the X chart. Why was there a small spike in our revenue? (this is masked data from a wine bar business)
- Driving to work…. How long does it take to get there each day?Read more: http://www.dtic.mil/dtic/tr/fulltext/u2/a238399.pdfStuctural variation = seasonality, for example
- “Management is prediction” – DemingIn a stable system, we can’t just exhort people to do better… we have to change the system and that’s management’s responsibility.
- “Some managers do not get the concept of variability in a process. This example is similar to one I experienced in a hospital. During a meeting of the board, a consulting heart surgeon was presenting data on AMI occurrence. The data showed a normal variation over several months, with an aggregate trend downward. Several members of the board, including the CEO, COO, and the hospital’s process expert voiced concerns that one month’s values were above the average then went down in the following month. This pattern repeated itself, and the individuals wanted to know why all the months did not show a value below the average. The surgeon was well versed in the principles of statistical process control, and he attempted to explain as did I. Alas, to no avail..”
- Here’s a basic control chart that shows a stable process. This means the future data points are predictable (within 23 and 53 or so). The mean (average) is about 38 in this chart.You could use a chart like this for any time of management data…
- These are the most basic rules that would indicate you have something other than common cause variation (or noise). Any one of these rules helps detect a special cause signal.
- This is real data (weekday data) from my website, leanblog.org… if you have more than 20 data points, you could use the first 20-25 to calculate control limits. This approach, because of the data required to calculate limits, works better for daily or weekly data unless you have a lot of historical data.
- In a stable system, we wouldn’t have any data points outside the control limits… that’s the most straightforward case. But, what if we DO have a data point outside the control limit(s)?The simplest rule to look at first is to see if there’s a single data point ABOVE the UCL or BELOW the LCLTopic that people didn’t like? Website down? Stop SOPA DayWe re-calculate limits based on removing that out of control data point
- Special cause of Jan 2 – holiday weekend, so that was eliminated. Not the most stable process, but can generally predict daily page loads would be between 1200 and 2200 without there being any special cause January 6 was a particularly popular post about the need for Kaizen in healthcare – slightly above the UCL
- The first two data points that jump out are the two ABOVE the UCL – there’s likely a special cause that’s actually worth looking for
- Here, we see a group of more than 8 consecutive data points ABOVE the mean – that’s a sign of a special cause
- We recalculate new control limits based on the first 20 data points… and continue evaluating to look for special causes
- This is how I would test if a client is not only sustaining gains, but also improving
- Read about the funnel experiment in Out of the Crisis -- http://books.google.com/books?id=LA15eDlOPgoC&pg=PA327&lpg=PA327&dq=lloyd+nelson+funnel+experiment&source=bl&ots=MsiysUhKJq&sig=GkN-H-tyg5R1MFN2u26Rj4pJjiM&hl=en&sa=X&ei=Dxy_UpE0j-zYBdfMgYAD&ved=0CEQQ6AEwAw#v=onepage&q=lloyd%20nelson%20funnel%20experiment&f=falseSee this online simulator: http://www.symphonytech.com/funnelexp.htm