Measurement Basics

600 views

Published on

This presentation was delivered in session A6 and B6 of Quality Forum 2014 by:

Melanie Rathgeber
Principal
MERGE Consulting

Geoff Schierbeck
Quality Leader
BCPSQC

Published in: Health & Medicine
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
600
On SlideShare
0
From Embeds
0
Number of Embeds
150
Actions
Shares
0
Downloads
3
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Measurement Basics

  1. 1. Building a Measurement Plan for QI Projects Quality Forum 2014 Melanie Rathgeber Geoff Schierbeck Merge Consulting BC Patient Safety & Quality Council Melanie@merge.ca gschierbeck@bcpsqc.ca www.mergeconsulting.ca www.bcpsqc.ca
  2. 2. What is your experience with Measures and Data in Healthcare?
  3. 3. Objectives This course is designed to demonstrate: 1. 2. 3. 4. 5. Guidelines for choosing indicators The use of run charts to display data Importance of sampling Importance of operational definitions Data Display tips
  4. 4. Traditional Problems with Data 1. The data are wrong. 2. The data are too old. 3. Don’t know what is might not be statistically significant? 4. We need to focus on this “outlier” or “trend”.
  5. 5. The data are wrong. Translation: stakeholders don’t agree with the way the data was collected in the first place.
  6. 6. The data are too old. Translation: Things have changed now; things are better now. These results don’t reflect our current environment.
  7. 7. These results might not be statistically significant. Translation: It is important to know what this data means before we jump to conclusions.
  8. 8. We need to focus on this “outlier” or “trend”. Translation: There are results that jump out at us and we naturally feel compelled to fix them.
  9. 9. Resources http://www.ihi.org/knowledge/Pages/Tools/RunChart.aspx Langley GJ, Moen R, Nolan KM, Nolan TW, Norman CL, Provost LP (2009) The Improvement Guide (2nd ed). Provost L, Murray S (2011) The Health Care Data Guide. Perla R, Provost L, Murray S (2013) Sampling Considerations for Health Care Improvement, Q Manage Health Care 22;1: 36–47 Perla R, Provost L, Murray S (2010) The run chart: a simple analytical tool for learning from variation in healthcare processes, BMJ Qual Saf 2011 20: 46-51. Perla R, Provost L, (2012). Judgment sampling: A health care improvement perspective., Q Manage Health Care 21;3: 169-175
  10. 10. On a scale of 1-4, how confident do you feel in building a measurement plan for a QI project? 1 Not at all confident 4 Extremely confident
  11. 11. Voting
  12. 12. Objectives This course is designed to demonstrate: 1. 2. 3. 4. 5. Guidelines for choosing indicators The use of run charts to display data Importance of sampling Importance of operational definitions Data Display tips
  13. 13. QI projects “Systematic, data guided, activities designed to bring about immediate improvement in a health care setting” Lynn et al. The ethics of using Quality Improvement methods in Health Care, Annals of Internal Medicine. 2007; 146: 666-73
  14. 14. Purpose of Data in QI Projects Need to know: - where we started (baseline) - how we change over time (e.g. each week) - when we have reached our target - Not for Accountability (doesn’t go on dashboards or to external agencies) - Not for Research
  15. 15. Model for Improvement What are we trying to accomplish? How will we know that a change is an improvement? Lean/Six Sigma Define Measure What changes can we make that will result in improvement? Analyze Act Plan Study Source: Langley et al. (2009). The Improvement Guide. 2nd edition Do Improve Control
  16. 16. A measurement plan starts with an Aim statement What are we trying to accomplish? Aim Statement How will we know that a change is an improvement? What changes can we make that will result in improvement? Act Study Plan Do
  17. 17. An Aim statements specifies What will improve? When will it improve? How much will it improve? For whom will it improve? Example: The percent of diabetes patients seen by their own GP at Canada Way Clinic will increase from 40% to 95% by May 2014.
  18. 18. Dissecting the Aim statement “Some is not a number; soon is not a time” Donald Berwick, Former CEO of IHI What will improve? When will it improve? How much will it improve? For whom will it improve? Percent of patients seeing their own GP By May 2013 From 15% to 90% Diabetes patients at Canada Way Clinic
  19. 19. Examples – share with partner What will improve? When will it improve? How much will it improve? (numerical goal) For whom will it improve?
  20. 20. What are we trying to accomplish? How will we know that a change is an improvement? What changes can we make that will result in improvement? Act Study Plan Do Family of Measures
  21. 21. Family of Measures Outcome measures  Based on your Aim statement  What is ultimately better?  Voice of the patient/customer Process measures  What are you changing – is it really happening?  Voice of the system – what is being done differently?  Change more quickly than outcomes Balancing measures  What unintended consequences might occur?
  22. 22. Example Aim Statement: The percent of diabetes patients seen by their own GP at Canada Way Clinic will increase from 40% to 95% by May 2013. Outcome Measure: Percent of diabetes patients seen by their own GP Process Measure(s): Balancing Measure(s):
  23. 23. What are we trying to accomplish? How will we know that a change is an improvement? What changes can we make that will result in improvement? Act Study Plan Do Identifying, testing, and implementing changes
  24. 24. Process measures are based on the changes you plan to make: - Changes that are tested and ready to implement: - Example: - Patients are booked for a follow-up before they leave the office
  25. 25. Process Measure(s) 1. Percent of diabetes patients that leave the clinic with their next appointment booked
  26. 26. Example: Balancing Measure Staff are concerned that other patients will have to wait longer for an appointment, if they are not able to be seen on Wednesday morning and Friday afternoon
  27. 27. Example: Balancing Measure 1. Average wait time for non-diabetes patients between calling for an appointment and being seen
  28. 28. On a scale of 1-4, how confident do you feel in building a measurement plan for a QI project? 1 Not at all confident 4 Extremely confident
  29. 29. On a scale of 1-4, how confident do you feel in building a measurement plan for a QI project? 1 Not at all confident 4 Extremely confident
  30. 30. Geoff’s project: Listen for the following: - What were the outcome/process/balancing measures? - How were they chosen? - How was the data useful in driving improvement? - What was the data showing us?
  31. 31. Ophthalmology Turn Over Times
  32. 32. Where do I start? • I have a hunch • I need to determine a target • How am I going to get the information • What do I actually want to accomplish?
  33. 33. I really needed to develop my AIM • What was I going to DO • by WHEN • by HOW MUCH
  34. 34. Aim Statement • By September the ophthalmology clinic turn over time will be decreased to 3 minutes. (feet out to feet in) • Change idea: each person is responsible to perform certain tasks to avoid the questions of who has done what. Maybe need to standardize the process. • Hunch “that there is no consistency in tasks” • I needed to gather the data to see: • What the actual turn over time was • What is the optimal turn over time
  35. 35. Collecting Data
  36. 36. Participation in Checklist 100 90 80 70 60 50 40 30 20 10 0 Dec Jan Feb Mar Apr May Jun Jul Aug Sept Oct Nov Dec Jan Feb
  37. 37. Where did the project start showing improvement? Participation in Checklist 100 90 80 70 60 50 40 30 20 10 0 Dec Jan Feb Mar Apr May Jun Jul Aug Sept Oct Nov Dec Jan Feb
  38. 38. Where did the project start showing improvement? Participation in Checklist 100 90 80 70 60 50 40 30 20 10 0 Dec Jan Feb Mar Apr May Jun Jul Aug Sept Oct Nov Dec Jan Feb
  39. 39. Turn Over Times 12 10 Time in min 8 6 4 Goal 2 0 Dec Jan Feb Mar Apr May Jun Jul Aug Sept Oct Nov Dec Jan Feb
  40. 40. Turn Over Times 12 10 Time in min 8 6 4 2 0 Dec Jan Feb Mar Apr May Jun Jul Aug Sept Oct Nov Dec Jan Feb
  41. 41. Checklist vs Turn Over Times 100 90 80 70 60 50 40 30 20 10 0 Dec Jan Feb Mar Apr May Jun Jul Aug Sept Oct Nov Dec Jan Feb
  42. 42. Partner discussion … - What were the outcome/process/balancing measures? - How were they chosen? - How was the data useful in driving improvement? - What was the data showing us?
  43. 43. On a scale of 1-4, how confident do you feel in building a measurement plan for a QI project? 1 Not at all confident 4 Extremely confident
  44. 44. Voting
  45. 45. Objectives This course is designed to demonstrate: 1. 2. 3. 4. 5. Guidelines for choosing indicators The use of run charts to display data Importance of sampling Importance of operational definitions Data Display tips
  46. 46. Run Charts
  47. 47. A Simple Run Chart: Building Block of Measurement for Improvement Data displayed in time order Data is collected and displayed weekly or monthly. 49
  48. 48. 50 A Simple Run Chart Data displayed in time order. Time is along X axis Data is collected and displayed weekly or monthly. 100% Percent of Patients with Diabetes with Self-Management Goals Documented 80% 60% 40% Centre line = median of the data points or = baseline value -Result along Y axis -One “dot” = one sample of data -Sample size = each “dot” should have the same n 20% 0% April May June July Aug Sept Oct Nov Dec Jan
  49. 49. The Value of Data in Real Time Run Charts can tell us what is happening in real time. 51
  50. 50. The Value of Data Over Time Run Charts detect true patterns and trends over time. Not just what is happening before and after a change. 52
  51. 51. Pre and Post Change Bar Chart – What is the Interpretation?
  52. 52. Let’s Look at the Data in a Run Chart: Scenario 1 Scenario 1 . Pre-post data. 10 8 6 Scenario 1 . Data displayed in a run chart over time. 4 2 0 change made between week 7 and 8 54
  53. 53. Scenario 2 Scenario 2 . Pre-post data. 10 9 8 7 6 5 4 3 2 1 0 Week… Week… Week… Week… Week… Week 9 Week 8 Week 7 Week 6 Week 5 Week 4 Week 3 Week 2 Week 1 Scenario 2 . Data displayed in a run chart over time. change made between week 7 and 8 55
  54. 54. Scenario 3 Scenario 3. Pre-post data. Scenario 3. Data displayed in a run chart over time. change made between week 7 and 8 56
  55. 55. Scenario 4: What if we use a t-test? 80 70 Average Before Change =70.0 60 50 40 Average After Change =30.1 30 20 10 0 The t-test shows a significant difference: Adapted from Perla R.J., Provost L.P., & Murray S.K. (2011). The run chart: a simple analytical tool t(22)=7.6, p<.001 for learning from variation in healthcare processes. BMJ Quality & Safety, 20(1):46-51. 57
  56. 56. 80 70 Average Before Change =70.0 90 80 60 70 60 50 post-change 50 40 Average After Change =30.1 30 40 pre-change 30 20 10 20 0 10 0 change is not sustained t(22)=7.6, p<.001 Adapted from Perla R.J., Provost L.P., & Murray S.K. (2011). The run chart: a simple analytical tool for learning from variation in healthcare processes. BMJ Quality & Safety, 20(1):46-51.
  57. 57. Making a Run Chart: - time along the bottom (X axis) - results along the side (Y axis) - a centre line which is the median of all data points on the chart - each dot on a run chart is the result for: one case or one day or one week or one month
  58. 58. Option 1: Making a Run Chart In Excel
  59. 59. 62 Option 2: Run Chart Template from IHI Run Chart Excel Template http://www.ihi.org/knowledge/Pages/Tools/RunChart.aspx note: you need to complete a one time free registration
  60. 60. Analyzing Run Charts - There are simple rules, based on probability, that are used to determine evidence of improvement in our projects - Interpretation: the rules tell us if there is a non-random pattern in our data. - If we have implemented a change, and we see a non- random pattern (going in the right direction), it is evidence of improvement
  61. 61. A Shift: 6 or more Too many or too few runs A Trend 5 or more An astronomical data point Evidence of a non-random signal if one or more of the circumstances depicted by these four rules are on the run chart. The first three rules are violations of random patterns and are based on a probability of less than 5% chance of occurring just by chance with no change. Perla, Provost, Murray (2011)
  62. 62. Shifts and Trends: A Shift: 6 or more consecutive points all on one side of the median Perla, Provost, Murray (2011) A Trend: 5 or more consecutive points all increasing or all decreasing
  63. 63. Example: The Trend Rule 50 40 30 20 10 0 A Trend: Five or More Consecutive Points All Going in the Same Direction
  64. 64. Example: The Shift Rule 60 Number of Falls in Facility X Jan 2010 - Nov 2011 50 40 Number of Falls 30 20 10 0 Jan-10 Mar-10May-10 Jul-10 Sep-10 Nov-10 Jan-11 Mar-11May-11 Jul-11 Sep-11 Nov-1 Number of falls Baseline median Baseline median extended Jan- Feb- Mar Apr- May Jun- Jul- Aug- Sep- Oct- Nov Dec- Jan- Feb- Mar Apr- May Jun- Jul- Aug- Sep- Oct- Nov 10 10 -10 10 -10 10 10 10 10 10 -10 10 11 11 -11 11 -11 11 11 11 11 11 -11 21 23 24 21 22 24 35 37 32 33 34 25 24 17 20 17 19 21 20 15 14 19 16 24 24 24 24 24 24 24 24 24 24 24 24 24 24 24 24 24 24 24 24 24 24 24 24
  65. 65. Plain Language Interpretation? 60 50 Number of Falls in Facility X Jan 2010 - Nov 2011 40 Number of Falls 30 20 10 0 Jan-10 Apr-10 Jul-10 Oct-10 Number of falls Ma Ma Jan Feb Apr Jun Jul- Aug Sep Oct ry-10 -10 -10 -10 10 -10 -10 -10 10 10 21 23 24 21 22 24 35 37 32 33 Baseline median Jan-11 Apr-11 Jul-11 Oct-11 24 24 24 24 24 24 24 24 24 24 24 Baseline median extended No Ma Ma Apr Jun Jul- Aug Sep Oct Dec Jan Feb vry-10 -11 -11 -11 -11 11 -11 -11 -11 10 11 11 34 25 24 17 20 17 19 21 20 15 14 19 No v11 16 24 24 24 24 24 24 24 24 24 24 24 24 24 There is evidence of improvement – the chance we would see a “shift” like this in data if there wasn’t a real change in what we were doing is less than 5%
  66. 66. On a scale of 1-4, how confident do you feel in building a measurement plan for a QI project? 1 Not at all confident 4 Extremely confident
  67. 67. Voting
  68. 68. Objectives This course is designed to demonstrate: 1. 2. 3. 4. 5. Guidelines for choosing indicators The use of run charts to display data Importance of sampling Importance of operational definitions Data Display tips
  69. 69. Small samples per day or week are okay. Sample size builds over time How much data to satisfy team that it is representative? Simple strategies: - every 5th patient - all patients on Thursday morning If you are reporting externally, or if you want to publish results of QI – may need different strategy See papers by Perla, Provost, and Murray
  70. 70. Objectives This course is designed to demonstrate: 1. 2. 3. 4. 5. Guidelines for choosing indicators The use of run charts to display data Importance of sampling Importance of operational definitions Data Display tips
  71. 71. Operational Definitions Deciding on an operational definition should be done with your QI team What time frame? Which patients? What criteria? What diagnosis? What constitutes “met the guideline?” What about patients that wanted something different? etcetera, etcetera, etcetera ……………………..
  72. 72. Operational Definition Example Basic definition: Patient satisfaction ratings from patient survey
  73. 73. Operational Definition Example Basic definition: Patient satisfaction ratings from patient survey Operational definition: Percent of surgical patients discharged this week that rated their experience with the discharge process as good or excellent, based on the surgical patient survey
  74. 74. “The Data Are Wrong” Not a matter of right versus wrong What is your operational definition? Involve others from the start in this decision.
  75. 75. Objectives This course is designed to demonstrate: 1. 2. 3. 4. 5. Guidelines for choosing indicators The use of run charts to display data Importance of sampling Importance of operational definitions Data Display tips
  76. 76. Data Display Principles
  77. 77. Percent of Forms That Were Illegible 20% 15% 10% 5% 0% April May June July Aug Sept Oct Nov Dec Jan
  78. 78. Percent of Forms That Were Illegible 20% 15% 10% 5% 0% April May June July Aug Sept Oct Nov Dec Jan
  79. 79. Percent of Forms That Were Illegible 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% April May June July Aug Sept Oct Nov Dec Jan
  80. 80. Percent of Forms That Were Illegible 20% 15% 10% 5% 0% April May June July Aug Sept Oct Nov Dec Jan
  81. 81. 40 35 30 25 Large Teaching Hospitals Large Community Hosptials 20 Medium Community Hospitals Small Community Hospitals 15 10 5 0 1 2 3 4 5 6 7 *hypothetical data – illustrative purposes only 8 9 10 11 12 13 14
  82. 82. SMALL MULTIPLES – all info on one page Provincial Readmission Rate 40 30 20 10 0 40 Large Teaching Hospitals Medium Community Hospitals 40 30 20 10 0 30 20 40 30 20 10 0 10 40 30 20 10 0 Small Community Hospitals 2/1/10 1/1/10 12/1/09 11/1/09 10/1/09 9/1/09 8/1/09 7/1/09 6/1/09 5/1/09 4/1/09 3/1/09 2/1/09 1/1/09 0 Large Community Hospitals *hypothetical data – illustrative purposes only
  83. 83. QI Responses to Traditional Problems with Data 1. The data are wrong. 2. The data are too old. 3. Don’t know what is might not be statistically significant? 4. We need to focus on this “outlier” or “trend”.
  84. 84. On a scale of 1-4, how confident do you feel in building a measurement plan for a QI project? 1 Not at all confident 4 Extremely confident
  85. 85. Voting

×