Using Outcomes in the Quality Improvement Process:

299 views

Published on

Published in: Health & Medicine
  • Be the first to comment

  • Be the first to like this

Using Outcomes in the Quality Improvement Process:

  1. 1. Using Outcomes in the Quality Improvement Process: Setting System Level Standards Erik R. Stewart, Ph.D. Vice President, System Performance [email_address] Renee Kopache, M.S. Recovery Coordinator [email_address]
  2. 2. History/Purpose of Outcomes <ul><li>Ohio Mental Health Outcomes Task Force (OTF) – 1996 </li></ul><ul><ul><li>Charge </li></ul></ul><ul><ul><ul><li>Develop a statewide approach to measuring consumer outcomes in Ohio’s publicly-supported mental health system </li></ul></ul></ul><ul><li>Vital Signs (1998) </li></ul><ul><li>Outcomes Implementation Pilot Coordinating Group </li></ul><ul><ul><li>Final recommendations were made in 1999. </li></ul></ul>
  3. 3. Adult Outcomes Domains <ul><li>Clinical Status </li></ul><ul><ul><li>looks at symptoms that a person may experience from their illness and how much they interfere with their daily living. </li></ul></ul><ul><li>Quality of Life </li></ul><ul><ul><li>examines how “good” a person’s life is, and if their needs are being met. Also examines how much control a person has over the events in their life (empowerment). </li></ul></ul>
  4. 4. Adult Outcomes Domains <ul><li>Functional Status </li></ul><ul><ul><li>This domain identifies how well a person is doing in the community including areas such as work, school and social relationships. </li></ul></ul><ul><li>Safety & Health </li></ul><ul><ul><li>Addresses how a person is doing physically and the amount of freedom they have from psychological harm from self and others. </li></ul></ul>
  5. 5. Adult Outcomes: Instruments <ul><li>Adult Consumer Form A </li></ul><ul><ul><li>This form is used by consumers with a severe mental illness. (case management) </li></ul></ul><ul><ul><li>The survey consists of four parts: Quality of Life, Safety and Health, Symptom Distress and Empowerment. </li></ul></ul>
  6. 6. Adult Outcomes: Instruments <ul><li>Adult Provider Form A </li></ul><ul><ul><li>consists of two parts: Functional Status and Safety & Health. </li></ul></ul><ul><ul><li>Provider’s observations and clinical judgments about the consumer’s social & role functioning, housing status, activities of daily living, criminal justice involvement, harmful behavior and victimization. </li></ul></ul>
  7. 7. Adult Outcomes: Instruments <ul><li>Adult Consumer Form B </li></ul><ul><ul><li>Consists of three parts: Quality of Life (excluding empowerment, Safety & Health and Symptom Distress. </li></ul></ul><ul><ul><li>This shorter version of the Adult A form is used for adult outpatient clients. Providers of non-SMD consumers are not required to complete a provider survey. </li></ul></ul>
  8. 8. Youth Outcomes Domains <ul><li>Four primary areas or domains of assessment were selected for Ohio Scales: </li></ul><ul><ul><li>Problem severity </li></ul></ul><ul><ul><li>Functioning </li></ul></ul><ul><ul><li>Hopefulness </li></ul></ul><ul><ul><li>Satisfaction with behavioral health services </li></ul></ul>
  9. 9. Youth Outcomes: Instruments <ul><li>Ohio Scales </li></ul><ul><li>Three parallel forms: </li></ul><ul><ul><li>Y-form is completed by the youth (self-report for ages 12 and older). </li></ul></ul><ul><ul><li>P-form is completed by the youth’s parent (or primary caretaker). </li></ul></ul><ul><ul><li>W-form is completed by the youth’s agency worker/case manager . </li></ul></ul>
  10. 10. Administration Periods <ul><li>Initial </li></ul><ul><li>6 Month </li></ul><ul><li>12 Month </li></ul><ul><li>Annually thereafter </li></ul><ul><li>(Adult Consumer Form B: Initial and Discharge administrations only) </li></ul>
  11. 11. OCO History in Hamilton County <ul><li>Data collection begins – September, 2001 </li></ul><ul><li>Regular and varied production reporting begins at same time </li></ul><ul><li>Production performance abysmal , though consistent with other participating areas of State </li></ul>
  12. 12. HCCMHB surveys agencies to discern impediments to production
  13. 13. Agencies React
  14. 14. HCCMHB representatives visit agencies to discern impediments to production
  15. 16. Impediments (in no particular order): 1) Excuses: <ul><li>POV boxes don’t turn on (batteries) </li></ul><ul><li>Staff refuse to complete or administer </li></ul><ul><li>Multi-million $ agency has only 1 printer </li></ul><ul><li>POV box times out too quickly </li></ul>
  16. 17. Impediments (in no particular order): 2) Collection technology: <ul><li>POV data system interface with other agency data system(s) </li></ul>
  17. 18. Impediments (in no particular order): 3) Lack of agency/staff “buy-in” <ul><li>We know we do good work </li></ul><ul><li>We don’t care about consumer outcomes </li></ul>
  18. 19. Impediments (in no particular order): 4) Lack of products/information to staff following administrations (initially) <ul><li>Staff frustration with initial software </li></ul><ul><li>Introduction of ARROW with use limited to POV </li></ul><ul><li>Immediate red flags printout </li></ul><ul><li>Weekly reports to staff </li></ul><ul><li>Consistent supervisor response to problems </li></ul><ul><li>Use of data in “assessment update printout” </li></ul>
  19. 20. Impediments (in no particular order): 5) Diffusion of responsibility (a social phenomenon that occurs in groups of people when responsibility is not explicitly assigned) <ul><li>Request made by QA/Outcomes member to resurrect regular IT/IS staff meetings at HCCMHB. Solicitation made. ONE response. </li></ul><ul><li>Lack of departmental ownership </li></ul>
  20. 21. Impediments (in no particular order): 6) Agency-specific challenges <ul><li>Central Point of Access </li></ul><ul><li>New administration at agency intake? </li></ul><ul><li>Paper and pencil administration </li></ul><ul><li>Program support staff enter data </li></ul>
  21. 22. QA Approach Introduced <ul><li>Quarterly indicator measuring Initial Adult Provider compliance added to existing indicators </li></ul>
  22. 23. ODMH Introduces Missing Data Reports - Summer, 2003 <ul><li>Hamilton County Performance </li></ul><ul><ul><li>23% Adult OR/OE </li></ul></ul><ul><ul><li>34% Youth OR/OE </li></ul></ul>
  23. 24. October, 2003 - HCCMHB and Agencies Form PI Sub-committee to Develop Production Logic
  24. 25. February, 2004 – Ohio Department of Mental Health Presents State Plan for Upcoming PI Requirements
  25. 26. PDCA Cycle Plan Do Check Act
  26. 27. April, 2004 – Proposed Logic for OCO Production Measures Shared With Membership
  27. 28. Initial Adult Consumer Production Analysis
  28. 29. Initial Adult Consumer Production Analysis <ul><li>Claims data compiled for quarter under review </li></ul>CPST 10006 19 7/25/2005 2052520 Med Som 10005 19 7/20/2005 2052520 CPST 10001 28 9/2/2005 1005002 Counseling 10002 42 8/1/2005 1000001 CPST 10001 42 7/10/2005 1000001 Service Type UPID Min Age First Svc. Date UCI
  29. 30. Initial Adult Consumer Production Analysis <ul><li>“ Pre-quarter period” is determined by subtracting 120 days from first date of service in period </li></ul>3/27/2005 7/25/2005 2052520 3/22/2005 7/20/2005 2052520 5/5/2005 9/2/2005 1005002 4/3/2005 8/1/2005 1000001 3/12/2005 7/10/2005 1000001 Pre-Quarter Period First Svc. Date UCI
  30. 31. Initial Adult Consumer Production Analysis <ul><li>Review of claims during pre-quarter period is conducted. Those exhibiting claims are removed </li></ul>No 3/27/2005 7/25/2005 2052520 No 3/22/2005 7/20/2005 2052520 Yes 5/5/2005 9/2/2005 1005002 No 4/3/2005 8/1/2005 1000001 No 3/12/2005 7/10/2005 1000001 Pre-Quarter Claims Pre-Quarter Period First Svc. Date UCI
  31. 32. Initial Adult Consumer Production Analysis <ul><li>Unduplicate list by selecting case with highest order service </li></ul>CPST 10006 19 3/27/2005 7/25/2005 2052520 Med Som 10005 19 3/22/2005 7/20/2005 2052520 Counseling 10002 42 4/3/2005 8/1/2005 1000001 CPST 10001 42 3/12/2005 7/10/2005 1000001 Service Type UPID Min Age Pre-Quarter Period First Svc. Date UCI
  32. 33. Initial Adult Consumer Production Analysis <ul><li>Create date range for acceptable OCO administrations </li></ul>9/7/2005 5/26/2005 7/25/2005 2052520 8/23/2005 5/11/2005 7/10/2005 1000001 1st Svc. Plus 44 1st Svc. Minus 60 First Svc. Date UCI
  33. 34. Initial Adult Consumer Production Analysis <ul><li>Merge Adult Consumer A and Adult Consumer B OCO administrations </li></ul><ul><li>Match by date when acceptable date exists </li></ul><ul><li>Determine production ratio/percentage </li></ul>
  34. 35. Period under review (4 th qtr. FY 2005) Su Mo Tu We Th Fr Sa 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 Su Mo Tu We Th Fr Sa 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 Su Mo Tu We Th Fr Sa 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 June 2005 May 2005 April 2005 Su Mo Tu We Th Fr Sa 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 Su Mo Tu We Th Fr Sa 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 Su Mo Tu We Th Fr Sa 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 March 2005 February 2005 January 2005
  35. 36. 1 st service in period May 11, 2005 Su Mo Tu We Th Fr Sa 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 Su Mo Tu We Th Fr Sa 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 Su Mo Tu We Th Fr Sa 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 June 2005 May 2005 April 2005 Su Mo Tu We Th Fr Sa 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 Su Mo Tu We Th Fr Sa 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 Su Mo Tu We Th Fr Sa 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 March 2005 February 2005 January 2005
  36. 37. Absence of service determined for 120 day period prior to first service (ensures client is new or in new episode of care) Su Mo Tu We Th Fr Sa 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 Su Mo Tu We Th Fr Sa 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 Su Mo Tu We Th Fr Sa 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 June 2005 May 2005 April 2005 Su Mo Tu We Th Fr Sa 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 Su Mo Tu We Th Fr Sa 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 Su Mo Tu We Th Fr Sa 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 March 2005 February 2005 January 2005
  37. 38. Outcome administration expected/sought between March 12 and June 24 Su Mo Tu We Th Fr Sa 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 Su Mo Tu We Th Fr Sa 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 Su Mo Tu We Th Fr Sa 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 June 2005 May 2005 April 2005 Su Mo Tu We Th Fr Sa 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 Su Mo Tu We Th Fr Sa 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 Su Mo Tu We Th Fr Sa 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 March 2005 February 2005 January 2005
  38. 39. HCCMHB OCO Production Performance Improvement Measures <ul><li>Initial Measures </li></ul><ul><ul><li>Adult Consumer </li></ul></ul>
  39. 40. HCCMHB OCO Production Performance Improvement Measures <ul><li>Initial Measures </li></ul><ul><ul><li>Adult Consumer </li></ul></ul><ul><ul><li>Adult Provider </li></ul></ul><ul><ul><li>Youth Consumer </li></ul></ul><ul><ul><li>Youth Provider </li></ul></ul><ul><ul><li>Youth Parent </li></ul></ul><ul><li>Six-Month Measures </li></ul><ul><ul><li>Adult Consumer </li></ul></ul><ul><ul><li>Adult Provider </li></ul></ul><ul><ul><li>Youth Consumer </li></ul></ul><ul><ul><li>Youth Provider </li></ul></ul><ul><ul><li>Youth Parent </li></ul></ul><ul><li>12-Month Measures </li></ul><ul><ul><li>Adult Consumer </li></ul></ul><ul><ul><li>Adult Provider </li></ul></ul><ul><ul><li>Youth Consumer </li></ul></ul><ul><ul><li>Youth Provider </li></ul></ul><ul><ul><li>Youth Parent </li></ul></ul><ul><li>Long-Term Measures </li></ul><ul><ul><li>Adult Consumer </li></ul></ul><ul><ul><li>Adult Provider </li></ul></ul><ul><ul><li>Youth Consumer </li></ul></ul><ul><ul><li>Youth Provider </li></ul></ul><ul><ul><li>Youth Parent </li></ul></ul>
  40. 41. June, 2004 – Contract Language Established Obligating Agencies to 70% Compliance on OCO Administrations
  41. 42. July, 2004 – HCCMHB Issues First “Test” Measures
  42. 44. PI Products Provided to Agencies:
  43. 45. yes 85.71428571 70 4 24 28 Child Consumer Annual 45 10000 yes 95 70 3 57 60 Child Parent Annual 44 10000 yes 93.5483871 70 4 58 62 Child Provider Annual 43 10000 yes 78.10218978 70 60 214 274 Adult Provider Annual 42 10000 yes 84.30656934 70 43 231 274 Adult Consumer Annual 41 10000 no 0 70 5 0 5 Child Consumer 12 month 35 10000 no 66.66666667 70 4 8 12 Child Parent 12 month 34 10000 no 66.66666667 70 4 8 12 Child Provider 12 month 33 10000 no 42.85714286 70 4 3 7 Adult Provider 12 month 32 10000 no 28.57142857 70 5 2 7 Adult Consumer 12 month 31 10000 no 33.33333333 70 8 4 12 Child Consumer 6 month 25 10000 no 56.52173913 70 10 13 23 Child Parent 6 month 24 10000 yes 78.26086957 70 5 18 23 Child Provider 6 month 23 10000 no 20 70 16 4 20 Adult Provider 6 month 22 10000 no 25 70 15 5 20 Adult Consumer 06 month 21 10000 yes 77.77777778 70 4 14 18 Child Consumer initial 15 10000 yes 86.95652174 70 6 40 46 Child Parent initial 14 10000 yes 84.7826087 70 7 39 46 Child Provider initial 13 10000 no 67.44186047 70 14 29 43 Adult Provider initial 12 10000 yes 82.69230769 70 9 43 52 Adult Consumer initial 11 10000 RatioMet Ratiocalc Ratio expectation Not Submitted Submitted Total Expected Type Type Number UPID
  44. 47. Spreadsheet containing: <ul><li>UCI </li></ul><ul><li>First service in period date </li></ul><ul><li>Last service in period date </li></ul><ul><li>OCO time period parameters </li></ul><ul><li>Submission status </li></ul><ul><li>Date of administration for submission </li></ul><ul><li>Age of client </li></ul><ul><li>Service type by numeric rank </li></ul><ul><li>Instrument type of submission </li></ul><ul><li>Admission date </li></ul>
  45. 48. Agencies may formally dispute findings on a case-specific basis
  46. 49. Measures allow agencies to look closely at specific cases missing administrations to discern issues
  47. 50. Had to educate agencies on intricacies of logic Tracking based upon most recent intake date
  48. 51. Had to cut through many “trivial” issues that impacted few clients but served as a distraction from dealing with larger issues (those effecting many expected administrations)
  49. 52. Had to deal with misunderstandings or “loose” understandings of the administration criteria (ex. Use of “person unable to complete” disposition for unplanned termination/discharges)
  50. 53. Had to move past focusing upon issues related to the logic used in the PI exercise to focus on agency issues needing attention
  51. 54. Had to manage areas subject to manipulation (ex. Use of “B” consumer administrations when “A” is warranted)
  52. 55. ….And in the End….. (Measures of Progress Over Time)
  53. 56. The Heavens Opened!!!!
  54. 58. Measures of Progress Relative to Other Board Areas
  55. 69. <ul><li>Additional “learnings” </li></ul><ul><ul><li>120 day absence without new OCO/tx. Plan update – not clinically sound practice </li></ul></ul><ul><ul><li>Delayed tx. Following assessment </li></ul></ul>
  56. 70. Data Use
  57. 71. Ohio Scales Data
  58. 72. Ohio Scales State Data Report
  59. 73. Youth Consumer Findings
  60. 76. Youth Parent Findings
  61. 79. Youth Worker Findings
  62. 82. Does Hamilton County data look like Statewide data?
  63. 83. Youth Consumer Findings
  64. 86. Youth Parent Findings
  65. 89. Youth Worker Findings
  66. 92. Does a longitudinal design change the picture?
  67. 93. Youth Consumer Findings
  68. 96. Youth Parent Findings
  69. 99. Youth Worker Findings
  70. 102. Does looking at everyone together make sense?
  71. 103. Should the data be apportioned in some manner when considering outcomes? By Diagnosis?
  72. 104. HCCMHB expenditures for treatment of youth with diagnosis of 314.01 (AD/HD) from September 2001 (inception of OCO/OYS) to ~ July, 2005 = $14,470,290 and provided care to 2,652 individuals under the age of 18.
  73. 111. Summary: Findings for all groups (Youth, Parents, & Agency Staff) indicate that statistically significant gains are made during the initial period (6 months) of treatment. Findings did not discern statistically significant differences beyond this point.
  74. 112. Examination of CY 2001 data indicates that of all children with dx.=314.01, 45% receive greater than 180 days of treatment. 25% receive greater than 375 days of treatment. 11% receive greater than 2 years of treatment.
  75. 113. Considerable disparity exists between agencies in regard to the average period of treatment for children with this diagnosis.
  76. 115. Ohio Consumer Outcomes Data Mart http://mhwwwtest.mh.state.oh.us/Screen1/odmhFirstScreen.jsp
  77. 116. Erik R. Stewart, Ph.D. Vice President, System Performance [email_address] Renee Kopache, M.S. Recovery Coordinator [email_address]

×