Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Big e not-so-small data nella pratica ortopedica

54 views

Published on

Talk held by Federico Cabitza at BDH2018, Big Data In Health conference in Salerno (Italy).

Published in: Healthcare
  • Be the first to comment

  • Be the first to like this

Big e not-so-small data nella pratica ortopedica

  1. 1. Big e not-so-Small Data nella pratica ortopedica. L'esperienza orientata alla valutazione del valore presso l'IRCCS Istituto Ortopedico Galeazzi Prof. Ing. Federico Cabitza, PhD IRCCS Istituto Ortopedico Galeazzi Università degli Studi di Milano-Bicocca (Dipartimento di Informatica, Sistemistica e Comunicazione)
  2. 2. Prof. Ing. Federico Cabitza, PhD IRCCS Istituto Ortopedico Galeazzi Università degli Studi di Milano-Bicocca (Dipartimento di Informatica, Sistemistica e Comunicazione) Professore aggregato di «Interazione Uomo-Macchina» e «DataVisualization»
  3. 3. When did the Big Data phenomenon emerge?
  4. 4. When did the Big Data phenomenon emerge? When companies began to collect and mine data from the Internet of Humans (social media)
  5. 5. When did the Big Data phenomenon emerge? When companies began to collect and mine data from the Internet of Humans (social media) Just imagine doing this from the Internet of Things (& humans!)
  6. 6. So what is Big Data?
  7. 7. What is Big Data? Anything that Won't Fit in Excel So what is Big Data? Anything that Won't Fit in Excel!
  8. 8. What is Big Data? Anything that Won't Fit in Excel So what is Big Data? Anything that Won't Fit in Excel! No need for big technology, but the right one!
  9. 9. What is Big Data? Anything that Won't Fit in Excel So what is Big Data? Anything that Won't Fit in Excel! No need for big technology, but the right one! We will focus on PROMs data. My back hurts! 6 on a scale from 1 to 10.
  10. 10. http://archive.is/Yvl4i based health care
  11. 11. http://archive.is/Yvl4i
  12. 12. http://archive.is/Yvl4i = ___Q $
  13. 13. http://archive.is/Yvl4i = ___ $ Q “value is defined as the patient health outcomes achieved per dollar spent”
  14. 14. http://archive.is/Yvl4i = ___ $ Q “value is defined as the patient health outcomes achieved per dollar spent” But this is value-for- money! I just wanna improve!
  15. 15. http://archive.is/Yvl4i
  16. 16. http://archive.is/Yvl4i
  17. 17. http://archive.is/Yvl4i = Outcome Expectations _______________ Inconveniences costs
  18. 18. http://archive.is/Yvl4i = Outcome Expectations _______________ Inconveniences costs Da Lane-Fall, M. B., & Neuman, M. D. (2013). Outcomes measures and risk adjustment. International anesthesiology clinics, 51(4).
  19. 19. http://archive.is/Yvl4i = Outcome Expectations _______________ Inconveniences costs Da Lane-Fall, M. B., & Neuman, M. D. (2013). Outcomes measures and risk adjustment. International anesthesiology clinics, 51(4).
  20. 20. http://archive.is/Yvl4i Outcome Patient Reported Measures
  21. 21. http://archive.is/Yvl4i A measure of what really counts for the patient after a medical intervention, i.e., the outcome in terms of perceived symptoms, recovered function, care satisfaction and current quality of life, as these are seen by the patient (on standard ordinal scales). Outcome Patient Reported Measures
  22. 22. http://archive.is/Yvl4i A measure of what really counts for the patient after a medical intervention, i.e., the outcome in terms of perceived symptoms, recovered function, care satisfaction and current quality of life, as these are seen by the patient (on standard ordinal scales). Outcome Patient Reported Measures
  23. 23. We stratify our patients in those who, 3 months after the operation, claim to have got better and those who assert to have got worse. Surgery at the IRCCS IOG
  24. 24. How to make the concept of improvement measurable, ie how to appraise what the patient values more? We stratify our patients in those who, 3 months after the operation, claim to have got better and those who assert to have got worse. Surgery at the IRCCS IOG
  25. 25. The Oswestry Disability Index expresses annoyance (in terms of disability and quality of life) of patients suffering from back pain. The lower the index the better. The treatment efficacy, and hence the improvement, is reflected by a reduction of this index over time. 1st example: Herniated Disk Surgery The Oswestry Disability Index
  26. 26. The Oswestry Disability Index expresses annoyance (in terms of disability and quality of life) of patients suffering from back pain. The lower the index the better. The treatment efficacy, and hence the improvement, is reflected by a reduction of this index over time. 1st example: Herniated Disk Surgery The Oswestry Disability Index BETTERWORSE DELTASCORE The surgeon proposed a MCSD of 10%. On the ODI this means 10 points.
  27. 27. The Oswestry Disability Index expresses annoyance (in terms of disability and quality of life) of patients suffering from back pain. The lower the index the better. The treatment efficacy, and hence the improvement, is reflected by a reduction of this index over time. 1st example: Herniated Disk Surgery The Oswestry Disability Index BETTERWORSE DELTASCORE The surgeon proposed a MCSD of 10%. On the ODI this means 10 points. Such a MCSD correponds to a size effect of ~0.5 (Cohen d), hence we need ~125 patients to detect it (beta=.8, alpha=.05)
  28. 28. The Oswestry Disability Index expresses annoyance (in terms of disability and quality of life) of patients suffering from back pain. The lower the index the better. The treatment efficacy, and hence the improvement, is reflected by a reduction of this index over time. 1st example: Herniated Disk Surgery The Oswestry Disability Index BETTERWORSE DELTASCORE COMI*: How much did you improved in your health thanks to the surgery? *: CORE OUTCOME MEASURES INDEX  mi ha aiutato molto  mi ha aiutato  non mi ha autato  ha peggiorato la situazione
  29. 29. The average ODI reduction (delta score) for those claiming to have got better (N=109) is -12.54, while the average ODI reduction for those claiming to have got worse (N=22) was -1.59. The difference btw these mean values is statistically significant (p = 0.046). The Oswestry Disability Index expresses annoyance (in terms of disability and quality of life) of patients suffering from back pain. The lower the index the better. The treatment efficacy, and hence the improvement, reflects in a reduction of this index over time. DELTASCORE BETTERWORSE 1st example: Herniated Disk Surgery The Oswestry Disability Index The average ODI reduction (delta score) for those claiming to have got better (N=139) is -14.0, while the average ODI reduction for those claiming to have got worse (N=8) was +8.1. The difference btw these mean values is statistically significant (p = 0.046).
  30. 30. The Oswestry Disability Index expresses annoyance (in terms of disability and quality of life) of patients suffering from back pain. The lower the index the better. The treatment efficacy, and hence the improvement, reflects in a reduction of this index over time. DELTASCORE BETTERWORSE 1st example: Herniated Disk Surgery The Oswestry Disability Index We found a Minimal clinically Important Difference (MID) threshold, which is anchor based and distribution based. This basically confirms the clinical knowledge of the surgeon. This threshold can be considered the upper bound of the CI of the mean improvement: for this pathology: 11 on the ODI.
  31. 31. The Oswestry Disability Index expresses annoyance (in terms of disability and quality of life) of patients suffering from back pain. The lower the index the better. The treatment efficacy, and hence the improvement, reflects in a reduction of this index over time. DELTASCORE BETTERWORSE 1st example: Herniated Disk Surgery The Oswestry Disability Index We found a Minimal clinically Important Difference (MID) threshold, which is anchor based and distribution based. This basically confirms the clinical knowledge of the surgeon. This threshold can be considered the upper bound of the CI of the mean improvement: for this pathology: 11 on the ODI. Effect Size: 0.5 (Hedges’ g)
  32. 32. Three items from COMI back questionnaire were used to assess minimal clinically important difference (MCID) in back and leg pain in patients affected by disc herniation. The first item was used to compute the difference between the pre-operatory pain and the pain at 3 months, like in the previous case. The other two were used to assess pain on a 1- 10 NRS scale, respectively on back and leg. Treatment effectiveness is rendered as a negative delta score.worse better LEG PAIN WORSE BETTER BACK PAIN 2nd example: Herniated Disk Surgery The COMI The surgeon proposed a MCSD of 1.5 points on the NRS.
  33. 33. Three items from COMI back questionnaire were used to assess minimal clinically important difference (MCID) in back and leg pain in patients affected by disc herniation. The first item was used to compute the difference between the pre-operatory pain and the pain at 3 months, like in the previous case. The other two were used to assess pain on a 1- 10 NRS scale, respectively on back and leg. Treatment effectiveness is rendered as a negative delta score.worse better LEG PAIN WORSE BETTER BACK PAIN 2nd example: Herniated Disk Surgery The COMI Such a MCSD correponds to a size effect of ~0.45 (Cohen d), hence we need ~165 patients to detect it (beta=.8, alpha=.05) The surgeon proposed a MCSD of 1.5 points on the NRS.
  34. 34. worse better LEG PAIN WORSE BETTER 2nd example: Herniated Disk Surgery The COMI BACK PAIN Three items from COMI back questionnaire were used to assess minimal clinically important difference (MCID) in back and leg pain in patients affected by disc herniation. The first item was used to compute the difference between the pre-operatory pain and the pain at 3 months, like in the previous case. The other two were used to assess pain on a 1- 10 NRS scale, respectively on back and leg. Treatment effectiveness is rendered as a negative delta score. Back: average score for “perceived worsening” is -0.96 (N = 80), average score for “perceived improvement” is -3.65 (N = 69).The difference is highly statistically significant (p < .001), after a T-test . Leg: average score for “perceived worsening” is -0.07 (N = 60), ), average score for “perceived improvement” is -3.95 (n = 92).The difference is statistically highly significant (p < .001) after a T test.
  35. 35. worse better LEG PAIN WORSE BETTER 2nd example: Herniated Disk Surgery The COMI BACK PAIN Three items from COMI back questionnaire were used to assess minimal clinically important difference (MCID) in back and leg pain in patients affected by disc herniation. The first item was used to compute the difference between the pre-operatory pain and the pain at 3 months, like in the previous case. The other two were used to assess pain on a 1- 10 NRS scale, respectively on back and leg. Treatment effectiveness is rendered as a negative delta score. In this case the PROM-based MCSD is more conservative than the MD’s one. We propose an improvement of at leat 3 points for leg pain, and a slightly lower improvement (2.8) for back pain.
  36. 36.  Oswestry Disability Index  Fabq  Physical activity  Work  Core Outcome Measures Index  SF36  General health  Physical functioning  Role limitation due to physical health  Role limitation due to emotional problems  Social functioning  Pain  Energy/fatigue  Emotional well-being  SRS22  Function  Pain Many available metrics…  VAS (dolore)  Koos-ps/Hoos-ps  Activity  Function  Harris Hip Score  Function  Pain  Deformity  Movement  Knee Society Score  Function  Pain  Stability
  37. 37.  Datareg @ Istituto Ortopedico Galeazzi (up-to-date 5 June 2018)  22 teams  3 specialties: chirurgia spinale (spinal surgery), protesica anca-ginocchio (Hip&Knee replacement surgery), protesica piede-caviglia (foot&ankle replacemente surgery).  Medical users: 104 (spinal) + 120 (others)  First patient: 11 November 2015  Enrolled patients: 1794 (spinal) + 1061 (others)  Yearly enrolled patients: ~3500 (after transitional period)
  38. 38.  Datareg @ Istituto Ortopedico Galeazzi (up-to-date 5 June 2018)  22 teams  3 specialties: chirurgia spinale (spinal surgery), protesica anca-ginocchio (Hip&Knee replacement surgery), protesica piede-caviglia (foot&ankle replacemente surgery).  Medical users: 104 (spinal) + 120 (others)  First patient: 11 November 2015  Enrolled patients: 1794 (spinal) + 1061 (others)  Yearly enrolled patients: ~3500 (after transitional period)  Questionnaires PROMS only: 22.668 (spinal) + 16.138 (others)  PROM items: 501.162 (spinal) 96.628 (others)  Yearly PROM questionnaires: 12-15 k
  39. 39.  Datareg @ Istituto Ortopedico Galeazzi (up-to-date 5 June 2018)  22 teams  3 specialties: chirurgia spinale (spinal surgery), protesica anca-ginocchio (Hip&Knee replacement surgery), protesica piede-caviglia (foot&ankle replacemente surgery).  Medical users: 104 (spinal) + 120 (others)  First patient: 11 November 2015  Enrolled patients: 1794 (spinal) + 1061 (others)  Yearly enrolled patients: ~3500 (after transitional period)  Questionnaires PROMS only: 22.668 (spinal) + 16.138 (others)  PROM items: 501.162 (spinal) 96.628 (others)  Yearly PROM questionnaires: 12-15 k Is this big enough?
  40. 40. Summer 2017 Still too low response rate…
  41. 41. It’s important to reach 80% at each step.
  42. 42. Two main methods for PROM interview.
  43. 43. ~15 miutes per interview. ca 600 interviews Per month / person The importance to make PROM interview efficient and sustainable.
  44. 44. Redemption / Conversion rate Asse X: giorno della settimana di compilazione dei formulari. Asse y: % di questionari compilati in un giorno, normalizzati sulla somma del giorno stesso e il precedente. It’s better to send invitations on Tuesdays and Wednesdays. Fridays must be avoided.
  45. 45. Spine surgery patients, June 2017 OVER TIME PROGRESS
  46. 46. Spine surgery patients, June 2017 Patients who underwent disc surgery get better more quickly and then stabilize (after 3 months). Spinal deformities need more time to show significant improvement (even after 12 months). -ODI DODI OVER TIME PROGRESS
  47. 47. Age P = 0.169 Rho = -0.08 49 Delta ODI (all the pathologies) CORRELATIONS IN GETTING BETTER Spine surgery patients, June 2017
  48. 48. 50 Quality of life (from COMI) P < 0.001 *** Rho = 0.2 CORRELATIONS IN GETTING BETTER Age Spine surgery patients, June 2017
  49. 49. 51 Expectations _______________Outcome P < 0.001 *** Rho = 0.2 Spine surgery patients, June 2017 CORRELATIONS IN GETTING BETTER Age Quality of life (from COMI)
  50. 50. Spine surgery patients, June 2017 Perceived improvement (5-value item) COMI – 3 months Result (Follow-up - 3 months): Excellent / Good / Not so good /Very Bad IMPROVEMENT CORRELATIONS
  51. 51. Spine surgery patients, June 2017 ACCEPTABLE AGREEMENT, BUT… Perceived improvement (5-value item) COMI – 3 months Result (Follow-up - 3 months): Excellent / Good / Not so good /Very Bad IMPROVEMENT CORRELATIONS
  52. 52. Spine surgery patients, June 2017 (score SF36 – 3 month) Perceived improvement (5-value item) COMI – 3 months Result (Follow-up - 3 months): Excellent / Good / Not so good /Very Bad IMPROVEMENT CORRELATIONS
  53. 53. Spine surgery patients, June 2017 (score SF36 – 3 month) Perceived improvement (5-value item) COMI – 3 months Result (Follow-up - 3 months): Excellent / Good / Not so good /Very Bad TO ASSESS THE OUTCOME BY INVOLVING THE PATIENT IS IMPORTANT BECAUSE IT ALLOWS TO DETECT ELEMENTS THAT ARE COMPLEMENTARY TO THOSE OBSERVED BY THE MDS.* IMPROVEMENT CORRELATIONS * See also: Atkinson Tmet al. E. The association between CTCAE and PRO: a systematic review. Supportive Care in Cancer. 2016 Aug 1;24(8):3669-76.
  54. 54. OUTCOME PREDICTION KNEE :: MENTAL SCORE :: 3 MONTHS KNEE :: FUNCTIONAL SCORE :: 3 MONTHS HIP :: MENTAL SCORE :: 3 MONTHS SPINE :: PAIN SCORE :: 3 MONTHS
  55. 55. Target variable:  Delta SF12 mental score (MCS) 0-3 months HIP:: MENTAL SCORE :: 3 MONTHS Prediction of the mental score from the SF12 questionnaire (MCS) of hip surgery (first intervention) after 3 months. Selected features: 1. Age, 2. gender, 3. pre-op HHS deformity, 4. per-op HSS movement, 5. per-op HSS total 6. pre-op SF12 mental score, 7. pre-op SF12 physical score. 8. Pre-op pain VAS 9. Pre-op BMI 10. Preop Hoos-ps activity OUTCOME PREDICTION
  56. 56. No cheating: input variables do not correlate with the target variable. Target variable:  Delta SF12 mental score (MCS) 0-3 months HIP:: MENTAL SCORE :: 3 MONTHS Prediction of the mental score from the SF12 questionnaire (MCS) of hip surgery (first intervention) after 3 months. Selected features: 1. Age, 2. gender, 3. pre-op HHS deformity, 4. per-op HSS movement, 5. per-op HSS total 6. pre-op SF12 mental score, 7. pre-op SF12 physical score. 8. Pre-op pain VAS 9. Pre-op BMI 10. Preop Hoos-ps activity OUTCOME PREDICTION
  57. 57. Target variable:  Delta SF12 mental score (MCS) 0-3 months Prediction of the mental score from the SF12 questionnaire (MCS) of hip surgery (first intervention) after 3 months. Selected features: 1. Age, 2. gender, 3. pre-op HHS deformity, 4. per-op HSS movement, 5. per-op HSS total 6. pre-op SF12 mental score, 7. pre-op SF12 physical score. 8. Pre-op pain VAS 9. Pre-op BMI 10. Preop Hoos-ps activity 11. Pre-op Hoos total 12. Pre-op HSS pain 13. Pre-op HHS function No cheating: worsened patients and those who got better look “mixed”. OUTCOME PREDICTION
  58. 58. The most accurate model is the Random Forest. Target variable:  Delta SF12 mental score (MCS) 0-3 months HIP:: MENTAL SCORE :: 3 MONTHS Prediction of the mental score from the SF12 questionnaire (MCS) of hip surgery (first intervention) after 3 months. Selected features: 1. Age, 2. gender, 3. pre-op HHS deformity, 4. per-op HSS movement, 5. per-op HSS total 6. pre-op SF12 mental score, 7. pre-op SF12 physical score. 8. Pre-op pain VAS 9. Pre-op BMI 10. Preop Hoos-ps activity OUTCOME PREDICTION
  59. 59. Target variable:  Delta SF12 mental score (MCS) 0-3 months HIP:: MENTAL SCORE :: 3 MONTHS Prediction of the mental score from the SF12 questionnaire (MCS) of hip surgery (first intervention) after 3 months. Selected features: 1. Age, 2. gender, 3. pre-op HHS deformity, 4. per-op HSS movement, 5. per-op HSS total 6. pre-op SF12 mental score, 7. pre-op SF12 physical score. 8. Pre-op pain VAS 9. Pre-op BMI 10. Preop Hoos-ps activity Accuracy: 97.9% TP rate: 97.4% FP rate: 0.04% Precision: 98.1% Recall: 97.9% F-score: 98.0% (on test set) 99% OUTCOME PREDICTION The most accurate discriminative model is the Random Forest. AUROC: 99%
  60. 60. HIP:: MENTAL SCORE :: 3 MONTHS Prediction of the mental score from the SF12 questionnaire (MCS) of hip surgery (first intervention) after 3 months. Selected features: 1. Age, 2. gender, 3. pre-op HHS deformity, 4. per-op HSS movement, 5. per-op HSS total 6. pre-op SF12 mental score, 7. pre-op SF12 physical score. 8. Pre-op pain VAS 9. Pre-op BMI 10. Preop Hoos-ps activity OUTCOME PREDICTION The most accurate regression model gives RMSE= 7.5. 7.5RMSE
  61. 61. Today
  62. 62. 64 Tomorrow?
  63. 63. if you don’t datafy it, you cannot …
  64. 64. if you don’t datafy it, you cannot discover lots of interesting patterns!
  65. 65. THE CENTAUR MODEL:THE MD IS AUGMENTED IN HER DECISION MAKING AND OUTCOME ASSESSMENT AND MONITORING. INTELLIGENCE AUGMENTATION , IA (NOT ARTIFICIAL INTELLIGENCE - AI!)
  66. 66. THE CENTAUR MODEL:THE MD IS AUGMENTED IN HER DECISION MAKING AND OUTCOME ASSESSMENT AND MONITORING. INTELLIGENCE AUGMENTATION , IA (NOT ARTIFICIAL INTELLIGENCE - AI!) YET THE HORSE NEEDS TO BE FED!
  67. 67. THE CENTAUR MODEL:THE MD IS AUGMENTED IN HER DECISION MAKING AND OUTCOME ASSESSMENT AND MONITORING. INTELLIGENCE AUGMENTATION , IA (NOT ARTIFICIAL INTELLIGENCE - AI!) YET THE HORSE NEEDS TO BE FED!
  68. 68. GRAZIE! cabitza @ disco.unimib.it @cabitzaf *
  69. 69. These slides are not intended nor produced to be made publicly available. However, I share them to a limited readership (those who possess the Web address where the slides are stored) in an effort of disseminating scientific ideas and reflections, with no intended aim of economic gain, and to get feedback and comments on those ideas, which I expresss under a CC-BY 4.0 license. I took the greatest possible care to identify all image copyright holders correctly. However, if I have omitted to do so in some individual instances, I would be most grateful if these copyright holders would inform me forthwith. It is my policy to immediately remove, upon notification and identification, any specific image displayed on this website for which the copyright holder deems the fair use (see below) cannot be associated. Upon request, I will remove immediately the content and update the content accordingly. FAIR USE NOTICE These slides may contain copyrighted (©) material the use of which has not always been specifically authorized by the copyright owner. Such material is made available to advance understanding of ecological, political, historical, human rights, economic, artistic expression, democracy, scientific, moral, ethical, and social justice issues, etc. It is believed that this constitutes a ’fair use’ of any such copyrighted material as provided for in section 107 of the US Copyright Law. In accordance with Title 17 U.S.C. Section 107, this material is distributed without profit to those who have expressed a prior general interest in receiving similar information for research and educational purposes. ** COPYRIGHT NOTICE** In accordance with Title 17 U.S.C. Se

×