Job Talk

714 views

Published on

Published in: Technology, Education
  • Be the first to comment

Job Talk

  1. 1. enabling eco-feedback and out-of-clinic health sensing indirect University of Washington Eric C. Larson UbiComp Lab electrical engineering computer science and engineering ubiquitous sensing
  2. 2. sensing desired quantity output direct
  3. 3. sensing desired quantity output direct calories burned =
  4. 4. sensing desired quantity output direct calories burned =
  5. 5. sensing desired quantity output direct calories burned =
  6. 6. sensing desired quantity output direct auxiliary quantity sensing calories burned =
  7. 7. sensing desired quantity output direct auxiliary quantity sensing processing calories burned =
  8. 8. sensing desired quantity output direct auxiliary quantity sensing processing estimated calories burned =
  9. 9. indirect sensing
  10. 10. indirect sensing •not exact
  11. 11. indirect sensing •not exact •calibration
  12. 12. indirect sensing •not exact •calibration •low cost
  13. 13. indirect sensing •not exact •calibration •low cost •easier to deploy
  14. 14. indirect sensing •not exact •calibration •low cost •easier to deploy •readily accepted
  15. 15. digital signal processing digital signal processing machine learning machine learning HCI mobile phone embedded references image processing time series evol. comp. ensemble/ graphical HCI mobile phone embedded references thermal imaging CHI 2010 ESPA 2012 facial analysis IJAEC 2010 image fidelity ICIP 2009 JEI 2010 power harvesting UbiComp 2010 gas sensing Pervasive 2010 water sensing UbiComp 2009 Pervasive 2011 CHI 2012 cough sensing UbiComp 2011 lung function UbiComp 2012 DEV 2013 interaction& imageanalysis sustainabilityhealth
  16. 16. digital signal processing digital signal processing machine learning machine learning HCI mobile phone embedded references image processing time series evol. comp. ensemble/ graphical HCI mobile phone embedded references thermal imaging CHI 2010 ESPA 2012 facial analysis IJAEC 2010 image fidelity ICIP 2009 JEI 2010 power harvesting UbiComp 2010 gas sensing Pervasive 2010 water sensing UbiComp 2009 Pervasive 2011 CHI 2012 cough sensing UbiComp 2011 lung function UbiComp 2012 DEV 2013 interaction& imageanalysis sustainabilityhealth
  17. 17. digital signal processing digital signal processing machine learning machine learning HCI mobile phone embedded references image processing time series evol. comp. ensemble/ graphical HCI mobile phone embedded references thermal imaging CHI 2010 ESPA 2012 facial analysis IJAEC 2010 image fidelity ICIP 2009 JEI 2010 power harvesting UbiComp 2010 gas sensing Pervasive 2010 water sensing UbiComp 2009 Pervasive 2011 CHI 2012 cough sensing UbiComp 2011 lung function UbiComp 2012 DEV 2013 interaction& imageanalysis sustainabilityhealth
  18. 18. digital signal processing digital signal processing machine learning machine learning HCI mobile phone embedded references image processing time series evol. comp. ensemble/ graphical HCI mobile phone embedded references thermal imaging CHI 2010 ESPA 2012 facial analysis IJAEC 2010 image fidelity ICIP 2009 JEI 2010 power harvesting UbiComp 2010 gas sensing Pervasive 2010 water sensing UbiComp 2009 Pervasive 2011 CHI 2012 cough sensing UbiComp 2011 lung function UbiComp 2012 DEV 2013 interaction& imageanalysis sustainabilityhealth
  19. 19. digital signal processing digital signal processing machine learning machine learning HCI mobile phone embedded references image processing time series evol. comp. ensemble/ graphical HCI mobile phone embedded references thermal imaging CHI 2010 ESPA 2012 facial analysis IJAEC 2010 image fidelity ICIP 2009 JEI 2010 power harvesting UbiComp 2010 gas sensing Pervasive 2010 water sensing UbiComp 2009 Pervasive 2011 CHI 2012 cough sensing UbiComp 2011 lung function UbiComp 2012 DEV 2013 interaction& imageanalysis sustainabilityhealth
  20. 20. digital signal processing digital signal processing machine learning machine learning HCI mobile phone embedded references image processin g time series evol. comp. ensembl egraphic al HCI mobile phone embedded references thermal imaging CHI 2010 ESPA 2012 facial analysis IJAEC 2010 image fidelity ICIP 2009 JEI 2010 power harvesting UbiComp 2010 gas sensing Pervasive 2010 UbiComp 2009 Pervasive 2011 CHI 2012 cough sensing UbiComp 2011 UbiComp 2012 DEV 2013 interaction imageanalysis sustainabilityhealth water sensing lung function
  21. 21. water sensing lung function
  22. 22. future research SNUPI Static E-Field HydroSense water sensing lung function
  23. 23. future research SNUPI Static E-Field HydroSense water sensing lung function
  24. 24. how can indirect sensing and machine learning be used for sustainability?
  25. 25. we are using water faster than it is being replenished Pacific Institute for Studies in Development, Environment, and Security, 2011
  26. 26. we are using water faster than it is being replenished Pacific Institute for Studies in Development, Environment, and Security, 2011
  27. 27. $2,994.83
  28. 28. water usage is vastly misunderstood
  29. 29. eco-feedback Geographic Comparisons Dashboards Metaphorical Unit Designs Recommendations
  30. 30. eco-feedback Geographic Comparisons Dashboards Metaphorical Unit Designs Recommendations
  31. 31. eco-feedback in electricity
  32. 32. 0% 5% 10% 15% 20% 1 2 3 4 5 Untitled 1 20% 12% 9.2%8.4% 6.8% 3.8% Enhanced Billing Web Based Daily Feedback Realtime Feedback Appliance Level + Personalized Feedback Annual%Savings Based on 36 studies between 1995-2010. Summarized by Ehrhardt-Matinez et al. >20% reduction: Gardner et al. (2008) and Laitner et al. (2009) Appliance Level eco-feedback in electricity
  33. 33. 0% 5% 10% 15% 20% 1 2 3 4 5 Untitled 1 20% 12% 9.2%8.4% 6.8% 3.8% Enhanced Billing Web Based Daily Feedback Realtime Feedback Appliance Level + Personalized Feedback Annual%Savings Based on 36 studies between 1995-2010. Summarized by Ehrhardt-Matinez et al. >20% reduction: Gardner et al. (2008) and Laitner et al. (2009) Appliance Level eco-feedback in electricity aggregate
  34. 34. 0% 5% 10% 15% 20% 1 2 3 4 5 Untitled 1 20% 12% 9.2%8.4% 6.8% 3.8% Enhanced Billing Web Based Daily Feedback Realtime Feedback Appliance Level + Personalized Feedback Annual%Savings Based on 36 studies between 1995-2010. Summarized by Ehrhardt-Matinez et al. >20% reduction: Gardner et al. (2008) and Laitner et al. (2009) Appliance Level eco-feedback in electricity aggregate disaggregated
  35. 35. Courtesy: Belkin, Inc.
  36. 36. Courtesy: Belkin, Inc.
  37. 37. meters flow rate fixture flow inline water
  38. 38. meters flow rate fixture flow inline water water pressure pressure sensor
  39. 39. meters flow rate fixture flow inline water water pressure pressure sensor
  40. 40. meters flow rate fixture flow inline water water pressure pressure sensor machine learning estimated
  41. 41. HydroSense
  42. 42. • single sensor HydroSense
  43. 43. • single sensor • easy to install HydroSense
  44. 44. • single sensor • easy to install • low cost HydroSense
  45. 45. • single sensor • easy to install • low cost • can observe every fixture HydroSense
  46. 46. HydroSense
  47. 47. HydroSense
  48. 48. 40# 50# 60# 70# 80# Cold Line Pressure (Hose Spigot) 0 94.5 time (s) psi HydroSense
  49. 49. 40# 50# 60# 70# 80# Cold Line Pressure (Hose Spigot) 0 94.5 time (s) psi HydroSense
  50. 50. 40# 50# 60# 70# 80# Cold Line Pressure (Hose Spigot) 0 94.5 time (s) psi HydroSense open close
  51. 51. 40# 50# 60# 70# 80# Cold Line Pressure (Hose Spigot) 0 94.5 time (s) psi HydroSense open close kitchen sink
  52. 52. 40# 50# 60# 70# 80# Cold Line Pressure (Hose Spigot) 0 94.5 time (s) psi HydroSense open close kitchen sink upstairs toilet
  53. 53. 40# 50# 60# 70# 80# Cold Line Pressure (Hose Spigot) 0 94.5 time (s) psi HydroSense open close kitchen sink downstairs toilet upstairs toilet
  54. 54. kitchen sink upstairs toilet downstairs toilet template matching unknown event
  55. 55. kitchen sink upstairs toilet downstairs toilet template matching unknown event
  56. 56. initial study Froehlich, J., Larson, E., Campbell, T., Haggerty, C., Fogarty, J., and Patel, S.N. HydroSense: infrastructure-mediated single-point sensing of whole-home water activity. Proceedings of the 11th ACM international conference on Ubiquitous computing, (2009), 235–244. Larson, E., Froehlich, J., Campbell, T., et al. Disaggregated water sensing from a single, pressure- based sensor: An extended analysis of HydroSense using staged experiments. Pervasive and Mobile Computing, (2010).
  57. 57. initial study • 10 homes Froehlich, J., Larson, E., Campbell, T., Haggerty, C., Fogarty, J., and Patel, S.N. HydroSense: infrastructure-mediated single-point sensing of whole-home water activity. Proceedings of the 11th ACM international conference on Ubiquitous computing, (2009), 235–244. Larson, E., Froehlich, J., Campbell, T., et al. Disaggregated water sensing from a single, pressure- based sensor: An extended analysis of HydroSense using staged experiments. Pervasive and Mobile Computing, (2010).
  58. 58. initial study • 10 homes • staged calibration Froehlich, J., Larson, E., Campbell, T., Haggerty, C., Fogarty, J., and Patel, S.N. HydroSense: infrastructure-mediated single-point sensing of whole-home water activity. Proceedings of the 11th ACM international conference on Ubiquitous computing, (2009), 235–244. Larson, E., Froehlich, J., Campbell, T., et al. Disaggregated water sensing from a single, pressure- based sensor: An extended analysis of HydroSense using staged experiments. Pervasive and Mobile Computing, (2010).
  59. 59. initial study • 10 homes • staged calibration • ~98% accuracy at identifying fixtures Froehlich, J., Larson, E., Campbell, T., Haggerty, C., Fogarty, J., and Patel, S.N. HydroSense: infrastructure-mediated single-point sensing of whole-home water activity. Proceedings of the 11th ACM international conference on Ubiquitous computing, (2009), 235–244. Larson, E., Froehlich, J., Campbell, T., et al. Disaggregated water sensing from a single, pressure- based sensor: An extended analysis of HydroSense using staged experiments. Pervasive and Mobile Computing, (2010).
  60. 60. 70 50 30 pressure(psi) 70 50 30 pressure(psi) initial study: staged events
  61. 61. 70 50 30 pressure(psi) 70 50 30 pressure(psi) initial study: staged events kitchen sink kitchen sink
  62. 62. 70 50 30 pressure(psi) 70 50 30 pressure(psi) natural water use
  63. 63. how well does HydroSense work in a natural setting?
  64. 64. longitudinal evaluation
  65. 65. totals days 33 33 30 27 33 156 events 2374 3075 4754 2499 2578 14,960 events/day 71.9 93.2 158.5 92.6 78.1 95.9 compound 22.2% 21.8% 16.6% 32% 21.3% 21.9% data collection Larson, E., Froehlich, J., Saba, E., et al. A Longitudinal Study of Pressure Sensing to Infer Real- World Water Usage Events in the Home. Pervasive Computing, Springer (2011), 50–69.
  66. 66. totals days 33 33 30 27 33 156 events 2374 3075 4754 2499 2578 14,960 events/day 71.9 93.2 158.5 92.6 78.1 95.9 compound 22.2% 21.8% 16.6% 32% 21.3% 21.9% data collection Larson, E., Froehlich, J., Saba, E., et al. A Longitudinal Study of Pressure Sensing to Infer Real- World Water Usage Events in the Home. Pervasive Computing, Springer (2011), 50–69. most comprehensive labeled dataset of hot and cold water ever collected
  67. 67. labeled natural water usage 70 50 30 pressure(psi) 70 50 30 pressure(psi)
  68. 68. kitchen sink kitchen sink toilet bathroom sink labeled natural water usage 70 50 30 pressure(psi) 70 50 30 pressure(psi)
  69. 69. kitchen sink kitchen sink toilet bathroom sink labeled natural water usage 70 50 30 pressure(psi) 70 50 30 pressure(psi) template matching: 98% 70%
  70. 70. kitchen sink kitchen sink toilet bathroom sink labeled natural water usage 70 50 30 pressure(psi) 70 50 30 pressure(psi) template matching: 98% 70% 10 fold cross validation
  71. 71. a new approach
  72. 72. a new approach templates feature vectors
  73. 73. a new approach templates feature vectors matching statistical approach
  74. 74. a new approach templates feature vectors matching statistical approach minimal training
  75. 75. 70 50 30 pressure(psi) 70 50 30 pressure(psi) feature vectors
  76. 76. 70 50 30 pressure(psi) 70 50 30 pressure(psi) feature vectors
  77. 77. 70 50 30 pressure(psi) 70 50 30 pressure(psi) feature vectors
  78. 78. 70 50 30 pressure(psi) 70 50 30 pressure(psi) feature vectors
  79. 79. feature vectors
  80. 80. feature vectors
  81. 81. feature vectors
  82. 82. 10 psi feature vectors
  83. 83. 10 psi 7.32 psi feature vectors
  84. 84. 10 psi 7.32 psi 15 Hz feature vectors
  85. 85. 10 psi 7.32 psi 15 Hz 200 ms feature vectors
  86. 86. 10 psi 7.32 psi 15 Hz 200 ms feature vectors
  87. 87. 10 psi 7.32 psi 15 Hz 200 ms feature vectors
  88. 88. 70 50 30 pressure(psi) 70 50 30 pressure(psi) feature vectors
  89. 89. 70 50 30 pressure(psi) 70 50 30 pressure(psi)
  90. 90. 70 50 30 pressure(psi) 70 50 30 pressure(psi)
  91. 91. 70 50 30 pressure(psi) 70 50 30 pressure(psi) feature vectors: sequence
  92. 92. 70 50 30 pressure(psi) 70 50 30 pressure(psi) feature vectors: sequence
  93. 93. y1 y2 y3 y4 y5 y6 x1 x2 x3 x4 x5 x6 statistical model observed hidden
  94. 94. y1 y2 y3 y4 y5 y6 x1 x2 x3 x4 x5 x6 statistical model p(x, y) = TY t=1 p(yt|yt 1) | {z } transition p(xt|yt) | {z } emission observed hidden
  95. 95. y1 y2 y3 y4 y5 y6 x1 x2 x3 x4 x5 x6 supervised training p(x, y) = TY t=1 p(yt|yt 1) | {z } transition p(xt|yt) | {z } emission observed observed
  96. 96. calibration, per home a few days of training labels
  97. 97. calibration, per home a few days of training labels remainder of data is test set
  98. 98. fixturelevelaccuracy(%) training amount (days)
  99. 99. 0 2 4 6 8 10 12 14 16 40 50 60 70 80 90 100 fixturelevelaccuracy(%) training amount (days)
  100. 100. 0 2 4 6 8 10 12 14 16 40 50 60 70 80 90 100 fixturelevelaccuracy(%) training amount (days) ideally
  101. 101. 0 2 4 6 8 10 12 14 16 40 50 60 70 80 90 100 fixturelevelaccuracy(%) training amount (days) ideally error bars = ±error bars
  102. 102. y1 y2 y3 y4 y5 y6 x1 x2 x3 x4 x5 x6 p(x, y) = TY t=1 p(yt|yt 1) | {z } transition p(xL t |yt) | {z } emission p(xU t ) | {z } unlabeled semi-supervised training
  103. 103. y1 y2 y3 y4 y5 y6 x1 x2 x3 x4 x5 x6 semi-supervised training p(x, y) = TY t=1 p(yt|yt 1) | {z } transition p(xL t |yt) | {z } emission p(xU t ) | {z } unlabeled
  104. 104. y1 y2 y3 y4 y5 y6 x1 x2 x3 x4 x5 x6 semi-supervised training p(x, y) = TY t=1 p(yt|yt 1) | {z } transition p(xL t |yt) | {z } emission p(xU t ) | {z } unlabeled
  105. 105. y1 y2 y3 y4 y5 y6 x1 x2 x3 x4 x5 x6 semi-supervised training p(x, y) = TY t=1 p(yt|yt 1) | {z } transition p(xL t |yt) | {z } emission p(xU t ) | {z } unlabeled
  106. 106. 40 50 60 70 80 90 100 0 2 4 6 8 10 12 14 16 fixturelevelaccuracy(%) training amount (days) error bars = ±error bars
  107. 107. 40 50 60 70 80 90 100 0 2 4 6 8 10 12 14 16 fixturelevelaccuracy(%) training amount (days) error bars = ±error bars
  108. 108. y1 y2 y3 y4 y5 y6 x1 x2 x3 x4 x5 x6 decoding: add pairing
  109. 109. 1 y1 y2 y3 y4 y5 y6 x1 x2 x3 x4 x5 x6 decoding: add pairing
  110. 110. 11111 y1 y2 y3 y4 y5 y6 x1 x2 x3 x4 x5 x6 decoding: add pairing
  111. 111. 40 50 60 70 80 90 100 0 2 4 6 8 10 12 14 16 fixturelevelaccuracy(%) training amount (days) error bars = ±error bars
  112. 112. 40 50 60 70 80 90 100 0 2 4 6 8 10 12 14 16 fixturelevelaccuracy(%) training amount (days) error bars = ±error bars
  113. 113. Currently Using 0.00 GPM
  114. 114. Currently Using 0.00 GPM
  115. 115. Static E-Field HydroSenseHydroSense • low cost • easily installed • accurate • quickly calibrated • potential for high impact
  116. 116. future research SNUPI Static E-Field HydroSense water sensing lung function
  117. 117. future research SNUPI Static E-Field HydroSense water sensing lung function
  118. 118. how can indirect sensing and machine learning be used for health?
  119. 119. SpiroSmart a smartphone based spirometer that leverages on-device microphone to help you keep track of your lung function.
  120. 120. SpiroSmart a smartphone based spirometer that leverages on-device microphone to help you keep track of your lung function.lung function spirometer
  121. 121. spirometer device that measures amount of air inhaled and exhaled.
  122. 122. lung function asthma COPD cystic fibrosis evaluates severity of pulmonary impairments
  123. 123. using a spirometer flow volume volume time
  124. 124. using a spirometer flow volume volume time
  125. 125. using a spirometer flow volume volume time
  126. 126. volume-time graph volume time
  127. 127. volume-time graph volume time
  128. 128. volume-time graph volume time FEV1 FVC FEV1: Forced Expiratory Volume in 1 second FVC: Forced Vital Capacity
  129. 129. volume-time graph volume time1 sec. FEV1 FVC FEV1: Forced Expiratory Volume in 1 second FVC: Forced Vital Capacity
  130. 130. volume-time graph volume time1 sec. FEV1 FVC FEV1% = FEV1/FVC FEV1: Forced Expiratory Volume in 1 second FVC: Forced Vital Capacity
  131. 131. FEV1: Forced Expiratory Volume in 1 second FVC: Forced Vital Capacity FEV1% = FEV1/FVC
  132. 132. FEV1: Forced Expiratory Volume in 1 second FVC: Forced Vital Capacity FEV1% = FEV1/FVC > 80% healthy 60 - 79% mild 40 - 59% moderate < 40% severe
  133. 133. flow-volume graph flow volume
  134. 134. flow-volume graph flow volume
  135. 135. flow volume FEV1 FVC 1 sec. PEF PEF: Peak Expiratory Flow FEV1: Forced Expiratory Volume in 1 second FVC: Forced Vital Capacity flow-volume graph
  136. 136. flow volume normal flow-volume graph
  137. 137. flow volume normal obstructive flow-volume graph
  138. 138. obstructive diseases resistance in air path leads to reduced air flow
  139. 139. obstructive diseases resistance in air path leads to reduced air flow
  140. 140. restrictive diseases lungs are unable to pump enough air and pressure
  141. 141. restrictive diseases lungs are unable to pump enough air and pressure
  142. 142. flow-volume graph Flow Volume normal obstructive
  143. 143. flow-volume graph Flow Volume normal restrictive obstructive
  144. 144. clinical spirometry
  145. 145. home spirometry
  146. 146. home spirometry faster detection rapid recovery trending
  147. 147. home spirometry high cost barrier patient compliance less coaching limited integration challenges with
  148. 148. flow rate volume lung function airflow sensor
  149. 149. flow rate volume lung function airflow sensor sound pressure microphone
  150. 150. flow rate volume lung function airflow sensor sound pressure microphone processing estimated
  151. 151. SpiroSmart availability cost portability more effective coaching interface integrated uploading
  152. 152. Using SpiroSmart
  153. 153. Using SpiroSmart
  154. 154. Using SpiroSmart ]
  155. 155. Using SpiroSmart ]
  156. 156. Using SpiroSmart ]
  157. 157. Using SpiroSmart
  158. 158. initial study design x 3 x 3
  159. 159. need for attachments
  160. 160. need for attachments
  161. 161. mouthpiece
  162. 162. sling
  163. 163. study design x 3 x 3
  164. 164. + study design x 3 x 3 x 3 + + x 3 x 3
  165. 165. + study design x 3 x 3 x 3 + + x 3 x 3
  166. 166. + study design x 3 x 3 x 3 + + x 3 x 3
  167. 167. + study design x 3 x 3 x 3 + + x 3 x 3
  168. 168. + study design x 3 x 3 x 3 + + x 3 x 3
  169. 169. dataset + + +
  170. 170. participants 52 duration 45 minutes first session 29 abnormals 12 revisits 10 dataset + + +
  171. 171. participants 52 duration 45 minutes first session 29 abnormals 12 revisits 10 dataset + + +
  172. 172. participants 52 duration 45 minutes first session 29 abnormals 12 revisits 10 dataset + + +
  173. 173. participants 52 duration 45 minutes first session 29 abnormals 12 revisits 10 dataset + + +
  174. 174. participants 52 duration 45 minutes first session 29 abnormals 12 revisits 10 dataset + + +
  175. 175. audio
  176. 176. audio flow features
  177. 177. audio flow features measures regression
  178. 178. audio flow features measures regression FEV1 FVC PEF
  179. 179. audio flow features measures regression curve regression FEV1 FVC PEF
  180. 180. 0 1 2 3 4 0 5 10 15 Flow(L/s) Volume(L) 1 2 3 4 Volume(L) 0 1 2 3 4 0 5 10 15 Flow(L/s) Volume(L) 0 2 4 6 8 10 0 1 2 3 4 time(s) Volume(L) audio flow features measures regression curve regression FEV1 FVC PEF
  181. 181. 0 1 2 3 4 0 5 10 15 Flow(L/s) Volume(L) 1 2 3 4 Volume(L) 0 1 2 3 4 0 5 10 15 Flow(L/s) Volume(L) 0 2 4 6 8 10 0 1 2 3 4 time(s) Volume(L) audio flow features measures regression curve regression lung function FEV1 FVC PEF
  182. 182. 0 1 2 3 4 5 6 7 −1 −0.5 0 0.5 1 time(s) amplitude flow features
  183. 183. 0 1 2 3 4 5 6 7 −1 −0.5 0 0.5 1 time(s) amplitude flow features 0 1 2 3 4 5 6 7 −1 −0.5 0 0.5 1 time(s) amplitude envelope detection
  184. 184. 0 1 2 3 4 5 6 7 −1 −0.5 0 0.5 1 time(s) amplitude flow features 0 1 2 3 4 5 6 7 −1 −0.5 0 0.5 1 time(s) amplitude envelope detection
  185. 185. 0 1 2 3 4 5 6 7 −1 −0.5 0 0.5 1 time(s) amplitude flow features 0 1 2 3 4 5 6 7 −1 −0.5 0 0.5 1 time(s) amplitude envelope detection
  186. 186. time(s) frequency(Hz) 1 2 3 4 5 6 0 500 1000 1500 2000 2500 0 1 2 3 4 5 6 7 −1 −0.5 0 0.5 1 time(s) amplitude flow features 0 1 2 3 4 5 6 7 −1 −0.5 0 0.5 1 time(s) amplitude envelope detection
  187. 187. time(s) frequency(Hz) 1 2 3 4 5 6 0 500 1000 1500 2000 2500 time(s) frequency(Hz) 1 2 3 4 5 6 0 500 1000 1500 2000 2500 0 1 2 3 4 5 6 7 −1 −0.5 0 0.5 1 time(s) amplitude flow features 0 1 2 3 4 5 6 7 −1 −0.5 0 0.5 1 time(s) amplitude envelope detection
  188. 188. time(s) frequency(Hz) 1 2 3 4 5 6 0 500 1000 1500 2000 2500 time(s) frequency(Hz) 1 2 3 4 5 6 0 500 1000 1500 2000 2500 0 1 2 3 4 5 6 7 −1 −0.5 0 0.5 1 time(s) amplitude flow features 0 1 2 3 4 5 6 7 −1 −0.5 0 0.5 1 time(s) amplitude envelope detection 0 1 2 3 4 5 6 7 −1 −0.5 0 0.5 1 time(s) amplitude resonance tracking
  189. 189. time(s) frequency(Hz) 1 2 3 4 5 6 0 500 1000 1500 2000 2500 time(s) frequency(Hz) 1 2 3 4 5 6 0 500 1000 1500 2000 2500 0 1 2 3 4 5 6 7 −1 −0.5 0 0.5 1 time(s) amplitude flow features 0 1 2 3 4 5 6 7 −1 −0.5 0 0.5 1 time(s) amplitude envelope detection 0 1 2 3 4 5 6 7 −1 −0.5 0 0.5 1 time(s) amplitude resonance tracking
  190. 190. 0 1 2 3 4 5 6 7 −1 −0.5 0 0.5 1 time(s) amplitude flow features 0 1 2 3 4 5 6 7 −1 −0.5 0 0.5 1 time(s) amplitude envelope detection 0 1 2 3 4 5 6 7 −1 −0.5 0 0.5 1 time(s) amplitude resonance tracking
  191. 191. 0 1 2 3 4 5 6 7 −1 −0.5 0 0.5 1 time(s) amplitude flow features 0 1 2 3 4 5 6 7 −1 −0.5 0 0.5 1 time(s) amplitude envelope detection 0 1 2 3 4 5 6 7 −1 −0.5 0 0.5 1 time(s) amplitude resonance tracking filtersource output
  192. 192. 0 1 2 3 4 5 6 7 −1 −0.5 0 0.5 1 time(s) amplitude flow features 0 1 2 3 4 5 6 7 −1 −0.5 0 0.5 1 time(s) amplitude envelope detection 0 1 2 3 4 5 6 7 −1 −0.5 0 0.5 1 time(s) amplitude resonance tracking filtersource output estimated
  193. 193. 0 1 2 3 4 5 6 7 −1 −0.5 0 0.5 1 time(s) amplitude flow features 0 1 2 3 4 5 6 7 −1 −0.5 0 0.5 1 time(s) amplitude envelope detection 0 1 2 3 4 5 6 7 −1 −0.5 0 0.5 1 time(s) amplitude resonance tracking 0 1 2 3 4 5 6 7 −1 −0.5 0 0.5 1 lpc8raw time(s) amplitude auto-regressive estimate filtersource output estimated
  194. 194. 0 1 2 3 4 5 6 7 −1 −0.5 0 0.5 1 time(s) amplitude flow features 0 1 2 3 4 5 6 7 −1 −0.5 0 0.5 1 time(s) amplitude envelope detection 0 1 2 3 4 5 6 7 −1 −0.5 0 0.5 1 time(s) amplitude resonance tracking 0 1 2 3 4 5 6 7 −1 −0.5 0 0.5 1 lpc8raw time(s) amplitude auto-regressive estimate filtersource output estimated
  195. 195. lung function audio flow features measures regression curve regression
  196. 196. −1 0 1 2 3 4 5 0 2 4 6 8 time(s) flow(L/s) −1 0 1 2 3 4 5 0 0.1 0.2 0.3 0.4 time(s) featurevalue 0 1 2 3 4 5 0 0.1 0.2 0.3 0.4 time(s) featurevalue measures regressionground truth feature 1 feature 2
  197. 197. −1 0 1 2 3 4 5 0 2 4 6 8 time(s) flow(L/s) measures regression −1 0 1 2 3 4 5 0 0.1 0.2 0.3 0.4 time(s) featurevalue 0 1 2 3 4 5 0 0.1 0.2 0.3 0.4 time(s) featurevalue ground truth feature 1 feature 2
  198. 198. −1 0 1 2 3 4 5 0 2 4 6 8 time(s) flow(L/s) measures regression −1 0 1 2 3 4 5 0 0.1 0.2 0.3 0.4 time(s) featurevalue 0 1 2 3 4 5 0 0.1 0.2 0.3 0.4 time(s) featurevalue 7.1 PEF featuresground truth feature 1 feature 2
  199. 199. −1 0 1 2 3 4 5 0 2 4 6 8 time(s) flow(L/s) measures regression −1 0 1 2 3 4 5 0 0.1 0.2 0.3 0.4 time(s) featurevalue 0 1 2 3 4 5 0 0.1 0.2 0.3 0.4 time(s) featurevalue 7.1 0.33 0.35 PEF featuresground truth feature 1 feature 2
  200. 200. −1 0 1 2 3 4 5 0 2 4 6 8 time(s) flow(L/s) measures regression −1 0 1 2 3 4 5 0 0.1 0.2 0.3 0.4 time(s) featurevalue 0 1 2 3 4 5 0 0.1 0.2 0.3 0.4 time(s) featurevalue 7.1 0.33 0.35 3.2 FEV1 features PEF featuresground truth feature 1 feature 2
  201. 201. −1 0 1 2 3 4 5 0 2 4 6 8 time(s) flow(L/s) measures regression −1 0 1 2 3 4 5 0 0.1 0.2 0.3 0.4 time(s) featurevalue 0 1 2 3 4 5 0 0.1 0.2 0.3 0.4 time(s) featurevalue 7.1 0.33 0.35 3.2 0.12 0.17 FEV1 features PEF featuresground truth feature 1 feature 2
  202. 202. measures regression FEV1 features PEF features
  203. 203. measures regression FEV1 features bagged decision tree PEF features bagged decision tree
  204. 204. measures regression FEV1 features bagged decision tree output PEF features bagged decision tree output
  205. 205. results measures regression Mean%Error Lower is better
  206. 206. results measures regression FEV1 FVC FEV1% PEF 0 2 4 6 8 10 Mean%Error Lower is better
  207. 207. results measures regression FEV1 FVC FEV1% PEF 0 2 4 6 8 10 Mean%Error Lower is better
  208. 208. results measures regression FEV1 FVC FEV1% PEF 0 2 4 6 8 10 No Personalization Personalization Mean%Error Lower is better
  209. 209. results measures regression general model = 8.8% error ATS criteria = 5-7% personal model = 5.1% error
  210. 210. results measures regression general model = 8.8% error ATS criteria = 5-7% normal attachments have no effect personal model = 5.1% error
  211. 211. lung function audio flow features measures regression curve regression
  212. 212. curve regression −1 0 1 2 3 4 5 0 0.1 0.2 0.3 0.4 time(s) featurevalue 0 1 2 3 4 5 0 0.1 0.2 0.3 0.4 time(s) featurevalue feature 1 feature 2
  213. 213. −1 0 1 2 3 4 5 0 2 4 6 8 time(s) flow(L/s) curve regression −1 0 1 2 3 4 5 0 0.1 0.2 0.3 0.4 time(s) featurevalue 0 1 2 3 4 5 0 0.1 0.2 0.3 0.4 time(s) featurevalue feature 1 feature 2 curve output
  214. 214. curve regression bagged decision tree −1 0 1 2 3 4 5 0 0.1 0.2 0.3 0.4 time(s) featurevalue 0 1 2 3 4 5 0 0.1 0.2 0.3 0.4 time(s) featurevalue
  215. 215. curve regression bagged decision tree −1 0 1 2 3 4 5 0 0.1 0.2 0.3 0.4 time(s) featurevalue 0 1 2 3 4 5 0 0.1 0.2 0.3 0.4 time(s) featurevalue CRF
  216. 216. curve regression bagged decision tree −1 0 1 2 3 4 5 0 0.1 0.2 0.3 0.4 time(s) featurevalue 0 1 2 3 4 5 0 0.1 0.2 0.3 0.4 time(s) featurevalue CRF −1 0 1 2 3 4 5 0 2 4 6 8 time(s) flow(L/s)
  217. 217. example curvescurve regression −2 0 2 4 6 0 5 10 15 volume(L) flow(L/s) −1 0 1 2 3 4 0 2 4 6 8 volume(L) flow(L/s)
  218. 218. example curvescurve regression −2 0 2 4 6 0 5 10 15 volume(L) flow(L/s) −2 0 2 4 6 0 5 10 15 volume(L) flow(L/s) −1 0 1 2 3 4 0 2 4 6 8 volume(L) flow(L/s) −1 0 1 2 3 4 0 2 4 6 8 volume(L) flow(L/s)
  219. 219. example curvescurve regression −2 0 2 4 6 0 5 10 15 volume(L) flow(L/s) −2 0 2 4 6 0 5 10 15 volume(L) flow(L/s) −2 0 2 4 6 0 5 10 15 volume(L) flow(L/s) −1 0 1 2 3 4 0 2 4 6 8 volume(L) flow(L/s) −1 0 1 2 3 4 0 2 4 6 8 volume(L) flow(L/s)
  220. 220. example curvescurve regression −2 0 2 4 6 0 5 10 15 volume(L) flow(L/s) −2 0 2 4 6 0 5 10 15 volume(L) flow(L/s) −2 0 2 4 6 0 5 10 15 volume(L) flow(L/s) −2 0 2 4 6 0 5 10 15 volume(L) flow(L/s) −1 0 1 2 3 4 0 2 4 6 8 volume(L) flow(L/s) −1 0 1 2 3 4 0 2 4 6 8 volume(L) flow(L/s)
  221. 221. can spirosmart curves be used for diagnosis?
  222. 222. survey
  223. 223. • 10 subjects curves survey
  224. 224. • 5 pulmonologists • 10 subjects curves survey
  225. 225. • 5 pulmonologists • 10 subjects curves • unaware if from SpiroSmart / spirometer survey
  226. 226. survey
  227. 227. survey
  228. 228. survey
  229. 229. survey
  230. 230. survey
  231. 231. results normal minimal obstructive mild obstructive moderate obstructive severe obstructive restrictive inadequate identical 32 / 50
  232. 232. results normal minimal obstructive mild obstructive moderate obstructive severe obstructive restrictive inadequate identical one off 32 / 50 5 / 18
  233. 233. results normal minimal obstructive mild obstructive moderate obstructive severe obstructive restrictive inadequate identical one off 32 / 50 5 / 18
  234. 234. results normal minimal obstructive mild obstructive moderate obstructive severe obstructive restrictive inadequate identical one off 32 / 50 5 / 18
  235. 235. results normal minimal obstructive mild obstructive moderate obstructive severe obstructive restrictive inadequate identical one off 32 / 50 5 / 18
  236. 236. FDA study underway
  237. 237. • part a: head to head clinical test FDA study underway
  238. 238. • part a: head to head clinical test • part b: home spirometry FDA study underway
  239. 239. future research SNUPI Static E-Field HydroSense water sensing lung function
  240. 240. future research SNUPI Static E-Field HydroSense water sensing lung function
  241. 241. thermal imaging facial analysis image fidelity power harvesting gas sensing cough sensing interaction& imageanalysis sustainabilityhealth water sensing lung function
  242. 242. thermal imaging facial analysis image fidelity power harvesting gas sensing cough sensing interaction& imageanalysis sustainabilityhealth water sensing lung function activity detection elder care daily activity logs congestive heart failure
  243. 243. thermal imaging facial analysis image fidelity power harvesting gas sensing cough sensing interaction& imageanalysis sustainabilityhealth water sensing lung function activity detection elder care daily activity logs congestive heart failure health markers via phone blood pressure pulse oximetry stress
  244. 244. thermal imaging facial analysis image fidelity power harvesting gas sensing cough sensing interaction& imageanalysis sustainabilityhealth water sensing lung function activity detection elder care daily activity logs congestive heart failure health markers via phone blood pressure pulse oximetry stress opportunistic sensing pain management detecting circulation
  245. 245. thermal imaging facial analysis image fidelity power harvesting gas sensing cough sensing interaction& imageanalysis sustainabilityhealth water sensing lung function activity detection elder care daily activity logs congestive heart failure health markers via phone blood pressure pulse oximetry stress opportunistic sensing pain management detecting circulation developing world
  246. 246. summary •sustainable water use, eco-feedback •lung function via mobile phone •future work in high impact areas
  247. 247. enabling eco-feedback and out-of-clinic health sensing indirect University of Washington Eric C. Larson UbiComp Lab electrical engineering computer science and engineering ubiquitous sensing eclarson.com eclarson@uw.edu @ec_larson Thank You! thermal imaging facial analysis image fidelity power harvesting gas sensing cough sensing interaction& imageanalysis sustainabilityhealth water sensing lung function
  248. 248. enabling eco-feedback and out-of-clinic health sensing indirect University of Washington Eric C. Larson PhD Candidate in School of Electrical and Computer Engineering UbiComp Lab electrical engineering computer science ubiquitous sensing eclarson.com eclarson@uw.edu @ec_larson acknowledgments: Jon Froehlich Leah Findlater Elliot Saba Eric Swanson Tim Campbell Gabe Cohn Mayank Goel TienJui Lee Sidhant Gupta Josh Peterson Conor Haggerty Jeff Beorse Shwetak Patel Les Atlas James Fogarty Jeff Bilmes Margaret Rosenfeld
  249. 249. water tower water tower thermal expansion tank hose spigot utility water meter pressure regulator laundry bathroom 1 hot water heater bathroom 2 dishwasher incoming cold water from supply line kitchen
  250. 250. water tower water tower thermal expansion tank hose spigot utility water meter pressure regulator laundry bathroom 1 hot water heater bathroom 2 dishwasher incoming cold water from supply line kitchen
  251. 251. bath% toilet& shower' kitchen(sink( features: single instance
  252. 252. bath% toilet& shower' kitchen(sink( features: single instance
  253. 253. bath% toilet& shower' kitchen(sink( features: single instance
  254. 254. bath% toilet& shower' kitchen(sink( features: single instance
  255. 255. stable pressure (psi) maxamplitude features: single instance
  256. 256. stable pressure (psi) maxamplitude Kitchen(Sink( Basement(Sink( Bathroom(Sink( Basement( Shower( ((((((((((((((((((((Upstairs(Shower( Upstairs(Bath( features: single instance
  257. 257. 42# 44# 46# 48# 50# 0# 5# 10# 15# 2 raw pressure (psi) 44" 46" 48" smoothed pressure (psi) bandpass derivative (psi/s) !4# 0# 4# detrended derivative bandpass derivative const-Q cepstrum signaltransforms features: single instance
  258. 258. frequency 42# 44# 46# 48# 50# 0# 5# 10# 15# 2 raw pressure (psi) *10# 10# 30# cepstral amplitude index detrended derivative bandpass derivative const-Q cepstrum signaltransforms features: single instance
  259. 259. features: single instance time frequency
  260. 260. features: single instance ] ] time frequency ]
  261. 261. features: single instance ] ] time frequency ]
  262. 262. 70 50 30 pressure(psi) 70 50 30 pressure(psi) feature extraction: sequence
  263. 263. 70 50 30 pressure(psi) 70 50 30 pressure(psi) feature extraction: sequence
  264. 264. 70 50 30 pressure(psi) 70 50 30 pressure(psi) 10 sec feature extraction: sequence
  265. 265. 70 50 30 pressure(psi) 70 50 30 pressure(psi) 10 sec compound feature extraction: sequence
  266. 266. 70 50 30 pressure(psi) 70 50 30 pressure(psi) 10 sec compound time of day feature extraction: sequence
  267. 267. accuracy
  268. 268. fixture level e.g., upstairs bathroom faucet accuracy
  269. 269. fixture level e.g., upstairs bathroom faucet accuracy fixture 50 60 70 80 90 100 92.2
  270. 270. fixture level e.g., upstairs bathroom faucet accuracy fixture 50 60 70 80 90 100 92.2 what does this tell us? 10 fold cross validation
  271. 271. cycles
  272. 272. cycles heated dry
  273. 273. cycles heated dry washing dishes
  274. 274. ~7:30AM
  275. 275. filling coffee pot coffee pot running ~7:30AM
  276. 276. bathroom usage
  277. 277. bathroom usage leaky flapper valve
  278. 278. flow features 0 1 2 3 4 −1 0 1 0 1 2 3 4 −0.5 0 0.5 Amplitude 0 1 2 3 4 −1 0 1 time(s)
  279. 279. flow features 0 1 2 3 4 −1 0 1 0 1 2 3 4 −0.5 0 0.5 Amplitude 0 1 2 3 4 −1 0 1 time(s) −1 0 1 2 3 4 5 0 2 4 6 8 time(s) flow(L/s)
  280. 280. flow features 0 1 2 3 4 −1 0 1 0 1 2 3 4 −0.5 0 0.5 Amplitude 0 1 2 3 4 −1 0 1 time(s) ~audio, pressure −1 0 1 2 3 4 5 0 2 4 6 8 time(s) flow(L/s)
  281. 281. flow features 0 1 2 3 4 −1 0 1 0 1 2 3 4 −0.5 0 0.5 Amplitude 0 1 2 3 4 −1 0 1 time(s) ~audio, pressure ~pressure at lips −1 0 1 2 3 4 5 0 2 4 6 8 time(s) flow(L/s)
  282. 282. flow features 0 1 2 3 4 −1 0 1 0 1 2 3 4 −0.5 0 0.5 Amplitude 0 1 2 3 4 −1 0 1 time(s) ~audio, pressure ~pressure at lips ~flow at lips −1 0 1 2 3 4 5 0 2 4 6 8 time(s) flow(L/s)
  283. 283. −1 0 1 2 3 4 5 0 0.1 0.2 0.3 0.4 time(s) featurevalue 0 1 2 3 4 5 0 0.1 0.2 0.3 0.4 time(s) featurevalue feature 1 feature 2 linear chain regression
  284. 284. −1 0 1 2 3 4 5 0 2 4 6 8 time(s) flow(L/s) −1 0 1 2 3 4 5 0 0.1 0.2 0.3 0.4 time(s) featurevalue 0 1 2 3 4 5 0 0.1 0.2 0.3 0.4 time(s) featurevalue feature 1 feature 2 curve output linear chain regression
  285. 285. CRF −1 0 1 2 3 4 5 0 0.1 0.2 0.3 0.4 time(s) featurevalue 0 1 2 3 4 5 0 0.1 0.2 0.3 0.4 time(s) featurevalue linear chain regression
  286. 286. CRF −1 0 1 2 3 4 5 0 0.1 0.2 0.3 0.4 time(s) featurevalue 0 1 2 3 4 5 0 0.1 0.2 0.3 0.4 time(s) featurevalue bagged decision tree linear chain regression
  287. 287. CRF −1 0 1 2 3 4 5 0 0.1 0.2 0.3 0.4 time(s) featurevalue 0 1 2 3 4 5 0 0.1 0.2 0.3 0.4 time(s) featurevalue bagged decision tree −1 0 1 2 3 4 5 0 2 4 6 8 time(s) flow(L/s) linear chain regression
  288. 288. example curvescurve regression 0 1 2 3 0 2 4 6 8 volume(L) flow(L/s) −1 0 1 2 3 4 0 2 4 6 8 10 volume(L) flow(L/s)
  289. 289. example curvescurve regression 0 1 2 3 0 2 4 6 8 volume(L) flow(L/s) 0 1 2 3 0 2 4 6 8 volume(L) flow(L/s) −1 0 1 2 3 4 0 2 4 6 8 10 volume(L) flow(L/s) −1 0 1 2 3 4 0 2 4 6 8 10 volume(L) flow(L/s)

×