Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Sensor Based Ambient Assisted Living

135 views

Published on

This is a presentation on Sensor Based Ambient Assisted Living architecture and approaches developed by the Multimedia Knowledge and Social Media Analytics Lab of CERTH-ITI. It includes sensors used for monitoring Activities of Daily Living of elders and persons with mild Dementia at home. Visual and sensor data analytics are combined with formal representations (ontology), fusion, reasoning techniques and visualizations in order to provide an objective view of everyday activities. Example projects and pilots are included. Clinical assessment show improvement in cognitive abilities of participants.

Published in: Technology
  • Be the first to comment

Sensor Based Ambient Assisted Living

  1. 1. Sensor-based AAL ΕΛΕΒΗΤ 2019 CERTH-ITI MKLab Group Dr Ioannis Kompatsiaris* Researcher Grade A, MKLab Director Dr Thanos Stavropoulos Postdoc Associate Dr Spiros Nikolopoulos Senior Researcher Mrs Ioulietta Lazarou Clinical Researcher
  2. 2. The problem: Dementia Care • Inaccessible – Unaffordable - Inefficient Care of Dementia • No specific treatment – lifestyle/non-pharmaceutical • High 1 Nurse per 1 Patient Effort • High cost • Lack of objective information Yearly cost of dementia care in 2030 People living with dementia in 2050 People living with dementia now 46M 135.5M 2T $ 1 in 2 over 80 (US) Growing Numbers for Dementia
  3. 3. Overall approach Enhance current clinical workflow:  Continuous, comprehensive monitoring of PwD, condition and progression  Objective multi-sensor measurements (limiting interpretation subjectivity) Connect monitoring results with dementia staging and assist diagnosis Provide PwD with regular personalized feedback, updates and interventions  Improving condition  Enhancing a sense of safety and increased independence Relieve informal carers
  4. 4. The Solution • Existing IoT and wearable sensors provide diverse measurements (steps, HR, presence, object usage) • Intelligent analysis turns them into meaningful and useful behaviors and symptoms (cooking, chores, TV, sleep, stress) … in order for clinicians to provide care more effectively and efficiently
  5. 5. Technological Approach Sensors, Analysis & Integration
  6. 6. • 2013 - Use of prototypes • Philips • 2015 - Emergence in the market • FitBit • Jawbone • MS Band • Withings • 2019 - Growing capabilities • Empatica Embrace • Epilepsy FDA-app. • Omron HeartGuide • 24/7 BPM • Withings Move / Move ECG • 1 year battery life + ECG Evolution of IoT Devices
  7. 7. Device/Hardware Layer • A moving target; Need for future-proof modular support • Service-oriented architecture • Components need to support data retrieval • Streaming Real-time transfer e.g. over Smartphone • Use of Manufacturer SDKs to build smartphone apps for storage or upload • MS Band, Empatica • Data-logging storage in the device or on 3rd party cloud • Use of 3rd party (provider) cloud API for retrieval • Fitbit, Withings, Jawbone Our App via SDK Our Cloud Device Device Cloud Device App Device Our Cloud
  8. 8. Device variety and modalities in the platform Sleep Sensor IR Presence Object Movement Door Sensor Wearables • Beddit • Withings Aura • Philips DTI2 • Jawbone UP24, UP3 • Fitbit Zip, Charge HR • Empatica E4 • MS Band • Wireless Sensor Tags Appliance Usage • Plugwise
  9. 9. Wearable Cameras • Go Pro Hero For more sophisticated functions such as: • Activity Recognition • Object Recognition • Room Recognition
  10. 10. Depth Cameras • Microsoft Kinect • ASUS Depth Camera • Activity Recognition • Indoor localization
  11. 11. Architecture for Integration • Integration of • Device Layer to retrieve data • Sensor & Image processing layer to analyze them • Store unanimously in a Knowledge Base using semantic web technologies • Ontology with context & clinical information • Semantic Interpretation • Provides activities, behavior using ontology + reasoning • Provides symptoms and problems using rules • User interfaces to present Stavropoulos, T. G., Meditskos, G., & Kompatsiaris, I. (2017). DemaWare2: Integrating sensors, multimedia and semantic analysis for the ambient care of dementia. Pervasive and Mobile Computing, 34, 126-145.
  12. 12. Visual Analytics for Activity Recognition 12 • Detailed activity and location detection and recognition at home and lab environments. Crispim-Junior, C. F., Buso, V., Avgerinakis, K., Meditskos, G., Briassouli, A., Benois-Pineau, J., ... & Bremond, F. (2016). Semantic event fusion of different visual modality concepts for activity recognition. IEEE transactions on pattern analysis and machine intelligence (IEEE TPAMI), 38(8), 1598- 1611.
  13. 13. Signal Analysis Raw skin conductance signal Baseline histogram for a week Filtering and segmentation in 5 Stress levels Stress level signal Developed with Philips NL, evaluated in Lulea Technical University Kikhia, B., Stavropoulos, T., Andreadis, S., Karvonen, N., Kompatsiaris, I., Sävenstedt, S., ... & Melander, C. (2016). Utilizing a wristband sensor to measure the stress level for people with dementia. Sensors, 16(12), 1989.
  14. 14. Semantic Knowledge Structures (OWL 2) • Formal vocabularies for capturing context in different levels of granularity • Low-level observations (e.g. objects, locations) • Complex activity models (e.g. tea preparation) • Clinical knowledge (e.g. problems, monitoring parameters) 14
  15. 15. Context-based Multi-sensor Fusion and Analysis 15 • Objective: Fusion of information coming from heterogeneous sources in order to derive high-level interpretations of the behaviour of the person • Our approach: Knowledge-driven semantic segmentation and classification of context • Combination of SPARQL and OWL 2 meta-modelling ADL REC PRE Prepare Drug Box 92.00% 88.46% Make Phone Call 89.29% 96.15% Watch TV 84.00% 95.45% Water the plant 80.00% 95.24% Read Article 95.83% 85.19% Meditskos, G., Dasiopoulou, S., & Kompatsiaris, I. (2016). MetaQ: A knowledge-driven framework for context-aware activity recognition combining SPARQL and OWL 2 activity patterns. Pervasive and Mobile Computing, 25, 104-124.
  16. 16. End-result: Signals + Activities
  17. 17. Applications & Pilots Use Cases, Clinical Approaches, Interventions
  18. 18. EU FP7 ICT Project • Dementia Ambient Care 2011 – 2015 • Multimodal sensing • 3 use cases • 300+ participants
  19. 19. @Lab • Controlling equipment • Dementia Assessment in a controlled environment
  20. 20. @Lab Results • 290 participants Nice & Thessaloniki • 80% accuracy Healthy - MCI - AD
  21. 21. @Home • System-supported interventions • 4 in Dublin, 6 in Thessaloniki for 4 – 12 months • Improvement or non-deterioration in cognitive state • Compared to non-system-supported interventions or regular care
  22. 22. Apps for self-management • Confidence, inclusion
  23. 23. Clinical Observations - Correlation between metrics Increased Physical Activity – Decreased Sleep Latency Increased Physical Activity – Increased Sleep Duration
  24. 24. Clinical-Neuropsychological Assessment • 6 Users • 4 with Mild Cognitive Impairment (MCI) • 2 with Alzheimer’s Disease (AD) • Neuropsychological and Clinical Evaluation • Baseline (1st half) and Follow-up Assessment (2nd half) • Standardized and Validated Cognitive Test • Tailored non-pharmaceutical interventions (4-12 months) i.e., Cognitive Behavioral psychotherapy, reminiscence, memory exercises etc Tests Assessment Time Users 1-6 M SD MMSE 1st half 26.50 3.56 2nd half 28.33 1.86 NPI 1st half 1.67 0.52 2nd half 0.83 1.33 FRSSD 1st half 3.83 3.13 2nd half 2.67 2.66 FUCAS 1st half 48.67 8.48 2nd half 44.00 2.45 RAVLT - copy spelling 1st half 4.83 1.72 2nd half 5.50 2.43 RAVLT - learning 1st half 7.67 3.01 2nd half 9.00 4.05 RAVLT - recall 1st half -3.50 1.05 2nd half -0.50 5.17 RAVLT total 1st half 38.67 13.53 2nd half 45.83 15.94
  25. 25. Clinical Observations User 1 User 2 User 3 User 4 User 5 User 6 Total Time Asleep (hours) 1st half 7.24 6.4 8.09 6.39 5.98 6.24 2nd half 8.14 7.36 8.62 6.47 7.1 7.58 P 0.001 0.0001 0.17 0.09 0.02 0.003 Number of Interruptions 1st half 2.32 5.52 5.8 3.74 3.72 3.67 2nd half 2.13 5.85 3.8 2.25 2.4 4.75 P 0.44 0.57 0.001 0.02 0.0001 0.001 Shallow Sleep (hours) 1st half 4.36 2.31 3.78 3.88 3.00 2.67 2nd half 3.88 3.04 3.67 4.37 3.35 3.1 P 0.01 0.02 0.64 0.36 0.0001 0.02 Sleep Latency (min) 1st half 6.83 0.47 8.8 10.11 5.81 8.8 2nd half 3.33 0.46 8.4 5.1 5.5 8.4 P 0.009 0.95 0.88 0.03 0.59 0.02 Deep Sleep (hours) 1st half 1.61 0.9 1.83 1.83 1.35 1.13 2nd half 1.94 1.22 2.13 2.13 1.42 1.54 P 0.02 0.0001 0.04 0.13 0.54 0.0001 Physical Activity (min) 1st half 56.07 43.31 68.62 33.91 13.1 109.7 2nd half 57.16 43.17 74.25 35.13 43.26 112.94 P 0.22 0.91 0.0001 0.29 0.04 0.72 • Improvement in Sleep Parameters and Physical Activity in the 2nd half of observational period
  26. 26. EU H2020 IoT LSP Project • EU IoT Large Scale Pilots 2017 – 2020 • 9 Deployment Sites All Partners 25.772.829 € MEDTRONIC (coord) & 40 partners • Integrate major open IoT platforms • Active and Healthy Ageing (AHA) applications • 9 Use Cases • Mobility & Transport, emergency, home activity & behavioral monitoring etc. • A Marketplace to discover and install apps
  27. 27. Activage IoT Ecosystem Suite - AIOTES • Not only dementia but any scenario of Active and Healthy Ageing • Allows you to build and monetize eHealth apps over 9 open European platforms • 2nd open call to be launched soon
  28. 28. National Project • EU-funded Erevno-Kainotomo-Dimiourgo 2018 – 2020 • 75 Home Participants • Wearable & Apps to support Alzheimer • Minimum equipment to support easy deployment and maintenance • Promoted to national Telecom providers All Partners 587.450 € Role CERTH Thessaloniki IoT Integration, AI Analysis, RnD ARX .NET Thessaloniki Mobile Apps, Business Dev Frontida Zois Patras Clinical Pilots & Evaluation
  29. 29. IMI2 RADAR AD • EU and EFPIA (Pharma) co-funded and co-developed 2019 – 2021 • Platform for Alzheimer’s “digital biomarkers” brought closer to medical practice • Building over RADAR-CNS platform and trials for multiple diseases • 3-Tiers of Clinical Trials • 1) Small-scale Homes 2) Large-scale Homes 3) CERTH Smart Home - multiple equipment and incubator environment
  30. 30. CERTH-ITI Smart Home • Rapid-prototyping & demonstration for actual living scenarios • 1st near Zero Energy Building in Greece • 2-story, 4-bedroom, 3-bathrooms, living room, kitchen • one of the most important pillars of the Digitise European Industry effort • Equipment • Smart Home • Energy – Solar Panel • Open to any choice of RADAR AD Tier 3 ROBOTICS & AI HEALTH ENERGY BIG DATA
  31. 31. User-acceptance and Perspective • A Patient-Advisory Board (PAB) to select devices • Game with cards to extract preferences (1st meeting in Luxembourg) • Anonymously select device representatives and aspects • Top voted aspects are • Appearance and Style, Weight, Water-proof, Emergency button feature and Battery life
  32. 32. Other Projects - Wearable Cameras for the Blind • eVision • National Pilot Project • Wearable Cameras for the blind • SUITCEYES • EU Pilot Project • Suit & Wearable Cameras for the deafblind All Partners 660.075 € CERTH Thessaloniki TETRAGON LTD Thessaloniki PRISMA Alexandroupoli MASOUTIS Thessaloniki THESSALONIKI MUNICIPALITY Thessaloniki All Partners 2.359.963 € CERTH Thessaloniki UNI BORAS Sweden UNI OFFENBURG Germany UNI LEEDS UK UNI EIDHOVEN NL LES DOIGTS France HARPO SP Poland
  33. 33. Conclusions • IoT Constantly evolving – especially wearables • New modalities, reliability, acceptance and user-related parameters • Platform flexibility, also to pilot – deployment scale and complexity • Sensors, analytics, visualization, interpretation can assist staging and interventions • Difficult to differentiate between actual contribution and enhanced social activity • Large-scale pilots and security aspects • Big Data – Machine Learning approaches • Emphasis on physical and cognitive interventions • EEG mobile devices • New applications • Outdoor environment, smart cities • Working environments, e.g. Mental Health of Employers
  34. 34. Thank You Email : ikom@iti.gr Links Projects: demcare.eu activageproject.eu radar-ad.org Lab & all other scientific activity mklab.iti.gr Videos Lab Trials - https://www.youtube.com/watch?v=AEuX58HLIDo Home Monitoring - https://www.youtube.com/watch?v=0JNlaM6BpMA CERTH-ITI Smart Home Video - https://www.youtube.com/watch?v=8pcw1Xhk240

×