Kognit – Cognitive Assistants for Dementia Patients


Published on

Daniel Sonntag from the German Research Center for Artificial Intelligence is making this presentation at the Cognitive Systems Institute Group Speaker Series on October 29, 2015.

Published in: Technology
1 Like
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Kognit – Cognitive Assistants for Dementia Patients

  1. 1. kognit.dfki.de Kognit – Cognitive Assistants for Dementia Patients Daniel Sonntag German Research Center for Artificial Intelligence
  2. 2. AI IUI HCI Interdisciplinary Field and Transcommunity https://dl.dropboxusercontent.com/u/48051165/ISMAR-2015-IUI-TUTORIAL.pdf
  3. 3. AI as the Basis for Multimodal Interaction in IUIs Multimodal Multisensor Interfaces
  4. 4. Kognit Theory Design Aspects
  5. 5. Topic •The use of AI to elders with dementia •Intelligent assistive technology •Intelligent cognitive assistance technology •Design of advanced assistive technology
  6. 6. • While many older adults will remain healthy and productive, overall this segment of the population is subject to physical and cognitive impairment at higher rates than younger people. • Because of the demographic chance, here will be fewer young people to help older adults cope with the challenges of aging. • Intelligent cognitive assistance technology may enable older adults to “age in place,” that is, remain living in their homes and independently for longer periods of time. Motivation
  7. 7. Kognit’s win-win effect Improve quality of life Living a self- determined independent life. Save enormous amounts of money Provide relief and more time for caregivers Reduce healthcare system costs Institutionalisation has an enormous financial cost
  8. 8. The change in demographics is immediately clear: older adults make up an increasingly greater proportion of the population. The most rapid growth will occur within a subgroup of this cohort— the so-called “oldest old,” or people over the age of 80.
  9. 9. Compensate for the physical and sensory deficits that may accompany aging no computer technology - lift chair, wheel chair - ergonomic handles - hearing aid device - cardiac pacemaker Advanced computer-based technologies for AAL (ambient assisted living) - SSPI - exoskeleton - control household appliances (using, e.g., head gestures) Towards cognitive enhancement computer technology AI technology - SSPI (Speech) - AI companion Assurance of, compensation for, assessment of cognitive deficits CIND / Dementia sensor-motor and psychosocial issues cognitive decline
  10. 10. Goals for Kognit Assurance and Monitoring: ensuring safety and well-being and reducing caregiver burden, by tracking an elder’s behaviour, assessment of regularities and providing up-to-date status reports to a caregiver. Compensation: provide guidance to people as they carry out their daily activities, reminding them of what they need to do, how to do it and related this to active memory training (in AR, MR, VR and serious games) and proactive multimodal help (in the field of view). Assessment: attempt to infer how well a person is doing—what his or her current cognitive level of functioning is—based on continual multimodal observation of performance of routine activities (in MR, VR and speech-based serious games)
  11. 11. Compensation Paradox compensation user must be made aware of planned task/activity and must be guided user and caregiver satisfaction - usability / utility avoid introducing inefficiency into user activities - usability / utility avoid making the user overly reliant on the compensation system request confirmation about whether an activity has been completed successfully
  12. 12. Sensors for Activity Monitoring Video Cameras GPS Bluetooth Beacons Eye Tracker Speech Input Bio- Sensors Domain and location model Interaction with smart objects Activity recognition Activity performance Serious Game Cognitive Status Task and user model Context Models
  13. 13. AI Technology • plan generation and execution monitoring • reasoning under uncertainty • machine learning • natural language processing • intelligent user interfaces • robotics and machine vision • collaboration with colleagues having expertise in • sensor-network architectures • privacy and security, and • human-machine interaction
  14. 14. • failure to eat or drink regularly, pill taking • wandering around • ATM: don’t give your money to strangers • avoid stress situations or recover from them • household chores and many more … • https:// www.linkedin.com/ topic/group/cognitive- systems-institute? gid=6729452 Scenario demands
  15. 15. Kognit Storyboard and implementation
  16. 16. • Memory disorder result in loss of episodic memory in particular, which accounts for our memory of specific events and experiences that can be associated with contextual information. Towards compensating such mental disorder, our goal is to provide the user with episodic memory augmentations by using AI technologies. • Autobiographical events (times, places, associated emotions, and other con- textual who, what, when, where, why knowledge) that can be explicitly stated constitute information fragments for which a prosthetic memory organisation would be needed. • A major question concerns the recall of only useful information along the thought process of the individual (and not to slow it down). • For everyday memory support, we aim to develop a system that can recognise everyday visual content that the user gazes at and construct an episodic memory database entry of the event. The episodic memory database is used to save and retrieve the user’s personal episodic memory events.
  17. 17. kognit.dfki.de/media
  18. 18. Text Recognizer “aspirin” Databases and recognition modules Object DB Activity DB Episodic Memory DB { id: “bread”, type: “object”,
   image: {“sample1.png”, ...},
   features: {“feature1.txt”, ...}, description: “bread is a food” } { id: “Takumi”, type: “person”, image: {“face1.png”,…}, features: {“feature1.txt”,…} description: “Mitarbeiter” } … { id: [UNIQUE ID], start: "2014/10/30/20:10:14", end: "2014/10/30/20:10:16", activity: "eat", object: "bread“ } { id: [UNIQUE ID], start: "2014/10/30/10:05:54", end: "2014/10/30/10:20:12", activity: “discuss", object: “Takumi“ } … { id: “eat”, level: 2,
   derived_from: {“bite”,”chew”,…}
   form: “have_a_meal” } { id: “have_a_meal”, level: 3, derived_from: {“eat”,”drink”,…}, form: “” } { id: “discuss”, level: 2, derived_from: {“look_at_face”,”speak”,…}, form: “meeting” } … Kognit Cloudant Database: https://kognit- tt.cloudant.com/ Face Recognizer Person A Bread Object Recognizer
  19. 19. Object database (including faces) Sensor Data Attention to … Gaze Gaze Face Object Text Gaze Episodic memory event encoding model (Breakfast Scenario) Encoded Event Person A Person A Cheese Spoon Bread “ingredients:…” “take 1 pill in the morning…” Eye tracker (with scene camera) GPS, or other sensors Location Living room Interpretation of raw sensor data: e.g., object recognition, location estimation,… Encode the observations into an episodic event: [Activity] -> [Object] Activity database (created by crowdsourcing platform: LabelMovie) Make a sandwich Speak with Person A Read Medication Instruction
  20. 20. Kognit Hardware Overview Narrative Clip Pupil Labs Eye-tracker Oculus DK 2 Anoto SMI Eye-tracker Space Glasses Meta Pro (3D cam) NAO Humanoid Epson Moverio BT-200 Structure Sensor Brother Airscouter Tobii EyeX WheelPhone Low range 3D cam Cybershot scene cam Leap Motion Accu LED projector
  21. 21. Serious Games in VR
  22. 22. Conclusion • Towards constructing episodic memory event database of the user (as the basis for compensation), we developed a method for recognition of the visual content that the user gazes at in an everyday scenario. • Though the face recognition showed robustness, we still have to improve object recognition in natural environments. • In future work, we will use an HMD to present the information of previous events or recognised objects to the user to further evaluate the presented technical implementation of episodic memory along the thought-process of the user. • 2D —> 3D, deep learning, GPU