Presentation by Hiro Takagi at HICSS50, 2017

1. Eldercare Research
Cognitive Computing for Aging Society
Hiro Takagi
STSM, IBM Research - Tokyo
Aging Strategic Initiative lead, IBM Research
Human Computer Interaction area lead, IBM Research
Academy of Technology member

2. 62 87
28%
27%
Dementia 15% + MCI 13%
estimation by Japanese Government, 2012
of people over the age of 65 are
at risk of dementia.
People over the age of 65, 2017.
70%of people want to work over the
age of 70 (and forever).
Life expectancy of women in
1950 and 2015.
* numbers in Japan

3. 2017

4. We need a new model for
active life-long society.

5. © 2016 IBM Corporation
2003
“Elderly Vision
Simulator”
2007
Eclipse.org
Open source
2011
Official tool for
JIS (Japanese Industrial Standard)
2011
Japanese Government
Science Grant
“IT for Active Aging Society”
2009
Open Collaborative Research
University of Tokyo
“Elderly Accessibility”
2013
Japanese Government
Pilot Grant
“Remote community
collaboration”
2015
Sekisui House
JDA
2017
Japan Post Official Service
“Conversation as Sensors”
GTO 2016 Demographic Shift
2015
Japan Post
Announcement
2015 – 2016
Japan Post Pilot
“Voice QA”
2005 2010 2015 2016 2017
Tokyo Research Aging Research History
Research
Aging Strategic Initiative2014
KDDI
“Senior Smartphone”
Government and University projects Customer projects Internal activities

6. Power of IT – Five Long-term Focus Areas
Social
Intelligent assist
Sensing
Accessibility
Brain

7. Social
Intelligent assist
Sensing
Accessibility
Brain

8. Use case barrier
Psychological barrier
User interface
barrier
Price Barrier
Small fonts, small buttons
Don’t know how to operate
Fear, uncertainty
Cannot imagine use cases
Barriers for Senior Citizens to Start Using
Information Technologies

9. Target Actual
(average)
Fingertip angle + Parallax
Landed on
Took off
(px)
(px)
Touch duration + Tremor
Unintentional Motion
Accessibility: Touching a target location

10. Gesture Analysis for Novice Elderly Assist (2014)
Press release & images (Japanese only)
IBM Japan Press Release: http://bit.ly/1gavLuN
KDDI R&D Labs Press Release: http://bit.ly/1hK1B2K
KDDI Corporation Press Release: http://bit.ly/1f5zbTx
Provided appropriate guidance to
users based on user's level of
smartphone operation
proficiency (skill and operational
misstep) by analyzing
smartphone operation logs with
mathematical analysis
technology

11. Improved
confidence
and
subjective
operation
speed
33 0 3 -- 7 0 -6 067 ,3
-- 7
6 067 ,3
2 7 - 2 1
172 17
You entered
“one-hundred
thousand” yen
Accessibility: Voice-over Effectiveness

12. Social
Intelligent assist
Sensing
Accessibility
Brain

13. Elderly Skill Matching System – J-scouter
Senior profile
datasets
(Free-form text)
Job / Activity
Descriptions
(Free-form text)
Find best
persons
Find best
jobs/activities
A Prime minister initiative
“Work-style transformation.”
“Senior workforce” is one of
focuses.
A senior worker with rich
experience in legal compliance is
working as an advisor in a start up
company
Supported by
Japan Science and Technology Agency
Senior HR company
10,000 seniors registered
from TV BS11
By Kyodo News

14. “Reimagining
Life”
“Target 5 million users.”
http://www-03.ibm.com/press/us/en/pressrelease/46740.wss
2015

15. Japan Post Project
• Pilot with 1000 elderlies completed (Oct. 2015 – Oct. 2016)
• Medicine reminders, video phones, shopping assist, photo sharing, etc.
Video

16. Global Apple+IBM
Apps for Elder
Services
Support
Worker
Elder
at
Home
Family /
Loved
Ones

17. Social
Intelligent assist
Sensing
Accessibility
Brain

18. )
( )
(
-
- (-
(
( )
)
-
( -
-
-
(
Understand
ambiguous
questions
Pick up
appropriate
answers
Learn
user
feedback
- -

19. Video

20. Social
Intelligent assist
Sensing
Accessibility
Brain

21. IBM Accessibility Research
IoT, Health and Aging
Ecosystem Partners /
Enriched Data
Watson IoT & Health Platform
Sensors &
Devices
Client’s
Data
Personality Insights &
Social Interaction Data
Other Data Sources
Applications
Data &
Platforms
01
0110
0010
001001
Cognitive Services for Elders
Empowered Living
Empowered Social
Empowered Care
Knowledge
Reactor
“Infrastructure
Providers”

23. • Voice-base natural user interface for daily watchover
• Analyze life patterns, feelings, interests and issues from daily conversation (e.g. cognitive decline detection)
• Share information with families and care givers
• Provide support that can keep the elderly self-sufficient
Conversation as Sensors

24. © 2014 IBM Corporation
Video

25. Intent of Each Questionnaire
Questionnaire Intent
“What did you have last night?” Dementia (Neuro Cognitive Disease) Risk
Assessment
“Did you go out today?” Activity checking
“You mentioned about gardening in diary. Are you
interested in community gardening?”
Recommendation of activities
”Which city do you want to travel?” Marketing
“The shop, you mentioned last week, is now on sale!” Promotion
“We delivered a package, but you were not there.
Please contact post office"
Notification (business related)
25

26. Emotion and Physical Condition Recognition
Voice data Features
988 acoustic features
for emotion recognition.
Intensity
Loudness
12 MFCC
Pitch (F0)
Probability of voicing, F0
envelope,
8 LSF,
Zero-Crossing Rate
Labeled
voice
dataset
ML-based
Regression
Estimation
Feeling good
or bad
40
0
100
Physical
condition
Ex: Good night
*Each circle represents one greeting.
*One participant data
Emotion
NegativePositive
Confusion FrustrationDelight Flow
Physical Condition
Emotion
Yorktown – Speech team
Tokyo Research
Happy
Sad

27. Dashboard for Family and Care Givers
27
Condition
Emotion (voice sound analysis)
Personality analysis (Watson
personality insights)
Daily activities
Eating habits
Dementia questionnaire scores

28. Dashboard Visualization Examples
28
“what did you do today?” = Variety of daily activities.
practice, songs, poems, musical instruments,
sessions, friends, music hall, …. House (largest), cleaning, washing, room,
study…..
Smaller variety and words are not so active.Wider variety, and words are active.
“what did you have today?” = Variety of nutrition
Wider variety, and healthy meals
Vegetables, salad, radish, juice, natto, miso, eggs,
….
Rice (largest), bento, sandwich, ….
Smaller variety and ready-made meals

29. Feature extraction of gait
• Gait speed & its variability
• Step frequency
• Stride time variability
• Step-length & its variability
• Foot swing velocity
• Stance and stride time
Detection
• Motor function
• Fall risks
• Cognitive decline
• Episodic memory
• Executive function
• Diseases
• Parkinson disease
• MCI
• Alzheimer’s disease
Gait and Cognitive Decline

30. Video

31. Social
Intelligent assist
Sensing
Accessibility
Brain

32. Brain Simulation: Understanding Cognitive Deficit
(2016)
Video

33. Power of IT – Five Long-term Focus Areas
Social
Intelligent assist
Sensing
Accessibility
Brain