This document discusses discovering human activities from sensor data. It describes deploying over 1000 sensors in skilled nursing and independent living facilities to capture data on activities of daily living (ADLs) like bathing, dressing, eating. Unsupervised machine learning techniques like LDA topic modeling are used to learn relationships between sensors and ADLs in order to summarize what activities occupants are performing based on sensor readings. Tuning of the models, handling of edge cases, and transferring learning across different environments are identified as areas for further work.

1. © Copyright IBM Corporation 2016
IBM Accessibility Research
1
Scott Gerard (sgerard@us.ibm.com)
Mar 10, 2018
Discovering Human Activities from Sensors

2. © Copyright IBM Corporation 2016
IBM Accessibility Research
Impact & business opportunity of a global demographic shift
• US – Estimated assets for this
demographic $8.4 to $11.6 Trillion
• China – Estimated “silver hair” market
to rise to $17 Trillion by 2050,
amounting to a third of the Chinese
economy.
• Japan – Estimated 65+ financial
assets $9.1 trillion
• Rising Eldercare costs will disrupt
economies 6% of US GDP and 4 to
8% of EU GDP will account for social
service costs for the Elder. PercentageofPopulation65yearsandolder
Japan
Italy
Germany
Ireland
China
Australia
Brazil
US
India
Egypt
2017
•http://www.icis.com/blogs/chemicals-and-the-economy/2015/03/worlds-demographic-dividend-turns-deficit-populations-age/
•https://www.metlife.com/assets/cao/mmi/publications/studies/2010/mmi-inheritance-wealth-transfer-baby-boomers.pdf
•http://blogs.ft.com/ftdata/2014/02/13/guest-post-adapting-to-the-aging-baby-boomers/
•http://www.marketsandmarkets.com/Market-Reports/healthcare-data-analytics-market-905.html
•http://www.bloomberg.com/bw/articles/2014-09-25/chinas-rapidly-aging-population-drives-652-billion-silver-hair-market
•Asian Journal of Gerontology & Geriatrics for Centenarians: According to the National Institute of Population and Social Security Research, Japan had 67,000 centenarians in 2014, but that number is forecast to reach 110,000 in 2020, 253,000 in 2030 and peak at 703,000 in the year 2051.

3. © Copyright IBM Corporation 2016
IBM Accessibility Research
Maintaining highest
possible level of
contribution
Living in Retirement
Maintaining
independenc
e & security
Mature Adult Pre-Retirement Retirement In-Home Care Assisted Living 24hr Care
Workforce
Assisted Living Providers
P&C Insurance
Financial
Services
Governments
Retail, Consumer Electronics
&Start-Tech for non-digital natives
Healthcar
e
1x cost 4x cost 8x costFixed budget
Augment
Cognitive
Capabilit
y
Biz
Cognitive Life Advisors
E
S
E
L
Empowered Living
Empowered Social
Empowered CareE
C

4. © Copyright IBM Corporation 2016
IBM Accessibility Research
ADLs (Activities of Daily Living)
• Activities we normally do. Determines level of care needed.
• Bathing and showering
• Personal hygiene and grooming (including brushing/combing/styling
hair)
• Dressing
• Toileting (getting to the toilet, cleaning oneself, and getting back up)
• Eating (self-feeding not including cooking or chewing and swallowing)
• Functional mobility, often referred to as "transferring", as measured by
the ability to walk, get in and out of bed, and get into and out of a chair;
the broader definition (moving from one place to another while
performing activities) is useful for people with different physical abilities
who are still able to get around independently.
• We expect to see additional ADLs in our data
• Sleeping, Watching TV, …
4
https://en.wikipedia.org/wiki/Activities_of_daily_living

5. © Copyright IBM Corporation 2016
IBM Accessibility Research
Core Technology: The Knowledge Reactor
5
We have developed a contextual data fusion
engine, the Knowledge Reactor (KR), that
centralizes IoT and System of Record/Engagement
data fusion to create a reactive knowledge graph
that can integrate and drive various cognitive
applications and services.
The KR is designed to scale-up and scale–down
as requirements dictate, exploiting container-based,
horizontally scalable pub-sub (Kafka) and graph
database (Tinkerpop) technologies that sit logically
atop the Watson IoT Platform.
While initially developed for the cognitive eldercare
solution, the KR is a designed to be a general-purpose
reactive data fusion platform for Cognitive IoT.
To apply to a new problem domain, only the new data
sources must be ingested and modeled in the knowledge
graph and the application-specific services added.
Existing approaches to IoT data fusion are either
ad hoc or highly application specific and not
reusable across cognitive applications, resulting in
expensive duplicate efforts in data curation,
integration and knowledge modeling for each
cognitive service or application.

6. © Copyright IBM Corporation 2016
IBM Accessibility Research
Knowledge Reactor Environment
6
OLTP
OLAP
Agent
WIoT
• rule-based • ML-based

7. © Copyright IBM Corporation 2016
IBM Accessibility Research
Avamere – High Density Sensor Deployment
Instrumenting 20 Patient rooms in Skilled Nursing Facility
& 5 Independent Living Apartment
Over 1000 sensors deployed

8. © Copyright IBM Corporation 2016
IBM Accessibility Research
Why Context is Crucial
Elder is reclining, watching TV, but
what is all that other activity?
No pets allowed in this facility…
but…
So what are the ADLs of an old dog?
Does it matter?

9. © Copyright IBM Corporation 2016
IBM Accessibility Research
The Moving Pieces
9
Master
Worker
Worker
Jupyter Notebook
runs in browser
BluemixWatson Data Platform
IBM Cloud

10. © Copyright IBM Corporation 2016
IBM Accessibility Research
ADL Topic Modeling
LDA (Latent Dirichlet Allocation)
10
docs topics
bag of
words
• birdRelated
• catRelated
• dogRelated
• document
• paper
• article
• 1 min window
• 2 min window
ADLs
• cooking/eating
• toileting
• bathing
• dressing
• sleeping
• transfer/mobility
• beak, fly, tweet
• paw, meow, milk
• paw, bark, bone
Bag of Words:
dog bites man ==
man bites dog
Sensors
• -fysmclent
• -fysmclsnk
• ab-nxbed--
• ab-smclbed
• ab-smcl000
• ab-smcl100
• ab-sptv--
• ab-smar3md
• ...
Bag of Readings:
using
Documents
using
Sensors

11. © Copyright IBM Corporation 2016
IBM Accessibility Research
Dirichlet Distribution
11
Di-ri-chlet
All distributions equal
(default)
Prefer equal mixturePrefer single topic
(not supported)

12. © Copyright IBM Corporation 2016
IBM Accessibility Research
Spark ML Pipeline
12
SQLTransformer
.transform()
OneHotEncoder
.transform()
GroupByWindow
.transform()
LDA
.fit()
SQLTransformer.
transform()
OneHotEncoder
.transform()
GroupByWindow
.transform()
LDAModel
.transform()
Training Pipeline
Evaluation Pipeline
sensor1 on
sensor2 on
[ 1, 0, 0 ]
[ 0, 1, 0 ]
[ 1, 1, 0 ]training data
test data

13. © Copyright IBM Corporation 2016
IBM Accessibility Research
ADL/Sensor
Distribution
13
• Learn sensor => ADL
• Unsupervised ML
• Spark ml LDA
SensorId cooking transferring toileting bathing TV watching sleeping
I01BBB-b-nw---- 0.16 0.13 0.18 0.18 21165.05 0.14 0.16
I01BBB-b-smar2md 100.97 40366.36 4002.56 5.99 0.39 0.32 0.41
I01BBB-b-smcl010 0.56 38.24 3051.03 0.85 0.71 0.29 55928.33
I01BBB-b-smcl020 0.27 2.27 39292.91 0.34 0.27 0.36 0.58
I01BBB-b-smclbed 0.19 0.23 0.57 0.27 0.38 0.15 24340.21
I01BBB-c-scdoor2 0.08 0.09 15012.48 0.09 0.11 0.07 0.09
I01BBB-dkscdoor- 0.19 0.15 0.23 0.21 4634.85 0.16 0.21
I01BBB-dnsachar1 0.13 15921.06 0.16 0.15 0.23 0.13 0.14
I01BBB-fyscdoor- 14182.51 0.08 0.09 0.08 0.10 0.07 0.08
I01BBB-fysmar3md 13814.08 2.98 2.20 20673.86 0.31 0.28 0.28
I01BBB-fysmclent 21057.01 0.84 0.68 15147.70 0.27 0.25 0.24
I01BBB-ktnw----- 0.14 0.12 0.17 0.17 21199.14 0.13 0.14
I01BBB-ktsccplat 0.11 0.10 0.13 0.12 11546.33 0.10 0.11
I01BBB-ktscfrez- 0.25 0.16 0.22 15388.63 0.37 0.18 0.20
I01BBB-ktscfrig- 49370.50 0.08 0.08 0.09 0.10 0.07 0.07
I01BBB-ktscutenz 0.13 0.09 0.12 3670.31 0.15 0.09 0.10
I01BBB-ktsmcl--- 6637.83 0.20 0.23 0.35 0.15 0.11 0.13
I01BBB-ktspmicrw 0.24 0.16 0.26 0.37 2.14 0.45 0.39
I01BBB-ldsawashr 0.07 0.06 0.08 0.08 0.10 0.06 24336.54
I01BBB-ldscdoor1 0.08 0.08 15140.45 0.10 0.12 0.08 0.09
I01BBB-ldsmcl--- 11246.61 29988.13 5645.84 511.05 0.49 0.37 0.51
I01BBB-lrsachar1 0.06 0.06 0.08 0.06 0.07 37033.62 0.05
I01BBB-lrsmar4md 0.32 0.32 1.29 0.29 0.34 21564.24 0.21
I01BBB-lrsmcl000 1.54 40986.87 2191.49 1.04 0.32 0.42 0.32
I01BBB-lrsmcl100 4.25 48.19 24648.51 1.04 0.62 5647.11 0.27
I01BBB-lrsmcl200 0.41 0.62 68.51 0.37 0.50 40365.33 0.26
I01BBB-lrsptv--- 0.17 0.14 0.22 0.22 0.92 0.15 0.18
I01BBB-rrnw----- 0.14 0.12 0.17 0.16 21185.14 0.13 0.14
I01BBB-rrscdoor- 0.14 0.12 0.17 0.16 0.24 16324.03 0.14
I01BBB-rrsmar1md 0.24 0.16 0.19 25992.93 0.21 0.12 0.15
I01209-rrsmclshw 0.16 0.14 0.21 0.39 10417.79 0.13 0.18
I01209-rrsmclsnk 0.88 3.46 12.06 240.63 60.90 0.60 11303.46
I01209-rrsmcltoi 0.17 0.14 0.18 26898.94 0.29 0.13 0.16
Grand Total 116420.59 127361.95 109073.76 108537.23 90219.09 120939.84 115914.54

14. © Copyright IBM Corporation 2016
IBM Accessibility Research
ADL by Time Window
14
cooking transferring toileting bathing TV watching sleeping max ADL
96K windows

15. © Copyright IBM Corporation 2016
IBM Accessibility Research
Conclusions
• Tuning
• Time window: 1 minute is good (5 min was too long)
• Alpha (# concurrent ADLs)
• Ideal: small alpha (0.1, 0.01, …)
• But Spark LDA ML doesn’t allow alpha < 1.0
• Iterations: 100 is good (35 was too few)
• Choose #ADLs up front. 6?, 7?, 10? …
• No ADL looks like “dressing” or “grooming”
• Found non-standard “Watch TV” ADL
• Interpretation
• Must manually characterize sensor sets (ADLs)
• How to transfer learning across apartments (diff sensors) ?
• Encouraging results, but more work is needed
15

16. © Copyright IBM Corporation 2016
IBM Accessibility Research
Backup
16