Do try this at home! Living with In-Home Health Monitoring Systems Beth Logan Intel Digital Health Product Research and Incubation With collaborators from Intel Research Seattle, MIT and University of Washington

Where would you rather live as you age? Or in your home? Likely cheaper with more privacy In a nursing home? More companionship Less privacy Far away? More expensive Easy monitoring

Or in your home?

Likely cheaper with more privacy

In a nursing home?

More companionship

Less privacy

Far away?

More expensive

Easy monitoring

Barriers to Staying Home Alone Disabilities Unable to perform ADLs Cognition problems Fear of worst case scenarios Falls, fires, heat/air-conditioning failures, break-ins Burden on caregivers too great Constant visits or phone calls needed Can sensor-based activity monitoring increase quality of life for elders home alone and reduce the effort of their caregivers?

Disabilities

Unable to perform ADLs

Cognition problems

Fear of worst case scenarios

Falls, fires, heat/air-conditioning failures, break-ins

Burden on caregivers too great

Constant visits or phone calls needed

Can sensor-based activity monitoring increase quality of life for elders home alone and reduce the effort of their caregivers?

ADL Monitoring Wish List Detect basic ADLs with good accuracy Eating, movement, bathroom use, etc Possibly detect emergencies Minimal invasion of privacy Easy installation Sensors are unobtrusive, work flawlessly and need little maintenance Wearables are a pleasure to wear and easy to maintain User understands enough about the system to fix small problems Remote maintenance for upgrades and major problems

Detect basic ADLs with good accuracy

Eating, movement, bathroom use, etc

Possibly detect emergencies

Minimal invasion of privacy

Easy installation

Sensors are unobtrusive, work flawlessly and need little maintenance

Wearables are a pleasure to wear and easy to maintain

User understands enough about the system to fix small problems

Remote maintenance for upgrades and major problems

Baby Steps Monitor people’s activities in a sensorized home Questions Can activities be detected with useful accuracy? Do people like/mind living in a sensorized home?

Monitor people’s activities in a sensorized home

Questions

Can activities be detected with useful accuracy?

Do people like/mind living in a sensorized home?

MIT House_n PlaceLab Condo near MIT Hundreds of sensors Video for annotation Subjects live undisturbed for weeks or months Data uploaded weekly Residents have ‘veto rights’

Condo near MIT

Hundreds of sensors

Video for annotation

Subjects live undisturbed for weeks or months

Data uploaded weekly

Residents have ‘veto rights’

Built-In Sensors 206 inputs total 101 wired reed switches 14 water flow 37 current 36 temperature 1 pressure 10 humidity 1 gas 6 light Barometric pressure Humidity Temperature IR video camera Color video camera Top-down camera Light sensor Microphone Switches to detect open/close Temperature Water flow Current

206 inputs total

101 wired reed switches

14 water flow

37 current

36 temperature

1 pressure

10 humidity

1 gas

6 light

RFID Tags and Bracelet 435 tags installed throughout condo RFID Bracelet RFID Tags

435 tags installed throughout condo

Motion Based Sensors 281 inputs total 265 on-object accelerometers 2x3axis on-body accelerometers 10 infra-red motion detectors

281 inputs total

265 on-object accelerometers

2x3axis on-body accelerometers

10 infra-red motion detectors

Typical study Joint with Intel 2007 Studied a young married couple living for 10 weeks in the PlaceLab 104 hours of data annotated Covers 15 days from middle of experiment Chose periods when male subject was home Independent annotator Annotated male’s activities from the video Used a 98 activity ontology Limited/no video in bathrooms and bedroom; still had audio

Studied a young married couple living for 10 weeks in the PlaceLab

104 hours of data annotated

Covers 15 days from middle of experiment

Chose periods when male subject was home

Independent annotator

Annotated male’s activities from the video

Used a 98 activity ontology

Limited/no video in bathrooms and bedroom; still had audio

Living in a sensorized home Participants maintained as normal routine as possible Brought many possessions into the apartment to feel “at home” Wore some sensors Both subjects wore accelerometers on wrist, hips and thigh Male subject wore a bulky RFID bracelet Installed some sensors

Participants maintained as normal routine as possible

Brought many possessions into the apartment to feel “at home”

Wore some sensors

Both subjects wore accelerometers on wrist, hips and thigh

Male subject wore a bulky RFID bracelet

Installed some sensors

Participants’ reactions “ We weren’t as conscious as I thought we would be, it was actually kind of natural being here…I didn’t notice some things as much as I thought I would, like the cameras.” Male subject “ I wasn’t bothered by it, really at all, I thought I might get weirded out every once in a while, but there were very few times where I was totally tired of being in the project, and I felt pretty comfortable here” Female subject

“ We weren’t as conscious as I thought we would be, it was actually kind of natural being here…I didn’t notice some things as much as I thought I would, like the cameras.” Male subject

“ I wasn’t bothered by it, really at all, I thought I might get weirded out every once in a while, but there were very few times where I was totally tired of being in the project, and I felt pretty comfortable here” Female subject

What do people do at home? Most and least observed activities in 104h annotated set, male subject only Activity Total Observed Time Using a computer 1866 min Background listening to music or radio 813 min Actively watching tv or movies 732 min Sleeping deeply 728 min Reading book/paper/magazine 359 min Preparing a snack 44 s Leaving the home 42 s Making the bed 34 s Washing hands 24 s Drying dishes 6 s

Most and least observed activities in 104h annotated set, male subject only

Activity Recognition Results Excellent  Poor  Sensors

Take-aways (more detail in the paper) People behave unpredictably “in the wild” Revealed by our long-term study with unaffiliated subjects and annotator Interrupted activities, multi-tasking, activities in multiple and unexpected locations Many activities are short and infrequent Collecting sufficient “real life” data very challenging Location “good enough” for recognizing many simple activities Likely not good enough for fine-grained activities Even with 100s of sensors, our sensor density was sometimes insufficient Opportunity for new sensors

People behave unpredictably “in the wild”

Revealed by our long-term study with unaffiliated subjects and annotator

Interrupted activities, multi-tasking, activities in multiple and unexpected locations

Many activities are short and infrequent

Collecting sufficient “real life” data very challenging

Location “good enough” for recognizing many simple activities

Likely not good enough for fine-grained activities

Even with 100s of sensors, our sensor density was sometimes insufficient

Opportunity for new sensors

Next Steps Move to in-home installed systems Address lack of data by studying ways users can give feedback to an adaptive system Ongoing work with MIT Shift focus to older subjects Collaboration with Intel Research Seattle (rest of this talk) FYI PlaceLab annotated dataset available online Sensor data, activity ontology, annotations http://architecture.mit.edu/house_n/data/PlaceLab/PLCouple1.htm

Move to in-home installed systems

Address lack of data by studying ways users can give feedback to an adaptive system

Ongoing work with MIT

Shift focus to older subjects

Collaboration with Intel Research Seattle

(rest of this talk)

FYI PlaceLab annotated dataset available online

Sensor data, activity ontology, annotations

http://architecture.mit.edu/house_n/data/PlaceLab/PLCouple1.htm

Technologies for Long-Term Care (TLC) Joint with Intel Research Seattle Sensors in elders’ homes track activities and generate a summary display for caregivers RFID bracelet and RFID tags “ shake” sensors Display

Sensors in elders’ homes track activities and generate a summary display for caregivers

Display Detail 5 day trend Eating and ambulation timeline Today’s grooming and medication status Customizable photo background

How to have an experiment in your home… My official roles Inference Lead responsible for algorithms to detect activities Middleware co-designer and contributor My unofficial roles Chief QA engineer Chief guinea pig Only way to get the data I needed to design the inference!

My official roles

Inference Lead responsible for algorithms to detect activities

Middleware co-designer and contributor

My unofficial roles

Chief QA engineer

Chief guinea pig

Only way to get the data I needed to design the inference!

12 shake sensors 10 RFID tags and RFID bracelet 1 aggregator (laptop) with attached 802.15.4 antenna Wore a phone camera around neck for 1 week to get ground truth Kept display at work to easily show demos Home Deployment Sep 2007 – Oct 2008

12 shake sensors

10 RFID tags and RFID bracelet

1 aggregator (laptop) with attached 802.15.4 antenna

Wore a phone camera around neck for 1 week to get ground truth

Kept display at work to easily show demos

Technical Experience Ease of install and setup 9/10 Some issues with sensors falling off at first Good debugging beforehand paid off Some tweaking of inference Maintenance 6/10 Constantly changing shake sensor batteries – typically one per week Bracelet needed as much attention as a cell phone Some work to keep the Ceiva display up and running Aggregator had 0 downtime due to bugs or failures! Ease of uninstall 10/10 no marks anywhere!

Ease of install and setup 9/10

Some issues with sensors falling off at first

Good debugging beforehand paid off

Some tweaking of inference

Maintenance 6/10

Constantly changing shake sensor batteries – typically one per week

Bracelet needed as much attention as a cell phone

Some work to keep the Ceiva display up and running

Aggregator had 0 downtime due to bugs or failures!

Ease of uninstall 10/10

no marks anywhere!

User Experience Aesthetics 8/10 Over time found a way to hide most sensors but hated having the laptop and the antenna in the dining room Ease of use Barely noticed shake sensors 9/10 Hated the bracelet after about a month 1/10 Fragile, uncomfortable, needed constant attention, too ugly to wear around guests Friends and family 10/10 (non-bracelet) Many people didn’t even notice the experiment until I pointed it out! I have plenty of computers Only wore the bracelet around my boyfriend Privacy No issues with the main experiment 9/10 Hated wearing camera phone for ground truth 0/10 bulky and felt very invasive Antenna and laptop

Aesthetics 8/10

Over time found a way to hide most sensors but hated having the laptop and the antenna in the dining room

Ease of use

Barely noticed shake sensors 9/10

Hated the bracelet after about a month 1/10

Fragile, uncomfortable, needed constant attention, too ugly to wear around guests

Friends and family 10/10 (non-bracelet)

Many people didn’t even notice the experiment until I pointed it out!

I have plenty of computers

Only wore the bracelet around my boyfriend

Privacy

No issues with the main experiment 9/10

Hated wearing camera phone for ground truth 0/10

bulky and felt very invasive

My Negative Bias I was developer, maintainer and user Typically if there was a problem, I owned it Felt like I was bringing work home I had the display screen at work, not home Much more public place I don’t (yet!) need TLC A burden, especially maintaining the bracelet

I was developer, maintainer and user

Typically if there was a problem, I owned it

Felt like I was bringing work home

I had the display screen at work, not home

Much more public place

I don’t (yet!) need TLC

A burden, especially maintaining the bracelet

Findings from TLC Elder Study 2008 Interviews from 4 elders, 2 formal, 2 family caregivers in Seattle area Roughly 10,000 hrs of data 3.5 days of camera ground truth Experience from 9 installs Still finishing the study 14 elders total but don’t expect major changes to findings

Interviews from 4 elders, 2 formal, 2 family caregivers in Seattle area

Roughly 10,000 hrs of data

3.5 days of camera ground truth

Experience from 9 installs

Still finishing the study

14 elders total but don’t expect major changes to findings

Finding: TLC is Useful! “ I think it’s a marvelous thing… my sister will call me up and say “I don’t have any apples, why haven’t you eaten ?” Elder “ He brushes his teeth more often now because I can keep track. That wasn't something I really did before.” Caregiver “ [T]here's footprints all over it-- so she wasn't sleeping last night … So, I won't bother her for a while.” Caregiver

“ I think it’s a marvelous thing… my sister will call me up and say “I don’t have any apples, why haven’t you eaten ?” Elder

“ He brushes his teeth more often now because I can keep track. That wasn't something I really did before.” Caregiver

“ [T]here's footprints all over it-- so she wasn't sleeping last night … So, I won't bother her for a while.” Caregiver

Finding: TLC is Easy to Use Elder’s ability to tolerate and manage sensor bracelet was a concern Feedback from elders is reassuring: “ Ain't nothing to it. Just takes 30 seconds. I take it off when I leave the house, but sometimes I won't take it off for a week…” TLC elders wear bracelets on over 95% of days.

Elder’s ability to tolerate and manage sensor bracelet was a concern

Feedback from elders is reassuring:

“ Ain't nothing to it. Just takes 30 seconds. I take it off when I leave the house, but sometimes I won't take it off for a week…”

TLC elders wear bracelets on over 95% of days.

Finding: TLC is Delightful Attractive design and appealing backgrounds are a key part of the displays “ I love it. It's interesting… I wish I could get the cat to come back more. It comes only every week and a half.” Elder

Attractive design and appealing backgrounds are a key part of the displays

“ I love it. It's interesting… I wish I could get the cat to come back more. It comes only every week and a half.” Elder

Finding: TLC is Well Behaved False-positive rates for activity recognition are under 10% Very simple inference (Naïve Bayes + rules) Acceptable for human-processed output Installation time is under 2 hours per user Failures are as expected from pilots Picture frame backup Battery drain Bracelet antenna damage

False-positive rates for activity recognition are under 10%

Very simple inference (Naïve Bayes + rules)

Acceptable for human-processed output

Installation time is under 2 hours per user

Failures are as expected from pilots

Picture frame backup

Battery drain

Bracelet antenna damage

Are we there yet? We’re making good progress! Heartening to see our system was well received and even liked Debugging at home Great way to increase robustness and get useful feedback instead of focusing on technical problems Sometimes painful but a good way to show faith in a system Definitely not 100% predictive of real users

We’re making good progress!

Heartening to see our system was well received and even liked

Debugging at home

Great way to increase robustness and get useful feedback instead of focusing on technical problems

Sometimes painful but a good way to show faith in a system

Definitely not 100% predictive of real users

Where to next? We are not far away from useful systems! Already some products on the market Biggest challenges Installation and maintenance Lack of real-world data Work on adaptive systems Make anonymized data public? CHI2009 workshop "Developing Shared Home Behavior Datasets” Move beyond sensing to actuation

We are not far away from useful systems!

Already some products on the market

Biggest challenges

Installation and maintenance

Lack of real-world data

Work on adaptive systems

Make anonymized data public?

CHI2009 workshop "Developing Shared Home Behavior Datasets”

Move beyond sensing to actuation

Collaborators Intel Matthai Philipose, TLC project lead Jennifer Healey, Brad Needham, Ken Lafond, Adam Rea, Polly Powledge, Beverly Harrison, Jean-Manuel Van Thong, Scott Blackwell. MIT House_n Stephen Intille, Director of Technology Emmanuel Munguia-Tapia, Jennifer Beudin University of Washington School of Public Health Sheri Reder, Clinical Instructor Susan Hedrick, Professor

Intel

Matthai Philipose, TLC project lead

Jennifer Healey, Brad Needham, Ken Lafond, Adam Rea, Polly Powledge, Beverly Harrison, Jean-Manuel Van Thong, Scott Blackwell.

MIT House_n

Stephen Intille, Director of Technology

Emmanuel Munguia-Tapia, Jennifer Beudin

University of Washington School of Public Health

Sheri Reder, Clinical Instructor

Susan Hedrick, Professor

Backup