Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Petia Radeva
petia.ivanova@ub.edu
University of Barcelona & Computer Vision Center
Lifelogging, egocentric
vision and heal...
Social networks & lifelogging
• Facebook.
– Facebook has more than 350 million active users and more than 2.5 billion phot...
The age of the quantified self
22:51 3
Our body is radiating data
Data: loudly, continuously,
honestly and individually
He...
Which wearables do consumers plan
to buy?
• It’s expected to double by 2018, to 81.7 million users.
• Almost 2 in 5 intern...
Which wearables do consumers plan
to buy?
The Consumer Technology Association (CTA), formerly the Consumer Electronics Ass...
Lifelogging
22:51 6
Technology for life-logging is here!
Evolution of life-logging apparatus, including wearable computer, camera, and viewfin...
Wearable cameras and the life-logging trend
8
Shipments of wearable computing devices
worldwide by category from 2013 to 2...
Visual life-logging
Benefits:
• A digital memory of people you met, conversations you had, places you visited, and events ...
10 things to know about lifelogging
1. Social services get logging (Facebook, Twitter, Spotify)
2. Fitness trackers are bi...
But!
22:51 11
Ethical guidelines for wearable
cameras
• Anonimity and confidentiality: Researchers coding image data should:
– not discu...
Understanding user privacy requirements and
risks from emerging technologies
• People are bad at:
1. understanding the fut...
Benefits and potential applications
• It will take quite some time for people to feel comfortable with ‘always connected’ ...
How else can be LL useful?
1522:51
Life-logging data
16
• What we have:
22:51
Wealth of life-logging data
• We propose an energy-based approach for
motion-based event segmentation of life-
logging seq...
Extracting semantics from egocentric
images
Computer Vision allows to process and analyze huge
amount of images and extrac...
22:51 19
Deep leearning everywhere
2022:51
Deep learning applications
2122:51
From NN to CNN
2222:51
Imagenet
2322:51
ConvNets are everywhere
24From: Fei-Fei Li & Andrej Karpathy & Justin Johnson22:51
ConvNets are everywhere
25From: Fei-Fei Li & Andrej Karpathy & Justin Johnson22:51
ConvNets are everywhere
26From: Fei-Fei Li & Andrej Karpathy & Justin Johnson22:51
ConvNets are everywhere
27From: Fei-Fei Li & Andrej Karpathy & Justin Johnson22:51
ConvNets are everywhere
28From: Fei-Fei Li & Andrej Karpathy & Justin Johnson22:51
Egocentric vision
• How can deep learning help process
egocentric images?
22:51 29
Informativeness
CNN
Day Lifelog
Informative
Images
Egocentric informativeness through deep
learning
22:51 30
Visual summary through keyframe extraction (joint work UPC &
UB)
31
Results
22:51
Analysis of human interaction
32
F-formations extraction from egocentric vision
22:51
Two main questions?
• What we eat?
– Automatic food recognition vs.
Food diaries
• And how we eat?
– Automatic eating patt...
Automatic Food Analysis
• Food detection
• Food recognition
• Food environment recognition
• Eating pattern extraction
342...
Food datasets
35
Food256 - 100images/class
Classes: 256
Food101 – 101.000 images
Classes: 101
Food101+FoodCAT: 146.392 (10...
Food-related environment
recognition
Food
Non Food
...
w1
w2
wn
G
oogleNet
Softm
ax
G
AP
inception4eoutput
Deep
Convolutio...
Food environment classification
37
Bakery
Banquet hall
Bar
Butcher shop
Cafetería
Ice cream parlor
Kitchen
Kitchenette
Mar...
Image Input
Foodness Map
Extraction
Food Detection CNN
Food Recognition CNN
Food Type
Recognition
Apple
Strawberry
Food re...
Demo
3922:51
What did I do today?
40
Wearable cameras allows to visualize our diary.
Computer Vision allows to process huge amount of d...
Will life-logging and internet of things help know us better?
4122:51
How much time I spend by day on different
activities?
4222:51
My working hours
4322:51
With whom I was interacting with?
4422:51
How stressful am I in different places?
4522:51
“Six Months of My Life”
46by David El Achkar22:51
4722:51
Metabolic diseases and health
4822:51
 4.2 million die of chronic diseases
in Europe (diabetes or cancer)
linked to lack ...
Health and medical care
• Today, 88% of healthcare costs are spent
on medical care – access to physicians,
hospitals, proc...
What are we missing in health
applications?
• Today, automatically measuring physical activity is not a problem.
• But wha...
Towards healthy habits
Towards visualizing summarized lifestyle data to ease the management of the user’s healthy
habits (...
Teleassistance and home monitoring
Using wearable cameras, we can have a clear picture of the lifestyle
the person is havi...
Life-logging for cognitive treatment of people with amnesia
53
Program based on life-logging captured by a Wearable Camera...
Life-logging for MCI treatment
54
Goal: using episodic images to develop cognitive exercises and tools for memory
reforcin...
Application of Lifelogging to Migraine
• Objective Biomarkers in Migraine - how to predict a migraine attack
• How does th...
Lifelogging and wearable cameras
• Lifelogging technology is here
• Computer Vision and Machine Learning can
help process ...
Thank you!
5722:51
Deep learning applications
5822:51
Upcoming SlideShare
Loading in …5
×

Lifelogging, egocentric vision and health: how a small wearable camera can help me improve my health state

322 views

Published on

Talk of Petia Radeva 16.09.2016

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Lifelogging, egocentric vision and health: how a small wearable camera can help me improve my health state

  1. 1. Petia Radeva petia.ivanova@ub.edu University of Barcelona & Computer Vision Center Lifelogging, egocentric vision and health: how a small wearable camera can help me improve my health state 22:51 1
  2. 2. Social networks & lifelogging • Facebook. – Facebook has more than 350 million active users and more than 2.5 billion photos are uploaded every month. • Flickr.com – Flickr has around 44 million users. • Twitter.com – More than 3 million tweets are published daily. • Youtube. – Youtube is one of the Top-5 most visited websites in the world with more than 5 billion videos uploaded monthly. • Instagram – Instagram exceed 20 billion photos shared 2 “To Log or Not to Log? Privacy Risks and Solutions for Lifelogging and Continuous Activity Sharing Applications”, Blaine Price, ENISA, Centre for Research in Computing. 22:51
  3. 3. The age of the quantified self 22:51 3 Our body is radiating data Data: loudly, continuously, honestly and individually Heart rate, risk for cancer, events that change our mood, stress, breading, sleep, etc. Sensors for caregivers, mothers, fermons sensors. Sensors for behavioural modifications to be more: mindful, aware, present, … human. Lauren Constantini: Wearable tech expands human potential
  4. 4. Which wearables do consumers plan to buy? • It’s expected to double by 2018, to 81.7 million users. • Almost 2 in 5 internet users will use wearables by 2019. 22:51 4 The Consumer Technology Association (CTA), formerly the Consumer Electronics Association (CEA), surveyed 1,001 US internet users. Source: eMarketer.
  5. 5. Which wearables do consumers plan to buy? The Consumer Technology Association (CTA), formerly the Consumer Electronics Association (CEA), surveyed 1,001 US internet users. Source: eMarketer. “Wearable usage will grow by nearly 60% this year. Wearables market grows 172% in a year; 78 Million devices shipped (21 million Fitbits)” IoT Daily, Connected thinking 22:51 5
  6. 6. Lifelogging 22:51 6
  7. 7. Technology for life-logging is here! Evolution of life-logging apparatus, including wearable computer, camera, and viewfinder with wireless Internet connection. Early apparatus used separate transmitting and receiving antennas. Later apparatus evolved toward the appearance of ordinary eyeglasses in the late 1980s and early 1990s . 7 “Quantified Self & life-logging MeetsInternet of Things (IOT)”, Mazzlan Abbas. 22:51
  8. 8. Wearable cameras and the life-logging trend 8 Shipments of wearable computing devices worldwide by category from 2013 to 2015 (in millions) Wearable Camera Shipments and Revenue, World Markets: 2015-2021, Source: Tractica 22:51
  9. 9. Visual life-logging Benefits: • A digital memory of people you met, conversations you had, places you visited, and events you participated in. – This memory would be searchable, retrievable, and shareable. • A 14/7/365 monitoring of daily activities. – This data could serve as a warning system and also as a personal base upon which to diagnosis illness and to prescribe medicines. • A way of organizing, shaping, and “reading” your own life. – A complete archive of your work and play, and your work habits. Deep comparative analysis of your activities could assist your productivity, creativity, and consumptivity. • To the degree this life-log is shared, this archive of information can be leveraged to help others work, amplify social interactions, and in the biological realm, shared medical logs could rapidly advance medicine discoveries. 9 The hard part is no longer deciding what to hold on to, but how to efficiently organize it, sort it, access it, and find patterns and meaning in it. 22:51
  10. 10. 10 things to know about lifelogging 1. Social services get logging (Facebook, Twitter, Spotify) 2. Fitness trackers are big market 3. Lifelogging apps for smartphones: Saga, Instant, Narrato, OptimizeMe 4. There is dedicated hardware (wearable cameras) 5. Big tech companies are sniffing around (Sony, Samsung, Apple, etc.) 6. Wearables capture more data (smart watches and augmented glasses) 7. It really isn't a new thing (MyLifeBits since 2000s, Gordon Bell) 8. Lifelogging gets emotional (MIT project) 1. Lifelogging can be art (Alan Kwan's Bad Trip, Stephen Cartwright) 22:51 10
  11. 11. But! 22:51 11
  12. 12. Ethical guidelines for wearable cameras • Anonimity and confidentiality: Researchers coding image data should: – not discuss the content with anyone outside of the team, – not identify anyone they recognize in the images, – be aware of how sensitive the data are. • Data encryption: Confıdentiality can be protected by confıguring devices and using specialist viewing software to make the images accessible only to the research team (lost devices). – Devices should be configured so that data can only be retrieved by the research team. It should be impossible for participants or third parties who find devices to access the images. • Data storage: Collected images should be stored securely and password-protected, according to national regulations. 1222:51
  13. 13. Understanding user privacy requirements and risks from emerging technologies • People are bad at: 1. understanding the future value of revealing private information today, 2. understanding the risks from technology they have not yet used or heard of. 13 Wearable cameras can be very useful: An estimated one million Russian motorists have dashboard video cameras installed in their cars. Police officers carring video camera units and using Velcro to place these cameras in police wagons, helmet cams, ear cams, chest cams with audio capability, GPS locators, taser cams • “Even with only half of the 54 uniformed patrol officers wearing cameras at any given time, a department in USA had an 88 % decline in the number of complaints filed against officers, compared with the 12 months before the study”, The New York Times, 4th of July, 2013. The world’s leading police body worn video camera deployed by over 4000 agencies in 16 countries. 22:51
  14. 14. Benefits and potential applications • It will take quite some time for people to feel comfortable with ‘always connected’ devices that can discreetly take photos or videos. Will the benefits outweigh the negatives? “Quantified Self & life-logging Meets Internet of Things (IOT)”, Dr. Mazlan Abbas, MIMOS Berhad 1422:51
  15. 15. How else can be LL useful? 1522:51
  16. 16. Life-logging data 16 • What we have: 22:51
  17. 17. Wealth of life-logging data • We propose an energy-based approach for motion-based event segmentation of life- logging sequences of low temporal resolution • - The segmentation is reached integrating different kind of image features and classifiers into a graph-cut framework to assure consistent sequence treatment. Complete dataset of a day captured with SenseCam (more than 4,100 images 17 Choice of devise depends on: 1) where they are set: a hung up camera has the advantage that is considered more unobtrusive for the user, or 2) their temporal resolution: a camera with a low fps will capture less motion information, but we will need to process less data. We chose a SenseCam or Narrative - cameras hung on the neck or pinned on the dress that capture 2-4 fps. The wealth or the hell of life-logging data 22:51
  18. 18. Extracting semantics from egocentric images Computer Vision allows to process and analyze huge amount of images and extract the semantics from them. 1822:51 What technology to apply?
  19. 19. 22:51 19
  20. 20. Deep leearning everywhere 2022:51
  21. 21. Deep learning applications 2122:51
  22. 22. From NN to CNN 2222:51
  23. 23. Imagenet 2322:51
  24. 24. ConvNets are everywhere 24From: Fei-Fei Li & Andrej Karpathy & Justin Johnson22:51
  25. 25. ConvNets are everywhere 25From: Fei-Fei Li & Andrej Karpathy & Justin Johnson22:51
  26. 26. ConvNets are everywhere 26From: Fei-Fei Li & Andrej Karpathy & Justin Johnson22:51
  27. 27. ConvNets are everywhere 27From: Fei-Fei Li & Andrej Karpathy & Justin Johnson22:51
  28. 28. ConvNets are everywhere 28From: Fei-Fei Li & Andrej Karpathy & Justin Johnson22:51
  29. 29. Egocentric vision • How can deep learning help process egocentric images? 22:51 29
  30. 30. Informativeness CNN Day Lifelog Informative Images Egocentric informativeness through deep learning 22:51 30
  31. 31. Visual summary through keyframe extraction (joint work UPC & UB) 31 Results 22:51
  32. 32. Analysis of human interaction 32 F-formations extraction from egocentric vision 22:51
  33. 33. Two main questions? • What we eat? – Automatic food recognition vs. Food diaries • And how we eat? – Automatic eating pattern extraction – when, where, how, how long, with whom, in which context? 3322:51
  34. 34. Automatic Food Analysis • Food detection • Food recognition • Food environment recognition • Eating pattern extraction 3422:51
  35. 35. Food datasets 35 Food256 - 100images/class Classes: 256 Food101 – 101.000 images Classes: 101 Food101+FoodCAT: 146.392 (101.000+45.392) Classes: 131 EgocentricFood: 5038 images Classes: 9 22:51
  36. 36. Food-related environment recognition Food Non Food ... w1 w2 wn G oogleNet Softm ax G AP inception4eoutput Deep Convolution X FAM Bounding Box G eneration 36 Our localization method based on Global Average Pooling (GAP) that produces a Food Activation Map (FAM). Examples of localization and recognition on UECFood256 (top) and EgocentricFood (bottom). Ground truth is shown in green and our method in blue. Marc Bolaños, Petia Radeva: Simultaneous Food Localization and Recognition. submitted to ICPR’16, arXiv.org>cs>arXiv:1604.07953, 2016. 22:51
  37. 37. Food environment classification 37 Bakery Banquet hall Bar Butcher shop Cafetería Ice cream parlor Kitchen Kitchenette Market Pantry Picnic Area Restaurant Restaurant Kitchen Restaurant Patio Supermarket Candy store Coffee shop Dinette Dining room Food court Galley Classification results: 0.92 - Food-related vs. Non-food-related 0.68 - 22 classes of Food-related categories22:51
  38. 38. Image Input Foodness Map Extraction Food Detection CNN Food Recognition CNN Food Type Recognition Apple Strawberry Food recognition Results: TOP-1 74.7% TOP-5 91.6% SoA (Bossard,2014): TOP-1 56,4% 22:51 38
  39. 39. Demo 3922:51
  40. 40. What did I do today? 40 Wearable cameras allows to visualize our diary. Computer Vision allows to process huge amount of data and automatically extracts the semantics from it: where, what, who, how, etc. 22:51
  41. 41. Will life-logging and internet of things help know us better? 4122:51
  42. 42. How much time I spend by day on different activities? 4222:51
  43. 43. My working hours 4322:51
  44. 44. With whom I was interacting with? 4422:51
  45. 45. How stressful am I in different places? 4522:51
  46. 46. “Six Months of My Life” 46by David El Achkar22:51
  47. 47. 4722:51
  48. 48. Metabolic diseases and health 4822:51  4.2 million die of chronic diseases in Europe (diabetes or cancer) linked to lack of physical activities and unhealthy diet.  Physical activities can increase lifespan by 1.5-3.7 years.  Obesity is a chronic disease associated with huge economic, social and personal costs.  Risk factors for cancers, cardiovascular and metabolic disorders and leading causes of premature mortality worldwide.
  49. 49. Health and medical care • Today, 88% of healthcare costs are spent on medical care – access to physicians, hospitals, procedures, drugs, etc. • However, medical care only accounts for approximately 10% of a person’s health. • Approximately half the decline in U.S. Deaths from coronary heart disease from 1980 through 2000 may be attributable to reductions in major risk factors (systolic blood pressure, smoking, physical inactivity). 4922:51
  50. 50. What are we missing in health applications? • Today, automatically measuring physical activity is not a problem. • But what about food and nutrition? – State of the art: Nutritional health apps are based on manual food diaries. 22:51 50 Sparkpeople LoseIt! MyFitnessPal Cronometer Fatsecret
  51. 51. Towards healthy habits Towards visualizing summarized lifestyle data to ease the management of the user’s healthy habits (sedentary lifestyles, nutritional activity, etc.). Life-logging can help us to extract our nutritional habits: taking photos of our everyday life and being able to analyse what we eat, starting by the dish recognition, where we eat, with whom we eat, how we eat. 22:51 51
  52. 52. Teleassistance and home monitoring Using wearable cameras, we can have a clear picture of the lifestyle the person is having during the whole day: • Is he/she active or passive – doing domestic duties, – performing intellectual activities like reading, – coping with daily functions, etc. 22:51 52
  53. 53. Life-logging for cognitive treatment of people with amnesia 53 Program based on life-logging captured by a Wearable Camera recording specific autobiographical episodes for stimulating posteriorly episodic memory function. Using wearable cameras and looking at their autobiographic experiences, people with amnesia improved their memory and cognitive facilities by using episodic images. Claire, a 49-year-old former nurse, six years ago suffered brain damage due to a rare viral infection called herpes encephalitis. Now an amnesiac who is unable to recognize faces, Claire lives in a world in which even her lifelong friends appear as strangers. 22:51
  54. 54. Life-logging for MCI treatment 54 Goal: using episodic images to develop cognitive exercises and tools for memory reforcing of MCI and Alzheimer people. To explore the association between changes in cognitive, functional and emotional outcomes. 22:51
  55. 55. Application of Lifelogging to Migraine • Objective Biomarkers in Migraine - how to predict a migraine attack • How does the brain interact with the environment - the cues that ensure adaptation 55 Lifelogging can assure individualized tools to detect which are the triggers of the migraine for an individual. 22:51
  56. 56. Lifelogging and wearable cameras • Lifelogging technology is here • Computer Vision and Machine Learning can help process big image data • How to get profit of it to help improve our health?! 22:51 56
  57. 57. Thank you! 5722:51
  58. 58. Deep learning applications 5822:51

×