Validating the Detection of Everyday     Concepts in Visual Lifelogs   Daragh Byrne1, Aiden R. Doherty1, Cees G.M. Snoek2,...
Overview•Introduction       • Introduction to lifelogging & the challenges involved       • Why Semantic Concept Detection...
LifeloggingLifelogging is about digitally recording your daily lifeSometimes its for a reasonWork           e.g. security ...
Lifelogging DevicesTano et. al. University of Electro-Communications, Tokyo, Japan                  UNIVERSITY COLLEGE DUB...
SenseCamSenseCam is a Microsoft Research PrototypeMulti-sensor deviceColour camera3 accelerometersLight meterPassive infra...
How to Review Images?Make a 2 minute movie of your day!                 UNIVERSITY COLLEGE DUBLIN  DUBLIN CITY UNIVERSITY...
SenseCam & Memory• SenseCam may be a very powerful memory aid• In autobiographical (long-term) memory      • “Cued Recall”...
Lifelog Processing SenseCam Images of a day (about 3,000)                   Event Segmentation                            ...
Can’t “recognise” events                 Mon                 Tue                 Wed                 Thr                 F...
Contributions of this work…  •Exploration of applying semantic concept detection to  the novel domain of lifelogging  •In-...
Overview•Introduction       • Introduction to lifelogging & the challenges involved       • Why Semantic Concept Detection...
Selecting RepresentativeConcepts•A subset of 5 user’s collection was visually inspectedby playing images in video-like fas...
UNIVERSITY COLLEGE DUBLIN  DUBLIN CITY UNIVERSITY  TYNDALL NATIONAL INSTITUTE   13
Concept detection process         Wiccest         Feature                            SVM         Feature                  ...
Image confidence values                                             Vehicles External = 0.002                             ...
Where are the <eating> images?     All 200k+ images in test set                                       Eating              ...
Where are the <eating> events?      All 200k+ images in test set     3k events                 Event Segmentation    ~90 i...
Overview•Introduction       • Introduction to lifelogging & the challenges involved       • Why Semantic Concept Detection...
Experimental Setup•5 users•1 month period each•257,518 images•3,030 events•Firstly create annotated training set      • Ev...
Select image(s)                                                            Select concept                                 ...
After annotation•38,206 images annotated (training set = 14.8%)•219,312 in test set (test set = 85.2%)•THEN we validated a...
Validation tool                             Concept to                             identify                               ...
Overview•Introduction      • Introduction to lifelogging & the challenges involved      • Why Semantic Concept Detection i...
Results• Precision    – Average = 0.57    – Median = 0.60• Judge Agreement    – Fleiss’s Kappa = 0.68• Strong correlation ...
ResultsBUT applying on image level isn’t so interesting       • Many SenseCam images are blurred, grainy, obscured by     ...
0                                                                                                                  200    ...
Lifestyle Variationstandard deviations away from sample mean                                                              ...
Overview•Introduction       • Introduction to lifelogging & the challenges involved       • Why Semantic Concept Detection...
Conclusions•For a long time focus of lifelogging community wason hardware minituratisation and storage•Recently focus has ...
Conclusions• Standard concept detection techniques applied tonew exciting field of lifelogging• Extensive evaluation carri...
Conclusions• Investigating concepts at the event level is exciting     • Allows us to identify “macro” lifestyle       tre...
Future Work•Improve concept performance      • Include sensor values      • Investigate “bag of words” approach      • Ada...
Thank You                   further information:http://www.cdvp.dcu.ie/SenseCam    adoherty@computing.dcu.ie UNIVERSITY CO...
Upcoming SlideShare
Loading in …5
×

Behaviour identification from wearable camera data

324 views

Published on

Presentation on automatically identifying activities/behaviours from wearable camera images

Presented in SAMT 2008 - 3rd International Conference on Semantic and Multimedia Technologies, 3-5 December 2008, Koblenz, Germany. ISBN 978-3-540-92234-6

Abstract:
The Microsoft SenseCam is a small lightweight wearable camera used to passively capture photos and other sensor readings from a use s day-today activities. It can capture up to 3,000 images per day, equating to almost 1 million images per year. It is used to aid memory by creating a personal multimedia lifelog, or visual recording of the wearer s life. However the sheer volume of image data captured within a visual lifelog creates a number of challenges, particularly for locating relevant content. Within this work, we explore the applicability of semantic concept detection, a method often used within video retrieval, on the novel domain of visual lifelogs. A concept detector models the correspondence between low-level visual features and highlevel semantic concepts (such as indoors, outdoors, people, buildings, etc.) using supervised machine learning. By doing so it determines the probability of a concept s presence. We apply detection of 27 everyday semantic concepts on a lifelog collection composed of 257,518 SenseCam images from 5 users. The results were then evaluated on a subset of 95,907 images, to determine the precision for detection of each semantic concept and to draw some interesting inferences on the lifestyles of those 5 users. We additionally present future applications of concept detection within the domain of lifelogging.

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
324
On SlideShare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
0
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Behaviour identification from wearable camera data

  1. 1. Validating the Detection of Everyday Concepts in Visual Lifelogs Daragh Byrne1, Aiden R. Doherty1, Cees G.M. Snoek2, Gareth J.F. Jones1 and Alan F. Smeaton1 1CLARITY: Centre for Sensor Web Technologies, Dublin City University 2ISLA, University of Amsterdam UNIVERSITY COLLEGE DUBLIN  DUBLIN CITY UNIVERSITY  TYNDALL NATIONAL INSTITUTE 1
  2. 2. Overview•Introduction • Introduction to lifelogging & the challenges involved • Why Semantic Concept Detection in Lifelogs?•Concept Detection Approach • Selecting concepts • Processing concepts • Image and event thresholds•Experimental Set-Up•Results•Conclusions • Future research UNIVERSITY COLLEGE DUBLIN  DUBLIN CITY UNIVERSITY  TYNDALL NATIONAL INSTITUTE 2
  3. 3. LifeloggingLifelogging is about digitally recording your daily lifeSometimes its for a reasonWork e.g. security personnel, medical staff, etc.Personal e.g. diaries, etc.Sometimes its for posterityRecording vacations, family gatherings, social occasionsSometimes its because we canAnd we’re not yet sure what we’ll do with it e.g. MyLifeBits UNIVERSITY COLLEGE DUBLIN  DUBLIN CITY UNIVERSITY  TYNDALL NATIONAL INSTITUTE 3
  4. 4. Lifelogging DevicesTano et. al. University of Electro-Communications, Tokyo, Japan UNIVERSITY COLLEGE DUBLIN  DUBLIN CITY UNIVERSITY  TYNDALL NATIONAL INSTITUTE 4
  5. 5. SenseCamSenseCam is a Microsoft Research PrototypeMulti-sensor deviceColour camera3 accelerometersLight meterPassive infrared sensor1GB flash memory storageSmart image capture ~3 images/minSince April 2006 we’ve had two SenseCams … in 2007 wereceived 5 more UNIVERSITY COLLEGE DUBLIN  DUBLIN CITY UNIVERSITY  TYNDALL NATIONAL INSTITUTE 5
  6. 6. How to Review Images?Make a 2 minute movie of your day! UNIVERSITY COLLEGE DUBLIN  DUBLIN CITY UNIVERSITY  TYNDALL NATIONAL INSTITUTE 6
  7. 7. SenseCam & Memory• SenseCam may be a very powerful memory aid• In autobiographical (long-term) memory • “Cued Recall” better than “Free Recall” • Visual Encoding has strong effect on retrieval•Memory studies on-going • Cambridge, U.K. • Leeds, U.K. • Toronto, Canada • Illinois, USA • etc. UNIVERSITY COLLEGE DUBLIN  DUBLIN CITY UNIVERSITY  TYNDALL NATIONAL INSTITUTE 7
  8. 8. Lifelog Processing SenseCam Images of a day (about 3,000) Event Segmentation 2 Sept 06http://www.cdvp.dcu.ie/SenseCam UNIVERSITY COLLEGE DUBLIN  DUBLIN CITY UNIVERSITY  TYNDALL NATIONAL INSTITUTE 8
  9. 9. Can’t “recognise” events Mon Tue Wed Thr Fri Sat Sun We can detect this event We know when this event is BUT We don’t RECOGNISE the event i.e. we don’t know “the what” of this event UNIVERSITY COLLEGE DUBLIN  DUBLIN CITY UNIVERSITY  TYNDALL NATIONAL INSTITUTE 9
  10. 10. Contributions of this work… •Exploration of applying semantic concept detection to the novel domain of lifelogging •In-depth evaluation of concept detectors •Allows possibilities to “gist” human lifestyle activities UNIVERSITY COLLEGE DUBLIN  DUBLIN CITY UNIVERSITY  TYNDALL NATIONAL INSTITUTE 10
  11. 11. Overview•Introduction • Introduction to lifelogging & the challenges involved • Why Semantic Concept Detection in Lifelogs?•Concept Detection Approach • Selecting concepts • Processing concepts • Image and event thresholds•Experimental Set-Up•Results•Conclusions • Future research UNIVERSITY COLLEGE DUBLIN  DUBLIN CITY UNIVERSITY  TYNDALL NATIONAL INSTITUTE 11
  12. 12. Selecting RepresentativeConcepts•A subset of 5 user’s collection was visually inspectedby playing images in video-like fashion•150 concepts initially identified•Through refinement we narrowed down to 27concepts • Most representative concepts selected • Concepts should be generalisable across users & collections UNIVERSITY COLLEGE DUBLIN  DUBLIN CITY UNIVERSITY  TYNDALL NATIONAL INSTITUTE 12
  13. 13. UNIVERSITY COLLEGE DUBLIN  DUBLIN CITY UNIVERSITY  TYNDALL NATIONAL INSTITUTE 13
  14. 14. Concept detection process Wiccest Feature SVM Feature Classifier Fusion SVM Fusion Gabor Feature SVM Lifelog images Labeled examples Visual features Concept probability UNIVERSITY COLLEGE DUBLIN  DUBLIN CITY UNIVERSITY  TYNDALL NATIONAL INSTITUTE 14
  15. 15. Image confidence values Vehicles External = 0.002 Road = 0.007 Steering wheel = 0.003 Sky = 0.002 … screen = 0.863 People = 0.012 Shopping = 0.003 All values are independent UNIVERSITY COLLEGE DUBLIN  DUBLIN CITY UNIVERSITY  TYNDALL NATIONAL INSTITUTE 15
  16. 16. Where are the <eating> images? All 200k+ images in test set Eating Kapur Thresholding • Non-parametric • Entropy based UNIVERSITY COLLEGE DUBLIN  DUBLIN CITY UNIVERSITY  TYNDALL NATIONAL INSTITUTE 16
  17. 17. Where are the <eating> events? All 200k+ images in test set 3k events Event Segmentation ~90 images in event X Eating ~45 eating images in event X Kapur Thresholding • Non-parametric event X has 50% eating images • Entropy based UNIVERSITY COLLEGE DUBLIN  DUBLIN CITY UNIVERSITY  TYNDALL NATIONAL INSTITUTE 17
  18. 18. Overview•Introduction • Introduction to lifelogging & the challenges involved • Why Semantic Concept Detection in Lifelogs?•Concept Detection Approach • Selecting concepts • Processing concepts • Image and event thresholds•Experimental Set-Up•Results•Conclusions • Future research UNIVERSITY COLLEGE DUBLIN  DUBLIN CITY UNIVERSITY  TYNDALL NATIONAL INSTITUTE 18
  19. 19. Experimental Setup•5 users•1 month period each•257,518 images•3,030 events•Firstly create annotated training set • Every 5th image selected for training set UNIVERSITY COLLEGE DUBLIN  DUBLIN CITY UNIVERSITY  TYNDALL NATIONAL INSTITUTE 19
  20. 20. Select image(s) Select concept type Select individualSelect event concept UNIVERSITY COLLEGE DUBLIN  DUBLIN CITY UNIVERSITY  TYNDALL NATIONAL INSTITUTE 20
  21. 21. After annotation•38,206 images annotated (training set = 14.8%)•219,312 in test set (test set = 85.2%)•THEN we validated accuracy of detectors on test set • 9 judges to validate system concepts • Each judge shown 200 positive & 200 negative images per concept • 50 “set” positive images & 50 “set” negative images per concept shown to all users (to investigate judge agreement) • 95,907 judgments made on test set!!! UNIVERSITY COLLEGE DUBLIN  DUBLIN CITY UNIVERSITY  TYNDALL NATIONAL INSTITUTE 21
  22. 22. Validation tool Concept to identify User selects positive examplesSystem Positive & Negativesamples randomly displayed UNIVERSITY COLLEGE DUBLIN  DUBLIN CITY UNIVERSITY  TYNDALL NATIONAL INSTITUTE 22
  23. 23. Overview•Introduction • Introduction to lifelogging & the challenges involved • Why Semantic Concept Detection in Lifelogs?•Concept Detection Approach • Selecting concepts • Processing concepts • Image and event thresholds•Experimental Set-Up•Results•Conclusions • Future research UNIVERSITY COLLEGE DUBLIN  DUBLIN CITY UNIVERSITY  TYNDALL NATIONAL INSTITUTE 23
  24. 24. Results• Precision – Average = 0.57 – Median = 0.60• Judge Agreement – Fleiss’s Kappa = 0.68• Strong correlation of 0.75 between the number of concept training samples and test set performance UNIVERSITY COLLEGE DUBLIN  DUBLIN CITY UNIVERSITY  TYNDALL NATIONAL INSTITUTE 24
  25. 25. ResultsBUT applying on image level isn’t so interesting • Many SenseCam images are blurred, grainy, obscured by hands, etc.HOWEVER•Considering groups of images (i.e. CONSIDERING EVENTS) • Reduces inaccuracies • Allows us map “macro trends” UNIVERSITY COLLEGE DUBLIN  DUBLIN CITY UNIVERSITY  TYNDALL NATIONAL INSTITUTE 25
  26. 26. 0 200 400 600 800 1000 1200 1400 ind o scr or ee ha n n ou ds tdo or off ic pe e o bu ple ildi ng s sky ste eri ng tree Wh ee l roa ea d tin ins mee g ide ting Ve hic le ve g fac e gra ve Num Events Across 5 Users hic shop ss les p Ex ing ter na l doUNIVERSITY COLLEGE DUBLIN  DUBLIN CITY UNIVERSITY  TYNDALL NATIONAL INSTITUTE rea or din g researchers!!! ho toi l Probably to be ho lding et expected of 5 IT ldin Cu g p pre Pho se ne vie ntati wH on ori zo n sta irs26
  27. 27. Lifestyle Variationstandard deviations away from sample mean steeringWheel eating 3.5 insideVehicle vehiclesExternal reading holdingPhone 2.5 steeringWheel vehiclesExternal holdingPhone reading 1.5 eating insideVehicle 0.5 -0.5 user 1 user 2 user 3 user 4 user 5 -1.5 UNIVERSITY COLLEGE DUBLIN  DUBLIN CITY UNIVERSITY  TYNDALL NATIONAL INSTITUTE 27
  28. 28. Overview•Introduction • Introduction to lifelogging & the challenges involved • Why Semantic Concept Detection in Lifelogs?•Concept Detection Approach • Selecting concepts • Processing concepts • Image and event thresholds•Experimental Set-Up•Results•Conclusions • Future research UNIVERSITY COLLEGE DUBLIN  DUBLIN CITY UNIVERSITY  TYNDALL NATIONAL INSTITUTE 28
  29. 29. Conclusions•For a long time focus of lifelogging community wason hardware minituratisation and storage•Recently focus has shifted to data management•Potential significance of SenseCam as memory aid•However recent efforts only focused on “detection”,not “recognition” UNIVERSITY COLLEGE DUBLIN  DUBLIN CITY UNIVERSITY  TYNDALL NATIONAL INSTITUTE 29
  30. 30. Conclusions• Standard concept detection techniques applied tonew exciting field of lifelogging• Extensive evaluation carried out • 27 concepts selected from 257,518 images • 38,206 images annotated for training set • 95,907 test set images manually evaluated • 17 concepts with > 60% precision UNIVERSITY COLLEGE DUBLIN  DUBLIN CITY UNIVERSITY  TYNDALL NATIONAL INSTITUTE 30
  31. 31. Conclusions• Investigating concepts at the event level is exciting • Allows us to identify “macro” lifestyle trends/profiles/signatures • Enables us to compare lifestyles of individuals UNIVERSITY COLLEGE DUBLIN  DUBLIN CITY UNIVERSITY  TYNDALL NATIONAL INSTITUTE 31
  32. 32. Future Work•Improve concept performance • Include sensor values • Investigate “bag of words” approach • Adaptively learn new concepts•Use concepts in search • Perhaps along with GPS & Bluetooth•Broadcast lifestyle signature/profile • e.g. in the last week I’ve been spending a lot of time in front of the PC but not so much time in the park UNIVERSITY COLLEGE DUBLIN  DUBLIN CITY UNIVERSITY  TYNDALL NATIONAL INSTITUTE 32
  33. 33. Thank You further information:http://www.cdvp.dcu.ie/SenseCam adoherty@computing.dcu.ie UNIVERSITY COLLEGE DUBLIN  DUBLIN CITY UNIVERSITY  TYNDALL NATIONAL INSTITUTE 33

×