Providing Effective Memory Retrieval Cues    through Automatic Structuring and    Augmentation of a Lifelog of Images     ...
CLARITY             [1/3]CLARITY: Centre for Sensor Web Technologies– Recently announced CSET (Centre for Science Engineer...
CLARITY               [2/3]CLARITY What ? “The Sensor Web” – Increasing availability of cheap, robust, and deployable   se...
CLARITY                   [3/3]Principal InvestigatorsPrincipal Investigators  Prof. Barry Smyth             - Personaliza...
Overview• WHAT IS LIFELOGGING?• THE HUMAN MEMORY SYSTEM• AUTOMATED PROCESSING TO SUMMARISE   SENSECAM DATA  –   Segmenting...
LifeloggingLifelogging is about digitally recording your daily lifeSometimes its for a reason Work           e.g. security...
Memory Aids Through the   Ages: Cave Paintings(approx. 30,000 years ago)             UNIVERSITY COLLEGE DUBLIN      DUBLIN...
Memory Aids Through the Ages: Storytelling        UNIVERSITY COLLEGE DUBLIN      DUBLIN CITY UNIVERSITY      TYNDALL NATIO...
Memory Aids Through the    Ages: Books(approx 5,000 years ago)             UNIVERSITY COLLEGE DUBLIN      DUBLIN CITY UNIV...
Memory Aids Through the    Ages: PERSONAL Diaries(approx 400 years ago)        UNIVERSITY COLLEGE DUBLIN      DUBLIN CITY ...
MEMEXThe first lifelogging scenario                       In 1945, Vanevar Bush wrote ‘As We                        May Th...
Lifelogging DevicesTano et. al. University of Electro‐Communications, Tokyo, Japan               UNIVERSITY COLLEGE DUBLIN...
Lifelogging DevicesLin & Hauptmann, Carnegie Mellon, PA, USA           UNIVERSITY COLLEGE DUBLIN      DUBLIN CITY UNIVERSI...
SenseCamMulti‐sensor device   Colour camera   3 accelerometers   Light meter   Passive infrared sensor1GB flash memory sto...
How to Review Images?Make a 2 minute movie of your day!                 UNIVERSITY COLLEGE DUBLIN      DUBLIN CITY UNIVERS...
Lifelogging Aiding Memory  • Preliminary Study carried out by     Cambridge Memory Clinic,     Addenbrooke’s Hospital  • 6...
SenseCam as a Memory Aid    Microsoft Research Cambridge presentation: http://research.microsoft.com/~shodges/presentation...
SenseCam as a Memory Aid    Microsoft Research Cambridge presentation: http://research.microsoft.com/~shodges/presentation...
SenseCam as a Memory Aid    Microsoft Research Cambridge presentation: http://research.microsoft.com/~shodges/presentation...
Overview• WHAT IS LIFELOGGING?• THE HUMAN MEMORY SYSTEM• AUTOMATED PROCESSING TO SUMMARISE   SENSECAM DATA  –   Segmenting...
Memory Systems of Interest•SENSORY•SHORT‐TERM•LONG‐TERM  –PROCEDURAL  –DECLARATIVE     • Semantic     • EPISODIC/ AUTOBIOG...
Cued Recall & Visual Encoding •Visual encodings are very strong (Brewer, ’88) •Encoding from same perspective/environment ...
Our Take…To effectively provide memory retrieval cues using SENSECAM we need to automatically:• “Chunk” similar images int...
A 2 MINUTE BREATHER!BBC Science Program – October 2008James May’s Big Ideas: Man‐Machine           UNIVERSITY COLLEGE DUBL...
Overview• WHAT IS LIFELOGGING?• THE HUMAN MEMORY SYSTEM• AUTOMATED PROCESSING TO SUMMARISE   SENSECAM DATA  –   Segmenting...
Daily Browser Overview          SenseCam Images of a day (about 3,000)                            Event Segmentation      ...
Event Segmentation   Breakfast        Work          Car   Talking to   colleague      Airplane                 UNIVERSITY ...
Event Segmentation                                         One Day’s ImagesRaw data                          For each imag...
How well does it work? Data divided into training and test sets with thousands of    different combinations evaluated From...
Retrieval Reminder          SenseCam Images of a day (about 3,000)                            Event Segmentation          ...
Finding similar events            Mon                                                                               Simila...
Event Retrieval1. Day of ~2,000 SenseCam images2. Segmented into ~20 events3. How to compare events to each               ...
SURF ApproachUse [Bay, ECCV 2006] algorithmHierarchical visual word vocabularyUsing 7M SURF descriptors and hierarchical K...
SURF Bi‐directional Matching        UNIVERSITY COLLEGE DUBLIN      DUBLIN CITY UNIVERSITY      TYNDALL NATIONAL INSTITUTE ...
How accurate is it?Again thousands of combinations investigated in training phase• Event retrieval works well for general ...
Importance Reminder          SenseCam Images of a day (about 3,000)                            Event Segmentation         ...
Importance                                                       • Greater emphasis is                                    ...
Automatic Face Detection                                                                          Trained on set of 1,758 ...
Novelty to Detect Importance     Mon                                                                   Similar Events - Ai...
UNIVERSITY COLLEGE DUBLIN      DUBLIN CITY UNIVERSITY      TYNDALL NATIONAL INSTITUTE    40
How well does it work? • Face Detection good at highlighting most    important events • Novelty good at detecting routine ...
Keyframe Reminder          SenseCam Images of a day (about 3,000)                            Event Segmentation           ...
Keyframe Selection Standard Approaches   – Middle Image   – Image Quality   – Image closest to others in same event   – Im...
Keyframe ExperimentsHow well does it work?• User judgements made on 2,235 events  – 6 keyframes judgements per event  – pr...
Augmentation Reminder          SenseCam Images of a day (about 3,000)                            Event Segmentation       ...
Event augmentation Here’s a SenseCam picture of me at a pier in Santa Barbara, CA. If I have GPS I can search for other pi...
Event augmentation• I receive the following “geotagged” images…• Then after some processing on text associated with these ...
Problem in selecting good tags    Tag                     #               Tag                       #        No spaces in ...
Using those tags…We can search for material from:• Flickr•Yahoo Search Engine•YouTube•MSN Search Engine•Original geotagged...
0                                                 20                                                        40            ...
AugmentationHow well does it work?• 11 users collected 1.9 million images   – from which 67 events were selected to be aug...
Overview• WHAT IS LIFELOGGING?• THE HUMAN MEMORY SYSTEM• AUTOMATED PROCESSING TO SUMMARISE   SENSECAM DATA  –   Segmenting...
Can’t “recognise” events                 Mon                 Tue                 Wed                 Thr                 F...
UNIVERSITY COLLEGE DUBLIN      DUBLIN CITY UNIVERSITY      TYNDALL NATIONAL INSTITUTE    54
Concept detection process          Wiccest          Feature                               SVM         Feature             ...
Concept Image Accuracy • Precision     – Average = 0.57     – Median = 0.60 • Judge Agreement     – Fleiss’s Kappa = 0.68 ...
ResultsBUT applying on image level isn’t so interesting       • Many SenseCam images are blurred, grainy, obscured by hand...
Where are the <eating> events?      All 200k+ images in test set     3k events                 Event Segmentation    ~90 i...
1000                                                                                                          1200        ...
Lifestyle Variationstandard deviations away from sample mean                                                              ...
Overview• WHAT IS LIFELOGGING?• THE HUMAN MEMORY SYSTEM• AUTOMATED PROCESSING TO SUMMARISE   SENSECAM DATA  –   Segmenting...
Conclusions• For a long time focus of lifelogging community was on   hardware minituratisation and storage• Recently focus...
Conclusions• Event Segmentation is pretty accurate, and VERY   fast• Event Retrieval is good for most queries and helps   ...
Conclusions• Determining event importance generally provides a   good “starter cue”• Augmentation provides many additional...
Future Work• Exploiting Augmented Images to Construct a   Narrative of One’s Tourist Trip   –Amusing/Interesting stories m...
Future Work• Diet Monitoring• Ethnographical Studies• Leveraging Other Sources of Information            UNIVERSITY COLLEG...
Data Sources: People Near Me Social Network Generation Based on real‐world interactions using  Bluetooth on mobile devices...
Data Sources: Logging Movement       UNIVERSITY COLLEGE DUBLIN      DUBLIN CITY UNIVERSITY      TYNDALL NATIONAL INSTITUTE...
Data Sources: Logging how we feel…Heart Rate MonitorBodyMedia ArmbandGalvanic Skin Response (GSR)Heat FluxSkin Temperature...
Nike+        UNIVERSITY COLLEGE DUBLIN      DUBLIN CITY UNIVERSITY      TYNDALL NATIONAL INSTITUTE    70
Data Sources:       Posture Monitoring         UNIVERSITY COLLEGE DUBLIN      DUBLIN CITY UNIVERSITY      TYNDALL NATIONAL...
Data Sources: “Web 2.0” • MS/Google Health Vaults • Flickr •MySpace •Facebook •Twitter •YouTube •BedPosted •etc.          ...
SenseCam & Memory• SenseCam may be a very powerful memory aid• In autobiographical (long‐term) memory      • “Cued Recall”...
Who knows what’s next…Mr Lee, the lifelogging cat!                   UNIVERSITY COLLEGE DUBLIN      DUBLIN CITY UNIVERSITY...
Thank You                               further information:             http://www.cdvp.dcu.ie/SenseCam       http://www....
Upcoming SlideShare
Loading in...5
×

Organising wearable camera data (given to Yahoo! Research Barcelona)

256

Published on

Presentation given to Yahoo! Research on organising lifelog data, and how it requires an understanding of human autobiographical memory

Seminar at the Multimedia Information Retrieval Group, Yahoo! Research, 11 December 2008, Barcelona, Spain.

Abstract:
Almost everything we do these days is in some way monitored or logged by computers. We've come to accept - or maybe just ignore – this massive surveillance because it brings us benefits. In the past we have been easily able to monitor activities on our PC's (e.g. documents we worked
on, e-mails received, etc.), however recently with the advent of numerous types of wearable sensors we have been able to log many more aspects of our activities. While we have the technology to log these activities, we're not really sure what applications we can develop to usefully use this data. One application of lifelogging is as a memory aid to sufferers of neurodegenerative diseases as Alzheimer’s. Indeed in studies on the human mind it has been shown that images are strongly encoded in the brain, and in this seminar we talk about our work in using visual lifelog images to provide effective memory cues. To passively capture these lifelog images we use a wearable camera called the SenseCam. This device captures an average of 2,000 images per day, which equates to over 600,000 images per year! This is too much information for any individual to sift through, therefore we firstly exploit various aspects of the human memory system to break
this data down into manageable chunks or events. Finally in order to provide additional memory cues to an individual (on say a tourist trip), we automatically augment their own lifelog images with images taken from web 2.0 sites such as Flickr. Through analysing the tags of geotagged images on Flickr, we are also automatically able to provide additional augmented material from other sites such as the Yahoo search engine for example. The talk will conclude by talking about future directions that lifelogging may take and the data sources that may drive these changes.

0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
256
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
0
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Organising wearable camera data (given to Yahoo! Research Barcelona)

  1. 1. Providing Effective Memory Retrieval Cues  through Automatic Structuring and  Augmentation of a Lifelog of Images Aiden Doherty (Supervisor: Prof. Alan F. Smeaton) Centre for Digital Video Processing (CDVP) & CLARITY: Centre for Sensor Web Technologies, Dublin City University UNIVERSITY COLLEGE DUBLIN    DUBLIN CITY UNIVERSITY    TYNDALL NATIONAL INSTITUTE  1
  2. 2. CLARITY  [1/3]CLARITY: Centre for Sensor Web Technologies– Recently announced CSET (Centre for Science Engineering &  Technology)– Funded by Science Foundation Ireland (SFI) with industry  contributions– 5 year duration, following on from previous 4‐year  “Adaptive  Information Cluster”– Administrative centre in UCD, researchers in DCU, UCD and  Tyndall Institute– Within DCU involves CDVP (Computing & EE), NCSR, Health  & Human Performance UNIVERSITY COLLEGE DUBLIN    DUBLIN CITY UNIVERSITY    TYNDALL NATIONAL INSTITUTE  2
  3. 3. CLARITY  [2/3]CLARITY What ? “The Sensor Web” – Increasing availability of cheap, robust, and deployable  sensor technologies ushering in a wave of new information  sources; – Ubiquitous, dynamic, noisy, reactive and yielding  unstructured data‐streams == sensor web – Realizing the sensor web demands a large‐scale, multi‐ disciplinary research effort == CLARITY – Moving beyond our research silos to novel research  interactions; – Demonstrator projects in: Personal health and wellness; Environmental monitoring; UNIVERSITY COLLEGE DUBLIN    DUBLIN CITY UNIVERSITY    TYNDALL NATIONAL INSTITUTE  3
  4. 4. CLARITY  [3/3]Principal InvestigatorsPrincipal Investigators Prof. Barry Smyth - Personalization, recommender systems, mobile computing Prof. Alan Smeaton - Content-based information retrieval Prof. Dermot Diamond - Materials research, wearable sensors Prof. Noel O’Connor - Audio-visual analysis, multi-modal information processing Mr. Gregory O’Hare - Ubiquitous computing, multi-agent systemsAssociate PIs Prof. Paddy Nixon - Pervasive computing, middleware, security, trust, privacy Prof. Niall Moyna - Sports Science, wearable sensing Dr. Simon Dobson - Middleware, pervasive computing Dr. Cian O’Mathuna - Sensor devices, energy-aware hardware Dr. Brian Caulfield - Physiotherapy, therapeutic gaming, wearable sensorsFunded CollaboratorsChris Bleakley (UCD), Conor Brennan (DCU), Rem Collier (UCD), Brian Corcoran (DCU),Cathal Gurrin (DCU), Neil Hurley (UCD), Lorraine McGinty (UCD), Kieran Moran (DCU),Kieran Nolan (DCU), Brendan O’Flynn (TNI), Donal O’Gorman (DCU), Brett Paull (DCU),Emanuel Popovici (TNI), Aaron Quigley (UCD), Mark Roantree (DCU) UNIVERSITY COLLEGE DUBLIN    DUBLIN CITY UNIVERSITY    TYNDALL NATIONAL INSTITUTE  4
  5. 5. Overview• WHAT IS LIFELOGGING?• THE HUMAN MEMORY SYSTEM• AUTOMATED PROCESSING TO SUMMARISE  SENSECAM DATA – Segmenting Sequences of Images into Events – Retrieval of Similar Events – Determining Important Events – Selecting Optimal Keyframe – Augmenting Events – Lifestyle Signatures• CONCLUSIONS – Future Work: Aggregating Data Sources UNIVERSITY COLLEGE DUBLIN    DUBLIN CITY UNIVERSITY    TYNDALL NATIONAL INSTITUTE  5
  6. 6. LifeloggingLifelogging is about digitally recording your daily lifeSometimes its for a reason Work e.g. security personnel, medical staff, etc.Personal e.g. diaries, etc.Sometimes its for posterityRecording vacations, family gatherings, social occasionsSometimes its because we canAnd we’re not yet sure what we’ll do with it e.g. MyLifeBits UNIVERSITY COLLEGE DUBLIN    DUBLIN CITY UNIVERSITY    TYNDALL NATIONAL INSTITUTE  6
  7. 7. Memory Aids Through the   Ages: Cave Paintings(approx. 30,000 years ago) UNIVERSITY COLLEGE DUBLIN    DUBLIN CITY UNIVERSITY    TYNDALL NATIONAL INSTITUTE  7
  8. 8. Memory Aids Through the Ages: Storytelling UNIVERSITY COLLEGE DUBLIN    DUBLIN CITY UNIVERSITY    TYNDALL NATIONAL INSTITUTE  8
  9. 9. Memory Aids Through the    Ages: Books(approx 5,000 years ago) UNIVERSITY COLLEGE DUBLIN    DUBLIN CITY UNIVERSITY    TYNDALL NATIONAL INSTITUTE  9
  10. 10. Memory Aids Through the    Ages: PERSONAL Diaries(approx 400 years ago) UNIVERSITY COLLEGE DUBLIN    DUBLIN CITY UNIVERSITY    TYNDALL NATIONAL INSTITUTE  10
  11. 11. MEMEXThe first lifelogging scenario In 1945, Vanevar Bush wrote ‘As We  May Think’… a prophetic view of  computing technology: –Hyperlinks & WWW –Modelling associative memory –Miniature wearable camera –Lifelogging UNIVERSITY COLLEGE DUBLIN    DUBLIN CITY UNIVERSITY    TYNDALL NATIONAL INSTITUTE  11
  12. 12. Lifelogging DevicesTano et. al. University of Electro‐Communications, Tokyo, Japan UNIVERSITY COLLEGE DUBLIN    DUBLIN CITY UNIVERSITY    TYNDALL NATIONAL INSTITUTE  12
  13. 13. Lifelogging DevicesLin & Hauptmann, Carnegie Mellon, PA, USA UNIVERSITY COLLEGE DUBLIN    DUBLIN CITY UNIVERSITY    TYNDALL NATIONAL INSTITUTE  13
  14. 14. SenseCamMulti‐sensor device Colour camera 3 accelerometers Light meter Passive infrared sensor1GB flash memory storageSmart image capture ~3 images/minWe’ve had access to 7 SenseCams in last 2 years UNIVERSITY COLLEGE DUBLIN    DUBLIN CITY UNIVERSITY    TYNDALL NATIONAL INSTITUTE  14
  15. 15. How to Review Images?Make a 2 minute movie of your day! UNIVERSITY COLLEGE DUBLIN    DUBLIN CITY UNIVERSITY    TYNDALL NATIONAL INSTITUTE  15
  16. 16. Lifelogging Aiding Memory • Preliminary Study carried out by  Cambridge Memory Clinic,  Addenbrooke’s Hospital • 63 year old, well‐educated married  woman, with limbic encephalitis  (usually has no memory a few days  after an event) • Attends events along with her partner UNIVERSITY COLLEGE DUBLIN    DUBLIN CITY UNIVERSITY    TYNDALL NATIONAL INSTITUTE  16
  17. 17. SenseCam as a Memory Aid Microsoft Research Cambridge presentation: http://research.microsoft.com/~shodges/presentations/UBICOMP_senseCam.pdf UNIVERSITY COLLEGE DUBLIN    DUBLIN CITY UNIVERSITY    TYNDALL NATIONAL INSTITUTE  17
  18. 18. SenseCam as a Memory Aid Microsoft Research Cambridge presentation: http://research.microsoft.com/~shodges/presentations/UBICOMP_senseCam.pdf UNIVERSITY COLLEGE DUBLIN    DUBLIN CITY UNIVERSITY    TYNDALL NATIONAL INSTITUTE  18
  19. 19. SenseCam as a Memory Aid Microsoft Research Cambridge presentation: http://research.microsoft.com/~shodges/presentations/UBICOMP_senseCam.pdf UNIVERSITY COLLEGE DUBLIN    DUBLIN CITY UNIVERSITY    TYNDALL NATIONAL INSTITUTE  19
  20. 20. Overview• WHAT IS LIFELOGGING?• THE HUMAN MEMORY SYSTEM• AUTOMATED PROCESSING TO SUMMARISE  SENSECAM DATA – Segmenting Sequences of Images into Events – Retrieval of Similar Events – Determining Important Events – Selecting Optimal Keyframe – Augmenting Events – Lifestyle Signatures• CONCLUSIONS – Future Work: Aggregating Data Sources UNIVERSITY COLLEGE DUBLIN    DUBLIN CITY UNIVERSITY    TYNDALL NATIONAL INSTITUTE  20
  21. 21. Memory Systems of Interest•SENSORY•SHORT‐TERM•LONG‐TERM –PROCEDURAL –DECLARATIVE • Semantic • EPISODIC/ AUTOBIOGRAPHICAL – “Cued Recall” better than “Free Recall” (Purdy, ’01) – Encoding has strong effect on retrieval (Godden, ’75) UNIVERSITY COLLEGE DUBLIN    DUBLIN CITY UNIVERSITY    TYNDALL NATIONAL INSTITUTE  21
  22. 22. Cued Recall & Visual Encoding •Visual encodings are very strong (Brewer, ’88) •Encoding from same perspective/environment as  viewer is powerful (Vargha‐Khadem, ’01) •Memories can be temporally encoded (Larsen, ’96) •Distinct memories are more strongly encoded (Purdy, ’01) •Memories stored by association (Baddeley, ’04) UNIVERSITY COLLEGE DUBLIN    DUBLIN CITY UNIVERSITY    TYNDALL NATIONAL INSTITUTE  22
  23. 23. Our Take…To effectively provide memory retrieval cues using SENSECAM we need to automatically:• “Chunk” similar images into distinct events• Suggest more “distinctive” events• “Associate” related events• Provide potentially additional retrieval cues from other sources UNIVERSITY COLLEGE DUBLIN    DUBLIN CITY UNIVERSITY    TYNDALL NATIONAL INSTITUTE  23
  24. 24. A 2 MINUTE BREATHER!BBC Science Program – October 2008James May’s Big Ideas: Man‐Machine UNIVERSITY COLLEGE DUBLIN    DUBLIN CITY UNIVERSITY    TYNDALL NATIONAL INSTITUTE  24
  25. 25. Overview• WHAT IS LIFELOGGING?• THE HUMAN MEMORY SYSTEM• AUTOMATED PROCESSING TO SUMMARISE  SENSECAM DATA – Segmenting Sequences of Images into Events – Retrieval of Similar Events – Determining Important Events – Selecting Optimal Keyframe – Augmenting Events – Lifestyle Signatures• CONCLUSIONS – Future Work: Aggregating Data Sources UNIVERSITY COLLEGE DUBLIN    DUBLIN CITY UNIVERSITY    TYNDALL NATIONAL INSTITUTE  25
  26. 26. Daily Browser Overview SenseCam Images of a day (about 3,000) Event Segmentation 2 Sept 06 Interactive Browser Event-Event Comparison within the Multi-day Event database Event AugmentationDay -1Day -2 Composition of the BrowserDay -3Day -4 LandmarkDay -5 Image SelectionDay -6 Novelty Calculation of 0.1 0.7 0.1 0.1 0.3 0.4 0.8 0.1 0.9 Each Event Event database containing last 7 days’ Events UNIVERSITY COLLEGE DUBLIN    DUBLIN CITY UNIVERSITY    TYNDALL NATIONAL INSTITUTE  26
  27. 27. Event Segmentation Breakfast Work Car Talking to colleague Airplane UNIVERSITY COLLEGE DUBLIN    DUBLIN CITY UNIVERSITY    TYNDALL NATIONAL INSTITUTE  27
  28. 28. Event Segmentation One Day’s ImagesRaw data For each image... • Accelerometer X/Y/Z Sensor values... • Light • Temperature • Passive Infra Red ... adjacent blocks of sensor valsSimilarity matching ...... ......120 149 289 …Normalisation & Data fusionThresholdingEvents Event-segmented images of a day UNIVERSITY COLLEGE DUBLIN    DUBLIN CITY UNIVERSITY    TYNDALL NATIONAL INSTITUTE  28
  29. 29. How well does it work? Data divided into training and test sets with thousands of  different combinations evaluated From groundtruth we noticed: Average of 1,785 images per user per day Average of 22 events groundtruthed per day Approach Recommended: Quick segmentation (sensor values only) Performance: F1 score of 60% against users’ semantic boundaries UNIVERSITY COLLEGE DUBLIN    DUBLIN CITY UNIVERSITY    TYNDALL NATIONAL INSTITUTE  29
  30. 30. Retrieval Reminder SenseCam Images of a day (about 3,000) Event Segmentation 2 Sept 06 Interactive Browser Event-Event Comparison within the Multi-day Event database Event AugmentationDay -1Day -2 Composition of the BrowserDay -3Day -4 LandmarkDay -5 Image SelectionDay -6 Novelty Calculation of 0.1 0.7 0.1 0.1 0.3 0.4 0.8 0.1 0.9 Each Event Event database containing last 7 days’ Events UNIVERSITY COLLEGE DUBLIN    DUBLIN CITY UNIVERSITY    TYNDALL NATIONAL INSTITUTE  30
  31. 31. Finding similar events Mon Similar Events - Aiden waiting for bus Tue Wed Thr Storing by  Fri association Sat Sun• Events are represented by the average values of all the  images present in that event• Investigated numerous computation approaches to match  similarity of any two given events UNIVERSITY COLLEGE DUBLIN    DUBLIN CITY UNIVERSITY    TYNDALL NATIONAL INSTITUTE  31
  32. 32. Event Retrieval1. Day of ~2,000 SenseCam images2. Segmented into ~20 events3. How to compare events to each ? other? Compare Event Averages Compare Event Averages Too CPU intensive (from middle n images)4. How do we represent events? … … UNIVERSITY COLLEGE DUBLIN    DUBLIN CITY UNIVERSITY    TYNDALL NATIONAL INSTITUTE  32
  33. 33. SURF ApproachUse [Bay, ECCV 2006] algorithmHierarchical visual word vocabularyUsing 7M SURF descriptors and hierarchical K‐Means clustering ‐vocabulary tree with 4096 leaf nodesL1 distance between histograms of visual wordsRankingRe‐rank top 20 results based on their number of bi‐directional matches UNIVERSITY COLLEGE DUBLIN    DUBLIN CITY UNIVERSITY    TYNDALL NATIONAL INSTITUTE  33
  34. 34. SURF Bi‐directional Matching UNIVERSITY COLLEGE DUBLIN    DUBLIN CITY UNIVERSITY    TYNDALL NATIONAL INSTITUTE  34
  35. 35. How accurate is it?Again thousands of combinations investigated in training phase• Event retrieval works well for general queries (69% accuracy  of top 5 results) – Can help to remove “clutter” of everyday events from visual diary• Retrieval for specific events much more challenging (30%  accuracy of top 5 results) – Less events in the collection, lack of semantic meaning• Query MAP scores ranging from 0.6% (talking to Lynda) to  94% (Michael at work on his PC) UNIVERSITY COLLEGE DUBLIN    DUBLIN CITY UNIVERSITY    TYNDALL NATIONAL INSTITUTE  35
  36. 36. Importance Reminder SenseCam Images of a day (about 3,000) Event Segmentation 2 Sept 06 Interactive Browser Event-Event Comparison within the Multi-day Event database Event AugmentationDay -1Day -2 Composition of the BrowserDay -3Day -4 LandmarkDay -5 Image SelectionDay -6 Novelty Calculation of 0.1 0.7 0.1 0.1 0.3 0.4 0.8 0.1 0.9 Each Event Event database containing last 7 days’ Events UNIVERSITY COLLEGE DUBLIN    DUBLIN CITY UNIVERSITY    TYNDALL NATIONAL INSTITUTE  36
  37. 37. Importance • Greater emphasis is  placed on important  events • Routine/mundane events  can be hidden Distinctive memories  encoded strongly UNIVERSITY COLLEGE DUBLIN    DUBLIN CITY UNIVERSITY    TYNDALL NATIONAL INSTITUTE  37
  38. 38. Automatic Face Detection Trained on set of 1,758  SenseCam images SenseCam images are  low quality Accuracy = 63% UNIVERSITY COLLEGE DUBLIN    DUBLIN CITY UNIVERSITY    TYNDALL NATIONAL INSTITUTE  38
  39. 39. Novelty to Detect Importance Mon Similar Events - Aiden waiting for bus Similar Events - Aiden at the office corridor Tue Similar Events - Aiden working on the desk Wed Unique Events Thr Fri Sat Sun Mon • Find the most dissimilar event of today by taking the previous 2 weeks into account. • Also for any event, we only look at how visually novel it is with  respect to events around the same time from other days UNIVERSITY COLLEGE DUBLIN    DUBLIN CITY UNIVERSITY    TYNDALL NATIONAL INSTITUTE  39
  40. 40. UNIVERSITY COLLEGE DUBLIN    DUBLIN CITY UNIVERSITY    TYNDALL NATIONAL INSTITUTE  40
  41. 41. How well does it work? • Face Detection good at highlighting most  important events • Novelty good at detecting routine events • Median Likert score of 4/5, so users generally  satisfied UNIVERSITY COLLEGE DUBLIN    DUBLIN CITY UNIVERSITY    TYNDALL NATIONAL INSTITUTE  41
  42. 42. Keyframe Reminder SenseCam Images of a day (about 3,000) Event Segmentation 2 Sept 06 Interactive Browser Event-Event Comparison within the Multi-day Event database Event AugmentationDay -1Day -2 Composition of the BrowserDay -3Day -4 LandmarkDay -5 Image SelectionDay -6 Novelty Calculation of 0.1 0.7 0.1 0.1 0.3 0.4 0.8 0.1 0.9 Each Event Event database containing last 7 days’ Events UNIVERSITY COLLEGE DUBLIN    DUBLIN CITY UNIVERSITY    TYNDALL NATIONAL INSTITUTE  42
  43. 43. Keyframe Selection Standard Approaches – Middle Image – Image Quality – Image closest to others in same event – Image that distinguishes event best from other  events UNIVERSITY COLLEGE DUBLIN    DUBLIN CITY UNIVERSITY    TYNDALL NATIONAL INSTITUTE  43
  44. 44. Keyframe ExperimentsHow well does it work?• User judgements made on 2,235 events – 6 keyframes judgements per event – providing a groundtruth of 13,410 judements• Selecting highest quality image works best,  although selecting middle image is also  effective UNIVERSITY COLLEGE DUBLIN    DUBLIN CITY UNIVERSITY    TYNDALL NATIONAL INSTITUTE  44
  45. 45. Augmentation Reminder SenseCam Images of a day (about 3,000) Event Segmentation 2 Sept 06 Interactive Browser Event-Event Comparison within the Multi-day Event database Event AugmentationDay -1Day -2 Composition of the BrowserDay -3Day -4 LandmarkDay -5 Image SelectionDay -6 Novelty Calculation of 0.1 0.7 0.1 0.1 0.3 0.4 0.8 0.1 0.9 Each Event Event database containing last 7 days’ Events UNIVERSITY COLLEGE DUBLIN    DUBLIN CITY UNIVERSITY    TYNDALL NATIONAL INSTITUTE  45
  46. 46. Event augmentation Here’s a SenseCam picture of me at a pier in Santa Barbara, CA. If I have GPS I can search for other pictures in the same location… UNIVERSITY COLLEGE DUBLIN    DUBLIN CITY UNIVERSITY    TYNDALL NATIONAL INSTITUTE  46
  47. 47. Event augmentation• I receive the following “geotagged” images…• Then after some processing on text associated with these images we get many  more images, and even YouTube videos at times too! UNIVERSITY COLLEGE DUBLIN    DUBLIN CITY UNIVERSITY    TYNDALL NATIONAL INSTITUTE  47
  48. 48. Problem in selecting good tags Tag # Tag # No spaces in tags e.g. •Use Yahoo Spell16Checker to ammend newyork statue 3 “statueofliberty”name unitedstates 12 thebigapple 2 nyc 11 thesphere 2 newyorkcity 9 us 2 manhattan 9 warmemorial 2 batterypark 8 worldtradecenter 2•Know place from 7GPS usa Country/region name creates a lot of  unitedstatesofamerica 6 worldwarii 2 noise e.g. “newyork”, •Use Gazetteer to get placename 2007 2 “unitedstates”, “nyc”, etc. statueofliberty 5 911 2•Use WordNet to expand possible memorial 5 eastcoastmemorial 2placenames park 4 geotagged 2 ny 3 gothamist 2 downtown 3 eagle 3 island 2 How many tags to select as text for  sculpture 2 america 3 next query?•Wearchitecture 4 go with 3 skyscraper 2 UNIVERSITY COLLEGE DUBLIN    DUBLIN CITY UNIVERSITY    TYNDALL NATIONAL INSTITUTE  48
  49. 49. Using those tags…We can search for material from:• Flickr•Yahoo Search Engine•YouTube•MSN Search Engine•Original geotagged images: Flickr & Panoramio UNIVERSITY COLLEGE DUBLIN    DUBLIN CITY UNIVERSITY    TYNDALL NATIONAL INSTITUTE  49
  50. 50. 0 20 40 60 80 100 120 0 20 40 60 80 100 120 140 160 180 Feb-04 Feb-04 Apr-04 Apr-04 Jun-04 Jun-04 Aug-04 Aug-04 Oct-04 Oct-04 Dec-04 Dec-04 Feb-05 Feb-05 Apr-05 Apr-05 Jun-05 Jun-05 Aug-05 Aug-05 Oct-05 Oct-05 Dec-05 Dec-05 PLACES Feb-06 Feb-06 big ben london Apr-06 Apr-06 ground zero new york Jun-06 Jun-06 Aug-06 Aug-06 Oct-06 Oct-06 Dec-06 Dec-06 Feb-07 Feb-07 Apr-07 Apr-07 Jun-07 Jun-07 Aug-07 Aug-07UNIVERSITY COLLEGE DUBLIN    Oct-07 Oct-07 Dec-07 Dec-07 0 20 40 60 80 100 120 140 160 180 200 0 50 100 150 200 250 300 350 400 450DUBLIN CITY UNIVERSITY    Feb-04 Feb-04 Apr-04 Apr-04 Jun-04 Jun-04 Aug-04 Aug-04 Oct-04 Oct-04 Dec-04 Dec-04 Feb-05 Feb-05 Apr-05 Apr-05 Jun-05 Jun-05 Aug-05 Aug-05TYNDALL NATIONAL INSTITUTE  Oct-05 Oct-05 Dec-05 Dec-05 Photo Upload Temporal Aspects EVENTS Feb-06 Feb-06 Apr-06 Apr-06 U2 Croke Park Dublin Jun-06 Jun-06 Aug-06 Aug-06 Oct-06 Oct-06 soccer world cup final berlin italy france Dec-06 Dec-06 Feb-07 Feb-07 Apr-07 Apr-07 Jun-07 Jun-07 Aug-07 Aug-07 Oct-07 Oct-07 Dec-07 Dec-0750
  51. 51. AugmentationHow well does it work?• 11 users collected 1.9 million images – from which 67 events were selected to be augmented• Users very satisfied with augmentation results of  famous tourist locations e.g. Sagrada Familia• Specific events still a challenge e.g. Tuesday night’s  Champions League match in Camp Nou UNIVERSITY COLLEGE DUBLIN    DUBLIN CITY UNIVERSITY    TYNDALL NATIONAL INSTITUTE  51
  52. 52. Overview• WHAT IS LIFELOGGING?• THE HUMAN MEMORY SYSTEM• AUTOMATED PROCESSING TO SUMMARISE  SENSECAM DATA – Segmenting Sequences of Images into Events – Retrieval of Similar Events – Determining Important Events – Selecting Optimal Keyframe – Augmenting Events – Lifestyle Signatures• CONCLUSIONS – Future Work: Aggregating Data Sources UNIVERSITY COLLEGE DUBLIN    DUBLIN CITY UNIVERSITY    TYNDALL NATIONAL INSTITUTE  52
  53. 53. Can’t “recognise” events Mon Tue Wed Thr Fri Sat Sun We can detect this event We know when this event is BUT We don’t RECOGNISE the event i.e. we don’t know “the what” of this event UNIVERSITY COLLEGE DUBLIN    DUBLIN CITY UNIVERSITY    TYNDALL NATIONAL INSTITUTE  53
  54. 54. UNIVERSITY COLLEGE DUBLIN    DUBLIN CITY UNIVERSITY    TYNDALL NATIONAL INSTITUTE  54
  55. 55. Concept detection process Wiccest Feature SVM Feature Classifier Fusion SVM Fusion Gabor Feature SVM Lifelog images Labeled examples Visual features Concept probability UNIVERSITY COLLEGE DUBLIN    DUBLIN CITY UNIVERSITY    TYNDALL NATIONAL INSTITUTE  55
  56. 56. Concept Image Accuracy • Precision – Average = 0.57 – Median = 0.60 • Judge Agreement – Fleiss’s Kappa = 0.68  (9 judges) • Strong correlation of 0.75  between the number of  concept training samples  and test set performance UNIVERSITY COLLEGE DUBLIN    DUBLIN CITY UNIVERSITY    TYNDALL NATIONAL INSTITUTE  56
  57. 57. ResultsBUT applying on image level isn’t so interesting • Many SenseCam images are blurred, grainy, obscured by hands, etc.HOWEVER•Considering groups of images (i.e. CONSIDERING EVENTS)  • Reduces inaccuracies • Allows us map “macro trends” UNIVERSITY COLLEGE DUBLIN    DUBLIN CITY UNIVERSITY    TYNDALL NATIONAL INSTITUTE  57
  58. 58. Where are the <eating> events? All 200k+ images in test set 3k events Event Segmentation ~90 images in event X Eating ~45 eating images in event X Kapur Thresholding • Non-parametric event X has 50% eating images • Entropy based UNIVERSITY COLLEGE DUBLIN    DUBLIN CITY UNIVERSITY    TYNDALL NATIONAL INSTITUTE  58
  59. 59. 1000 1200 1400 0 200 400 600 800 ind o s cr o r ee ha n n ou d s td o o o ff r ic pe e b u o p le il d i ng s s ky ste eri n g t re e Wh ee lUNIVERSITY COLLEGE DUBLIN    ro a ea d t ins m e in g ide eting Ve h ic le ve gDUBLIN CITY UNIVERSITY    fa c e g ra ve Num Events Across 5 Users h i c s ho p s s les p E x in g te r na l doTYNDALL NATIONAL INSTITUTE  re a o r din g researchers!!! ho to i l Probably to be h o ld i n g e t expected of 5 IT ld i n C u p p re g P ho se ne vie ntat w H ion o ri zo n sta irs59
  60. 60. Lifestyle Variationstandard deviations away from sample mean steeringWheel eating 3.5 insideVehicle vehiclesExternal reading holdingPhone 2.5 steeringWheel vehiclesExternal holdingPhone reading 1.5 eating insideVehicle 0.5 -0.5 user 1 user 2 user 3 user 4 user 5 -1.5 UNIVERSITY COLLEGE DUBLIN    DUBLIN CITY UNIVERSITY    TYNDALL NATIONAL INSTITUTE  60
  61. 61. Overview• WHAT IS LIFELOGGING?• THE HUMAN MEMORY SYSTEM• AUTOMATED PROCESSING TO SUMMARISE  SENSECAM DATA – Segmenting Sequences of Images into Events – Retrieval of Similar Events – Determining Important Events – Selecting Optimal Keyframe – Augmenting Events – Lifestyle Signatures• CONCLUSIONS – Future Work: Aggregating Data Sources UNIVERSITY COLLEGE DUBLIN    DUBLIN CITY UNIVERSITY    TYNDALL NATIONAL INSTITUTE  61
  62. 62. Conclusions• For a long time focus of lifelogging community was on  hardware minituratisation and storage• Recently focus has shifted to data management• Potential significance of SenseCam as memory aid UNIVERSITY COLLEGE DUBLIN    DUBLIN CITY UNIVERSITY    TYNDALL NATIONAL INSTITUTE  62
  63. 63. Conclusions• Event Segmentation is pretty accurate, and VERY  fast• Event Retrieval is good for most queries and helps  direct the user to “associated/related” events• Suggested Keyframes are on the whole a good  approximation of nearly all events UNIVERSITY COLLEGE DUBLIN    DUBLIN CITY UNIVERSITY    TYNDALL NATIONAL INSTITUTE  63
  64. 64. Conclusions• Determining event importance generally provides a  good “starter cue”• Augmentation provides many additional images and  is especially useful when visiting big tourist sites• Detecting semantic concepts begins to allow us to  analyse the signature of a user’s lifestyle UNIVERSITY COLLEGE DUBLIN    DUBLIN CITY UNIVERSITY    TYNDALL NATIONAL INSTITUTE  64
  65. 65. Future Work• Exploiting Augmented Images to Construct a  Narrative of One’s Tourist Trip –Amusing/Interesting stories may provide good memory  cues• Browsing Lifelog Content on Mobile Devices –Eventually integrate into cell phones• Automated Blogging UNIVERSITY COLLEGE DUBLIN    DUBLIN CITY UNIVERSITY    TYNDALL NATIONAL INSTITUTE  65
  66. 66. Future Work• Diet Monitoring• Ethnographical Studies• Leveraging Other Sources of Information UNIVERSITY COLLEGE DUBLIN    DUBLIN CITY UNIVERSITY    TYNDALL NATIONAL INSTITUTE  66
  67. 67. Data Sources: People Near Me Social Network Generation Based on real‐world interactions using  Bluetooth on mobile devices This allows us to log who is near to us at  any one time UNIVERSITY COLLEGE DUBLIN    DUBLIN CITY UNIVERSITY    TYNDALL NATIONAL INSTITUTE  67
  68. 68. Data Sources: Logging Movement UNIVERSITY COLLEGE DUBLIN    DUBLIN CITY UNIVERSITY    TYNDALL NATIONAL INSTITUTE  68
  69. 69. Data Sources: Logging how we feel…Heart Rate MonitorBodyMedia ArmbandGalvanic Skin Response (GSR)Heat FluxSkin TemperatureMovementsFoster Miller VestRespiration RateBody TemperatureHeart Rate UNIVERSITY COLLEGE DUBLIN    DUBLIN CITY UNIVERSITY    TYNDALL NATIONAL INSTITUTE  69
  70. 70. Nike+ UNIVERSITY COLLEGE DUBLIN    DUBLIN CITY UNIVERSITY    TYNDALL NATIONAL INSTITUTE  70
  71. 71. Data Sources:  Posture Monitoring UNIVERSITY COLLEGE DUBLIN    DUBLIN CITY UNIVERSITY    TYNDALL NATIONAL INSTITUTE  71
  72. 72. Data Sources: “Web 2.0” • MS/Google Health Vaults • Flickr •MySpace •Facebook •Twitter •YouTube •BedPosted •etc. UNIVERSITY COLLEGE DUBLIN    DUBLIN CITY UNIVERSITY    TYNDALL NATIONAL INSTITUTE  72
  73. 73. SenseCam & Memory• SenseCam may be a very powerful memory aid• In autobiographical (long‐term) memory • “Cued Recall” better than “Free Recall” • Visual Encoding has strong effect on retrieval•Memory studies on‐going • Cambridge, U.K. • Leeds, U.K. • Toronto, Canada • Illinois, USA • etc. UNIVERSITY COLLEGE DUBLIN    DUBLIN CITY UNIVERSITY    TYNDALL NATIONAL INSTITUTE  73
  74. 74. Who knows what’s next…Mr Lee, the lifelogging cat! UNIVERSITY COLLEGE DUBLIN    DUBLIN CITY UNIVERSITY    TYNDALL NATIONAL INSTITUTE  74
  75. 75. Thank You further information: http://www.cdvp.dcu.ie/SenseCam http://www.computing.dcu.ie/~adohertyUNIVERSITY COLLEGE DUBLIN    DUBLIN CITY UNIVERSITY    TYNDALL NATIONAL INSTITUTE  75

×