Managing a Life of Lifelogged    SenseCam Images                   Aiden R. Doherty  CLARITY: Centre for Sensor Web Techno...
The CLARITY SenseCam team Alan Smeaton                                     Noel O’Connor Gareth Jones                     ...
Thanks to …  Microsoft Research (Cambridge)                      (for SenseCams)         Science Foundation Ireland       ...
Overview• OUR SENSECAM DATA COLLECTION  – CLARITY  – Visual Lifelogging Analysis• BROWSING & SEARCHING SENSECAM DATA• SENS...
CLARITY             [1/3]CLARITY: Centre for Sensor Web Technologies– CSET (Centre for Science Engineering & Technology) f...
CLARITY              [2/3]CLARITY What ? “The Sensor Web” – Increasing availability of cheap, robust, and deployable   sen...
CLARITY                   [3/3]Principal InvestigatorsPrincipal Investigators  Prof. Barry Smyth             - Personaliza...
LifeloggingLifelogging is about digitally recording your daily lifeSometimes its for a reason Work           e.g. security...
Visual Lifelogging DevicesMuch past research focus on miniaturising hardware and increasing battery‐life + storage e.g. vi...
SenseCamSenseCam is a Microsoft Research PrototypeMulti‐sensor deviceColour camera3 accelerometersLight meterPassive infra...
SenseCam images       UNIVERSITY COLLEGE DUBLIN      DUBLIN CITY UNIVERSITY      TYNDALL NATIONAL INSTITUTE    11
Addenbrooke’s: SenseCam Work  Preliminary Study carried out by Cambridge Memory Clinic,      Addenbrooke’s Hospital  63 ye...
Addenbrooke’s: SenseCam Work    Microsoft Research Cambridge presentation: http://research.microsoft.com/~shodges/presenta...
Addenbrooke’s: SenseCam Work    Microsoft Research Cambridge presentation: http://research.microsoft.com/~shodges/presenta...
Addenbrooke’s: SenseCam Work    Microsoft Research Cambridge presentation: http://research.microsoft.com/~shodges/presenta...
~4,000,000 SenseCam Images• One user wearing SC for over 3 years  – Each with GPS position!  – Many other users too• Exper...
Image Quality Analysis400                                        346350300                                                ...
Overview• OUR SENSECAM DATA COLLECTION• BROWSING & SEARCHING SENSECAM DATA  – Event Segmentation/Searching/Interest/Augmen...
Our Take…To effectively provide memory retrieval cues using SENSECAM we need to automatically:• Group similar images into ...
Daily Browser Overview SenseCam Images of a day (about 3,000)                   Event Segmentation                        ...
Visual Search Facilities          SenseCam Images of a day (about 3,000)                            Event Segmentation    ...
Selecting Event “Keyframe”          SenseCam Images of a day (about 3,000)                            Event Segmentation  ...
Suggest Interesting Events                Mon                                                                             ...
Event augmentation Here’s a SenseCam picture of Aiden at a pier in Santa Barbara, CA. If he has GPS he can search for othe...
Event augmentation – more cues• He receives the following “geotagged” images…• Then after some processing on text associat...
Event AugmentationDoes it work?Yes – we have it operational from 6 image sources, tested and evaluated with users.Bringing...
UNIVERSITY COLLEGE DUBLIN      DUBLIN CITY UNIVERSITY      TYNDALL NATIONAL INSTITUTE    27
Released Software                                                    Features:                                            ...
Event Segmentation S/W         • Carnegie Mellon University         •CWI, Amsterdam         •Lulua University of Technolog...
Overview• OUR SENSECAM DATA COLLECTION• BROWSING & SEARCHING SENSECAM DATA• SENSECAM SUMMARISATION: THE NEXT   GENERATION ...
Dublin SenseCam WorkActivityRecognition27 “concepts”Outputs manually judgedon ~95k images (5 users)                UNIVERS...
Comparison of Lifestyle Within                                                           Social Groups                    ...
Dietry habitsConsider using even only the “Eating” concept...• Detect events where user is eating• Allows us/family/dietat...
Cathal    |       HELP   |   ABOUT      T   F     S   S    M    T    W        T   F   S   S   M   T     W    T     F   S  ...
Advanced Image Matching                                                            Each feature                           ...
Setting Detection – Watching TV        UNIVERSITY COLLEGE DUBLIN      DUBLIN CITY UNIVERSITY      TYNDALL NATIONAL INSTITU...
Setting Detection – In the Park        UNIVERSITY COLLEGE DUBLIN      DUBLIN CITY UNIVERSITY      TYNDALL NATIONAL INSTITU...
Trajectory Estimation                                                               Area Map                              ...
Trajectory Estimation Results                                                                                             ...
Other Data SourcesSENSECAM: Images                                                                                        ...
Using Context in Personal Information Management• Represent events as text documents, then “Google”  them • Search using k...
iCLIPS Browsing InterfacePresent landmarks: real life events (Photos) and computer activities (Keywords and     Thumbnails...
Overview• OUR SENSECAM DATA COLLECTION• BROWSING & SEARCHING SENSECAM DATA• SENSECAM SUMMARISATION: THE NEXT   GENERATION•...
Lifelogs & StorytellingLifelogs offer huge opportunity for telling life stories.The Need for Narrative:1. Humans like stor...
Lifelogs & StorytellingClear Challenges:1. What components of a lifelog should be used in the   composition of digital lif...
Classifying SC Motion Data• Use accelerometer data to identify various states: sitting, walking, running, driving, on bus,...
Designing for Older AdultsAreas Affected by Ageing Implications for DesignCognitive Skills -                              ...
Designing for Older Adults         UNIVERSITY COLLEGE DUBLIN      DUBLIN CITY UNIVERSITY      TYNDALL NATIONAL INSTITUTE  ...
Designing for Older Adults         UNIVERSITY COLLEGE DUBLIN      DUBLIN CITY UNIVERSITY      TYNDALL NATIONAL INSTITUTE  ...
CLARITY + U. LeedsMartin Conway & Chris Moulin, Institute of  Psychological Sciences, University Leeds1 healthy subject ‐>...
Recollective experience as cued by SenseCam stills50 “keyframe” images reviewed – 2 months between viewingsConsistency of ...
Episodic details 1when / where / who / what           UNIVERSITY COLLEGE DUBLIN      DUBLIN CITY UNIVERSITY      TYNDALL N...
Episodic details 2       UNIVERSITY COLLEGE DUBLIN      DUBLIN CITY UNIVERSITY      TYNDALL NATIONAL INSTITUTE    53
Next – Influence F‐R shift                                   R                                         F                  ...
M                                            0.00                                                   0.02                  ...
Results so farThere is no considerable overlap between the most important browser    events and those recalled by CGr = .1...
Next?Using user generated important events to help train   system to improve…How does user recollection of “free recall” m...
Summary• More SenseCams  – we’d love 50+ of them!• Increased accuracy/flexibility in recognising a   person’s lifestyle   ...
Managing a Life of Lifelogged    SenseCam Images                     Aiden R. Doherty                                     ...
Upcoming SlideShare
Loading in …5
×

Bio-mimicry of human memory using wearable camera data

501 views

Published on

Presentation on supporting human autobiographical memory through an automated system to organise wearable camera images

Seminar to the MemoryLane Research Team, Luleå University of Technology, 18 August 2009, Luleå, Sweden.

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
501
On SlideShare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
0
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Bio-mimicry of human memory using wearable camera data

  1. 1. Managing a Life of Lifelogged SenseCam Images Aiden R. Doherty CLARITY: Centre for Sensor Web Technologies, Dublin City University, Ireland UNIVERSITY COLLEGE DUBLIN    DUBLIN CITY UNIVERSITY    TYNDALL NATIONAL INSTITUTE  1
  2. 2. The CLARITY SenseCam team Alan Smeaton Noel O’Connor Gareth Jones Cathal Gurrin Hyowon Lee Ciarán Ó Conaire Aiden Doherty Daragh Byrne Liadh Kelly Yi (Yuki) Chen Zhengwei Qiu Peng Wang Niamh Caprani Carolina Camacho Dian Zhang UNIVERSITY COLLEGE DUBLIN    DUBLIN CITY UNIVERSITY    TYNDALL NATIONAL INSTITUTE  2
  3. 3. Thanks to … Microsoft Research (Cambridge) (for SenseCams) Science Foundation Ireland Science Foundation Ireland & Science Foundation IrelandUNIVERSITY COLLEGE DUBLIN    DUBLIN CITY UNIVERSITY    TYNDALL NATIONAL INSTITUTE  3
  4. 4. Overview• OUR SENSECAM DATA COLLECTION – CLARITY – Visual Lifelogging Analysis• BROWSING & SEARCHING SENSECAM DATA• SENSECAM SUMMARISATION: THE NEXT  GENERATION• THE FUTURE UNIVERSITY COLLEGE DUBLIN    DUBLIN CITY UNIVERSITY    TYNDALL NATIONAL INSTITUTE  4
  5. 5. CLARITY  [1/3]CLARITY: Centre for Sensor Web Technologies– CSET (Centre for Science Engineering & Technology) funded  by Science Foundation Ireland (SFI) with industry  contributions– 5 year duration, following on from previous 4‐year  “Adaptive  Information Cluster”– Administrative centre in UCD, researchers in DCU, UCD and  Tyndall Institute, up to 100 researchers– Within DCU involves CDVP (Computing & EE), NCSR (sensor  people), Health & Human Performance (sports people) UNIVERSITY COLLEGE DUBLIN    DUBLIN CITY UNIVERSITY    TYNDALL NATIONAL INSTITUTE  5
  6. 6. CLARITY  [2/3]CLARITY What ? “The Sensor Web” – Increasing availability of cheap, robust, and deployable  sensor technologies ushering in a wave of new information  sources; – Ubiquitous, dynamic, noisy, reactive and yielding  unstructured data‐streams == sensor web – Realizing the sensor web demands a large‐scale, multi‐ disciplinary research effort == CLARITY – Moving beyond our research silos to novel research  interactions; – Demonstrator projects in: TennisSense (and other sports); Environmental monitoring; Karbon footprinting; Ambient Assisted Living; UNIVERSITY COLLEGE DUBLIN    DUBLIN CITY UNIVERSITY    TYNDALL NATIONAL INSTITUTE  6
  7. 7. CLARITY  [3/3]Principal InvestigatorsPrincipal Investigators Prof. Barry Smyth - Personalization, recommender systems, mobile computing Prof. Alan Smeaton - Content-based information retrieval Prof. Dermot Diamond - Materials research, wearable sensors Prof. Noel O’Connor - Audio-visual analysis, multi-modal information processing Mr. Gregory O’Hare - Ubiquitous computing, multi-agent systemsAssociate PIs Prof. Paddy Nixon - Pervasive computing, middleware, security, trust, privacy Prof. Niall Moyna - Sports Science, wearable sensing Dr. Simon Dobson - Middleware, pervasive computing Dr. Cian O’Mathuna - Sensor devices, energy-aware hardware Dr. Brian Caulfield - Physiotherapy, therapeutic gaming, wearable sensorsFunded CollaboratorsChris Bleakley (UCD), Conor Brennan (DCU), Rem Collier (UCD), Brian Corcoran (DCU),Cathal Gurrin (DCU), Neil Hurley (UCD), Lorraine McGinty (UCD), Kieran Moran (DCU),Kieran Nolan (DCU), Brendan O’Flynn (TNI), Donal O’Gorman (DCU), Brett Paull (DCU),Emanuel Popovici (TNI), Aaron Quigley (UCD), Mark Roantree (DCU) UNIVERSITY COLLEGE DUBLIN    DUBLIN CITY UNIVERSITY    TYNDALL NATIONAL INSTITUTE  7
  8. 8. LifeloggingLifelogging is about digitally recording your daily lifeSometimes its for a reason Work e.g. security personnel, medical staff, etc.Personal e.g. diaries, etc.Sometimes its for posterityRecording vacations, family gatherings, social occasionsSometimes its because we canAnd we’re not yet sure what we’ll do with it e.g. MyLifeBits UNIVERSITY COLLEGE DUBLIN    DUBLIN CITY UNIVERSITY    TYNDALL NATIONAL INSTITUTE  8
  9. 9. Visual Lifelogging DevicesMuch past research focus on miniaturising hardware and increasing battery‐life + storage e.g. visual lifelogging domain Steve Mann. Wearable computing: a first step toward personal imaging. Computer, 30:25–32, Feb 1997. TIMELINE Tano et. al. University of Electro-Communications, Tokyo, Japan Microsoft Research SenseCam UNIVERSITY COLLEGE DUBLIN    DUBLIN CITY UNIVERSITY    TYNDALL NATIONAL INSTITUTE  9
  10. 10. SenseCamSenseCam is a Microsoft Research PrototypeMulti‐sensor deviceColour camera3 accelerometersLight meterPassive infrared sensor1GB flash memory storageSmart image capture ~3 images/minSince April 2006 we’ve had two SenseCams … in 2007 we received 5 more UNIVERSITY COLLEGE DUBLIN    DUBLIN CITY UNIVERSITY    TYNDALL NATIONAL INSTITUTE  10
  11. 11. SenseCam images UNIVERSITY COLLEGE DUBLIN    DUBLIN CITY UNIVERSITY    TYNDALL NATIONAL INSTITUTE  11
  12. 12. Addenbrooke’s: SenseCam Work Preliminary Study carried out by Cambridge Memory Clinic,  Addenbrooke’s Hospital 63 year old, well‐educated married woman, with limbic  encephalitis (usually has no memory a few days after an  event) Attends events along with her partner UNIVERSITY COLLEGE DUBLIN    DUBLIN CITY UNIVERSITY    TYNDALL NATIONAL INSTITUTE  12
  13. 13. Addenbrooke’s: SenseCam Work Microsoft Research Cambridge presentation: http://research.microsoft.com/~shodges/presentations/UBICOMP_senseCam.pdf UNIVERSITY COLLEGE DUBLIN    DUBLIN CITY UNIVERSITY    TYNDALL NATIONAL INSTITUTE  13
  14. 14. Addenbrooke’s: SenseCam Work Microsoft Research Cambridge presentation: http://research.microsoft.com/~shodges/presentations/UBICOMP_senseCam.pdf UNIVERSITY COLLEGE DUBLIN    DUBLIN CITY UNIVERSITY    TYNDALL NATIONAL INSTITUTE  14
  15. 15. Addenbrooke’s: SenseCam Work Microsoft Research Cambridge presentation: http://research.microsoft.com/~shodges/presentations/UBICOMP_senseCam.pdf UNIVERSITY COLLEGE DUBLIN    DUBLIN CITY UNIVERSITY    TYNDALL NATIONAL INSTITUTE  15
  16. 16. ~4,000,000 SenseCam Images• One user wearing SC for over 3 years – Each with GPS position! – Many other users too• Experiences: – Most people don’t notice camera – Those that do always remember! Millionth Image – Most people don’t mind the camera – Have been spotted/greeted by people who  have heard about the ‘guy with the camera’ UNIVERSITY COLLEGE DUBLIN    DUBLIN CITY UNIVERSITY    TYNDALL NATIONAL INSTITUTE  16
  17. 17. Image Quality Analysis400 346350300 •40% of images250 252 227 are of low quality200150 143 •Many “boring”100 images of50 42 mundane tasks 0 Terrible Poor OK High Photo AlbumOver last 3 years we’ve developed techniques for SenseCam data management, without having user input or direction …… so our work is technologically-driven rather than based on user pull … let’s look at it !! UNIVERSITY COLLEGE DUBLIN    DUBLIN CITY UNIVERSITY    TYNDALL NATIONAL INSTITUTE  17
  18. 18. Overview• OUR SENSECAM DATA COLLECTION• BROWSING & SEARCHING SENSECAM DATA – Event Segmentation/Searching/Interest/Augmentation – Browsing Application• SENSECAM SUMMARISATION: THE NEXT  GENERATION• THE FUTURE UNIVERSITY COLLEGE DUBLIN    DUBLIN CITY UNIVERSITY    TYNDALL NATIONAL INSTITUTE  18
  19. 19. Our Take…To effectively provide memory retrieval cues using SENSECAM we need to automatically:• Group similar images into distinct “events”• Suggest more “interesting/distinctive” events• “Associate” related events• Provide potentially additional retrieval cues from other sources UNIVERSITY COLLEGE DUBLIN    DUBLIN CITY UNIVERSITY    TYNDALL NATIONAL INSTITUTE  19
  20. 20. Daily Browser Overview SenseCam Images of a day (about 3,000) Event Segmentation EVENT SEGMENTATION Using MOTION sensors – very quick & accurate UNIVERSITY COLLEGE DUBLIN    DUBLIN CITY UNIVERSITY    TYNDALL NATIONAL INSTITUTE  20
  21. 21. Visual Search Facilities SenseCam Images of a day (about 3,000) Event Segmentation Event-Event Comparison within the Multi-day Event ? database Better:Compare Eventslow Cross compare -Too Averages Best: Compare Event AveragesDay -1 (from middle n images)Day -2Day -3Day -4Day -5 … …Day -6 Event database containing last 7 days’ Events UNIVERSITY COLLEGE DUBLIN    DUBLIN CITY UNIVERSITY    TYNDALL NATIONAL INSTITUTE  21
  22. 22. Selecting Event “Keyframe” SenseCam Images of a day (about 3,000) Event Segmentation Event-Event Comparison within the Multi-day Event database Best QUALITY image around MIDDLE of eventDay -1Day -2Day -3Day -4 LandmarkDay -5 Image SelectionDay -6 Event database containing last 7 days’ Events UNIVERSITY COLLEGE DUBLIN    DUBLIN CITY UNIVERSITY    TYNDALL NATIONAL INSTITUTE  22
  23. 23. Suggest Interesting Events Mon Similar Events - Aiden waiting for bus SenseCam Images of a day (about 3,000) Similar Events - Aiden at the office corridor Tue Similar Events - Aiden working on the desk Wed Event Segmentation Unique Events 2 Sept 06 Interactive Thr Browser Fri Event-Event Comparison Sat within the Multi-day Event database Sun MonDay -1Day -2 CALCULATE INTERESTINGNESS VISUAL NOVELTY OF EVENTSDay -3Day -4 + FACE DETECTION LandmarkDay -5 Image SelectionDay -6 Event database containing last 7 days’ Events UNIVERSITY COLLEGE DUBLIN    DUBLIN CITY UNIVERSITY    TYNDALL NATIONAL INSTITUTE  23
  24. 24. Event augmentation Here’s a SenseCam picture of Aiden at a pier in Santa Barbara, CA. If he has GPS he can search for other pictures in the same location… UNIVERSITY COLLEGE DUBLIN    DUBLIN CITY UNIVERSITY    TYNDALL NATIONAL INSTITUTE  24
  25. 25. Event augmentation – more cues• He receives the following “geotagged” images…• Then after some processing on text associated with these images we get many  more images, and even YouTube videos at times too! UNIVERSITY COLLEGE DUBLIN    DUBLIN CITY UNIVERSITY    TYNDALL NATIONAL INSTITUTE  25
  26. 26. Event AugmentationDoes it work?Yes – we have it operational from 6 image sources, tested and evaluated with users.Bringing the threads together … event segmentation, keyframe selection, event importance, event searching, and event augmentation …… we have a system to manage a lifelog UNIVERSITY COLLEGE DUBLIN    DUBLIN CITY UNIVERSITY    TYNDALL NATIONAL INSTITUTE  26
  27. 27. UNIVERSITY COLLEGE DUBLIN    DUBLIN CITY UNIVERSITY    TYNDALL NATIONAL INSTITUTE  27
  28. 28. Released Software Features: • Database – image  management • Quick “event  segmentation” (1‐2  seconds per folder) UNIVERSITY COLLEGE DUBLIN    DUBLIN CITY UNIVERSITY    TYNDALL NATIONAL INSTITUTE  28
  29. 29. Event Segmentation S/W • Carnegie Mellon University •CWI, Amsterdam •Lulua University of Technology •Olivier Zangwell Centre •“Mrs. W.” •University of Leeds •University of Limerick •University of Toronto •University of Utrecht •University of Illinois •University of Tampere UNIVERSITY COLLEGE DUBLIN    DUBLIN CITY UNIVERSITY    TYNDALL NATIONAL INSTITUTE  29
  30. 30. Overview• OUR SENSECAM DATA COLLECTION• BROWSING & SEARCHING SENSECAM DATA• SENSECAM SUMMARISATION: THE NEXT  GENERATION – Activity Recognition THIS IS WHERE – Diet Monitoring THE REAL FUN – Scene Detection STARTS ! – Trajectory Estimation – Incorporating Contextual Information – Keyword Searching• THE FUTURE UNIVERSITY COLLEGE DUBLIN    DUBLIN CITY UNIVERSITY    TYNDALL NATIONAL INSTITUTE  30
  31. 31. Dublin SenseCam WorkActivityRecognition27 “concepts”Outputs manually judgedon ~95k images (5 users) UNIVERSITY COLLEGE DUBLIN    DUBLIN CITY UNIVERSITY    TYNDALL NATIONAL INSTITUTE  31
  32. 32. Comparison of Lifestyle Within  Social Groups steeringWheel 3.5 eating insideVehicle vehiclesExternal readingstandard deviations away from sample mean 2.5 holdingPhone steeringWheel vehiclesExternal holdingPhone reading 1.5 eating insideVehicle 0.5 -0.5 user 1 user 2 user 3 user 4 user 5 -1.5 UNIVERSITY COLLEGE DUBLIN    DUBLIN CITY UNIVERSITY    TYNDALL NATIONAL INSTITUTE  32
  33. 33. Dietry habitsConsider using even only the “Eating” concept...• Detect events where user is eating• Allows us/family/dietations gain more complete  record of our eating habits UNIVERSITY COLLEGE DUBLIN    DUBLIN CITY UNIVERSITY    TYNDALL NATIONAL INSTITUTE  33
  34. 34. Cathal | HELP | ABOUT T F S S M T W T F S S M T W T F S S M T W T F S S M T W T F S S M T W > CALENDAR WEDNESDAY 17 OCT 2008 My total calorie This day’s food intake is as following: Meal 1 Little Much balance for each day 950 Calories 10:47am overFats Sweets Oil / time… 4 Meals Milk Meat 2,118 Calories Vegetables Fruits Select a meal to annotate and/or see Bread / Rice the type of food eaten for that meal Other Starch Meal 2 Little Much 243 Calories Detected ‘eating’ 1:20pm Sweets Oil / Fats events listed, for the Milk MeatThisuser to more the re-calculates Vegetables Fruitsoverall calorie- indicate accurately Bread / Rice Other Starch what s/he ate duringexercise balance and Little Much 80 Calories Meal 3displaysday the the on 3:15pm Sweets Oil / Fatsscreen Milk Meat Vegetables Fruits Bread / Rice Other Starch Meal 4 Little Much 845 Calories 8:08pm Sweets UNIVERSITY COLLEGE DUBLIN    DUBLIN CITY UNIVERSITY    TYNDALL NATIONAL INSTITUTE  34 Oil / Fats
  35. 35. Advanced Image Matching Each feature point casts a weighted vote for multiple database images SURF feature are extracted Votes are accumulated & the best match is foundBi-directional Match Verification & re-ranking of Top results UNIVERSITY COLLEGE DUBLIN    DUBLIN CITY UNIVERSITY    TYNDALL NATIONAL INSTITUTE  35
  36. 36. Setting Detection – Watching TV UNIVERSITY COLLEGE DUBLIN    DUBLIN CITY UNIVERSITY    TYNDALL NATIONAL INSTITUTE  36
  37. 37. Setting Detection – In the Park UNIVERSITY COLLEGE DUBLIN    DUBLIN CITY UNIVERSITY    TYNDALL NATIONAL INSTITUTE  37
  38. 38. Trajectory Estimation Area Map Location DatabaseA User’s Images Matching UNIVERSITY COLLEGE DUBLIN    DUBLIN CITY UNIVERSITY    TYNDALL NATIONAL INSTITUTE  38
  39. 39. Trajectory Estimation Results Ground truthAccuracy ~1.5 meters UNIVERSITY COLLEGE DUBLIN    DUBLIN CITY UNIVERSITY    TYNDALL NATIONAL INSTITUTE  39
  40. 40. Other Data SourcesSENSECAM: Images PHYSIOLOGICAL: Heart Rate, body temperature, breathing rate, sweat ph analysis GPS: Location BLUETOOTH: PHYSIOLOGICAL: People around me + Posture monitoring FAMILIARITY PC: E-mail, web pages visited, documents worked on UNIVERSITY COLLEGE DUBLIN    DUBLIN CITY UNIVERSITY    TYNDALL NATIONAL INSTITUTE  40
  41. 41. Using Context in Personal Information Management• Represent events as text documents, then “Google” them • Search using keywords to find the desired target  (e.g. pics of SenseCam event): – You may recall: • This document was for the Conference X. • I worked on it before meeting with Professor A. • It was a hot day • I was really tired • It was some restaurant in the city centre where we met UNIVERSITY COLLEGE DUBLIN    DUBLIN CITY UNIVERSITY    TYNDALL NATIONAL INSTITUTE  41
  42. 42. iCLIPS Browsing InterfacePresent landmarks: real life events (Photos) and computer activities (Keywords and  Thumbnails)Refine searching by RECOGNIZING landmarks and Estimating the relevant Temporal distance  from the Targets to the landmarks Traditional Searching Panel also provide rich searching options: • Keywords •Target type • Flexible time/date •Geo-location •People •And more… UNIVERSITY COLLEGE DUBLIN    DUBLIN CITY UNIVERSITY    TYNDALL NATIONAL INSTITUTE  42
  43. 43. Overview• OUR SENSECAM DATA COLLECTION• BROWSING & SEARCHING SENSECAM DATA• SENSECAM SUMMARISATION: THE NEXT  GENERATION• THE FUTURE – Storytelling – Energy Consumption – Designing for the Elderly – Summary UNIVERSITY COLLEGE DUBLIN    DUBLIN CITY UNIVERSITY    TYNDALL NATIONAL INSTITUTE  43
  44. 44. Lifelogs & StorytellingLifelogs offer huge opportunity for telling life stories.The Need for Narrative:1. Humans like stories ‐ we tell them everyday2. Lifelogs are complex & voluminous ‐ we can’t just  present the material ‐ we need to tame it somehow3. Storyform communicates experience effectively &  enables reflection and introspection UNIVERSITY COLLEGE DUBLIN    DUBLIN CITY UNIVERSITY    TYNDALL NATIONAL INSTITUTE  44
  45. 45. Lifelogs & StorytellingClear Challenges:1. What components of a lifelog should be used in the composition of digital life stories and how should they be structured to enable retelling?2. What information should be captured about the relationships between the various story elements in order to facilitate the reasoning required to build the end narrative?3. How should an author be supported in the process of composing a life story and how should these stories be presented to their intended audience? UNIVERSITY COLLEGE DUBLIN    DUBLIN CITY UNIVERSITY    TYNDALL NATIONAL INSTITUTE  45
  46. 46. Classifying SC Motion Data• Use accelerometer data to identify various states: sitting, walking, running, driving, on bus, on airplane• Can estimate energy output• Can estimate our “carbon footprint”• This work is currently “early‐stage” UNIVERSITY COLLEGE DUBLIN    DUBLIN CITY UNIVERSITY    TYNDALL NATIONAL INSTITUTE  46
  47. 47. Designing for Older AdultsAreas Affected by Ageing Implications for DesignCognitive Skills - Providing feedback to showWorking Memory what has been selected. Use combination of text and icons to support recall.Sensory Skills - Use of large images and text,Vision and Hearing large target areas for buttons and high colour contrast. Use of low frequency auditory signals.Psychomotor skills Use direct input devices (touch screen). Reduce scrolling. UNIVERSITY COLLEGE DUBLIN    DUBLIN CITY UNIVERSITY    TYNDALL NATIONAL INSTITUTE  47
  48. 48. Designing for Older Adults UNIVERSITY COLLEGE DUBLIN    DUBLIN CITY UNIVERSITY    TYNDALL NATIONAL INSTITUTE  48
  49. 49. Designing for Older Adults UNIVERSITY COLLEGE DUBLIN    DUBLIN CITY UNIVERSITY    TYNDALL NATIONAL INSTITUTE  49
  50. 50. CLARITY + U. LeedsMartin Conway & Chris Moulin, Institute of  Psychological Sciences, University Leeds1 healthy subject ‐> 3 years of SenseCam imagesHow does SenseCam effect “normal” people? UNIVERSITY COLLEGE DUBLIN    DUBLIN CITY UNIVERSITY    TYNDALL NATIONAL INSTITUTE  50
  51. 51. Recollective experience as cued by SenseCam stills50 “keyframe” images reviewed – 2 months between viewingsConsistency of judgements (R / F / DK) … Same judgement on 78% of memories 60 first 50 second 40 30 20 10 0 DK % F % R % UNIVERSITY COLLEGE DUBLIN    DUBLIN CITY UNIVERSITY    TYNDALL NATIONAL INSTITUTE  51
  52. 52. Episodic details 1when / where / who / what UNIVERSITY COLLEGE DUBLIN    DUBLIN CITY UNIVERSITY    TYNDALL NATIONAL INSTITUTE  52
  53. 53. Episodic details 2 UNIVERSITY COLLEGE DUBLIN    DUBLIN CITY UNIVERSITY    TYNDALL NATIONAL INSTITUTE  53
  54. 54. Next – Influence F‐R shift R F DK R 19 3 0 F 37 3 18 0 DK 3 +18 0 2 2- Working with SenseCam films instead of stills ( using thesame events)- Working with surrounding events to provide furthercontextual information (to allow mind to storyboard)- Change in recollective experience? F to R shift? UNIVERSITY COLLEGE DUBLIN    DUBLIN CITY UNIVERSITY    TYNDALL NATIONAL INSTITUTE  54
  55. 55. M 0.00 0.02 0.04 0.06 0.08 0.10 0.12 0.14 0.16 ay -0 Ju 6 n- 06 Ju l- 0 Au 6 g- 0 Se 6 p- 0 O 6 ct -0 No 6 v- 06 D ec -0 Ja 6 n- 0 Fe 7 b- 07 M ar -0 Ap 7 r- 07 M ay -0 Ju 7 n-UNIVERSITY COLLEGE DUBLIN    07 Ju l- 0 Au 7 g- human memory 0 Se 7 p- 0 O 7 ct -0 No 7 v- 07 D ec -0DUBLIN CITY UNIVERSITY    Ja 7 n- 0 Fe 8 b- 08 M ar -0 Ap 8 r- 08 M ay -0 Ju 8 n- 08 Ju l- 0TYNDALL NATIONAL INSTITUTE  Au 8 g- 0 Se 8 The SenseCam browser and  p- 0 O 8 ct -0 No 8 CG v- 08 D ec -0 Browser 855
  56. 56. Results so farThere is no considerable overlap between the most important browser  events and those recalled by CGr = .17Spearman’s rank correlation = .008 p = .967novelty and personal relevance ratings given by CG on memories recalled by  him and those generated by the browser Novelty: 5.0 and 3.22  P.relevance: 4.0 and 3.0 t‐test highly significant UNIVERSITY COLLEGE DUBLIN    DUBLIN CITY UNIVERSITY    TYNDALL NATIONAL INSTITUTE  56
  57. 57. Next?Using user generated important events to help train  system to improve…How does user recollection of “free recall” memory  change after being presented with SenseCam  images of this memory… UNIVERSITY COLLEGE DUBLIN    DUBLIN CITY UNIVERSITY    TYNDALL NATIONAL INSTITUTE  57
  58. 58. Summary• More SenseCams – we’d love 50+ of them!• Increased accuracy/flexibility in recognising a  person’s lifestyle – More SenseCams = better recognition of lifestyle  “norms”• Increased collaboration with memory experts e.g.  as with Leeds – we’re good at summarising SenseCam data, but  not at neuro psychology! UNIVERSITY COLLEGE DUBLIN    DUBLIN CITY UNIVERSITY    TYNDALL NATIONAL INSTITUTE  58
  59. 59. Managing a Life of Lifelogged SenseCam Images Aiden R. Doherty further information: http://www.cdvp.dcu.ie/SenseCam http://www.computing.dcu.ie/~adoherty (case sensitive) UNIVERSITY COLLEGE DUBLIN    DUBLIN CITY UNIVERSITY    TYNDALL NATIONAL INSTITUTE  59

×