• Save
Episode identification algorithm
Upcoming SlideShare
Loading in...5
×
 

Episode identification algorithm

on

  • 232 views

Presentation on segmenting wearable camera data into events/episodes ...

Presentation on segmenting wearable camera data into events/episodes

Presented in: WIAMIS 2008 - 9th International Workshop on Image Analysis for Multimedia Interactive Services, 7-9 May 2008, Klagenfurt, Austria

Abstract
A personal lifelog of visual information can be very helpful as a human memory aid. The SenseCam, a passively capturing wearable camera, captures an average of 1,785 images per day, which equates to over 600,000 images per year. So as not to overwhelm users it is necessary to deconstruct this substantial collection of images into digestable chunks of information, i.e. into distinct events or activities. This paper improves on previous work on automatic segmentation of SenseCam images into events by up to 29.2\%, primarily through the introduction of intelligent threshold selection techniques, but also through improvements in the selection of normalisation, fusion, and vector distance techniques. Here we use the most extensive dataset ever used in this domain, 271,163 images collected by 5 users over a time period of one month with manually groundtruthed events

Statistics

Views

Total Views
232
Views on SlideShare
231
Embed Views
1

Actions

Likes
0
Downloads
0
Comments
0

1 Embed 1

http://www.linkedin.com 1

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    Episode identification algorithm Episode identification algorithm Presentation Transcript

    • DUBLIN CITY UNIVERSITY ADAPTIVE INFORMATION CLUSTER CENTRE FOR DIGITAL VIDEO PROCESSING Automatically Segmenting LifeLog Data into Events Aiden R. Doherty and Alan F. Smeaton Centre for Digital Video Processiong (CDVP) & Adaptive Information Cluster (AIC), Dublin City University (DCU), Ireland -1-
    • DUBLIN CITY UNIVERSITY ADAPTIVE INFORMATION CLUSTER CENTRE FOR DIGITAL VIDEO PROCESSING Overview • INTRODUCTION – Introduction to lifelogging & the challenges – Event segmentation and previous work • EVENT SEGMENTATION APPROACHES – Overview – TextTiling & peak scoring – Thresholding – Post-processing boundary gap • EXPERIMENTAL SETUP • RESULTS • CONCLUSIONS – Future Segmentation Work – Importance of segmentation to other lifelogging processing stages -2-
    • DUBLIN CITY UNIVERSITY ADAPTIVE INFORMATION CLUSTER CENTRE FOR DIGITAL VIDEO PROCESSING Lifelogging • Lifelogging is about recording daily life, digitally • Sometimes its for a reason, – work … e.g. security personnel, medical staff, – personal … e.g. diaries, etc. • Sometimes its for posterity, recording vacations, family gatherings, social occasions • Sometimes its because we can, and we’re not yet sure what we’ll do with lifelogs, e.g. MyLifeBits -3-
    • DUBLIN CITY UNIVERSITY ADAPTIVE INFORMATION CLUSTER CENTRE FOR DIGITAL VIDEO PROCESSING SenseCam • SenseCam is a Microsoft Research Prototype • Multi-sensor device – Colour camera – 3 accelerometers – Light meter – Passive infrared sensor • 1GB flash memory storage • Smart image capture ~3 images/min • Since April 2006 we’ve had two SenseCams … recently have received 5 more -4-
    • DUBLIN CITY UNIVERSITY ADAPTIVE INFORMATION CLUSTER CENTRE FOR DIGITAL VIDEO PROCESSING How to Review Images? • Make a 2 minute movie of your day! -5-
    • DUBLIN CITY UNIVERSITY ADAPTIVE INFORMATION CLUSTER CENTRE FOR DIGITAL VIDEO PROCESSING Lifelog Processing Overview SenseCam Images of a day (about 3,000) Event Segmentation 2 Sept 06 Interactive Browser Event-Event Comparison within the Multi-day Event databaseDay -1Day -2 Composition of the BrowserDay -3Day -4 LandmarkDay -5 Image SelectionDay -6 Novelty Calculation of 0.1 0.7 0.1 0.1 0.3 0.4 0.8 0.1 0.9 Each Event Event database containing last 7 days’ Events -6-
    • DUBLIN CITY UNIVERSITY ADAPTIVE INFORMATION CLUSTER CENTRE FOR DIGITAL VIDEO PROCESSING Summary of Other Approaches • Wang et. al. of Princeton temporally segment lifelog data into events of 5 minutes long • Yeung & Yeo used a form of time- constrained clustering in the video domain … we test against an adaptation of this approach -7-
    • DUBLIN CITY UNIVERSITY ADAPTIVE INFORMATION CLUSTER CENTRE FOR DIGITAL VIDEO PROCESSING Contributions of This Paper • Previous segmentation work published at RIAO conference in Pittsburgh in 2007 – 1 user collecting 22,173 images – No recall basis defined • This work – 5 users collecting 271,163 images – Recall basis/groundtruth of 2,986 events – 29.2% better than previous approaches -8-
    • DUBLIN CITY UNIVERSITY ADAPTIVE INFORMATION CLUSTER CENTRE FOR DIGITAL VIDEO PROCESSING Overview • INTRODUCTION – Introduction to lifelogging & the challenges – Event segmentation and previous work • EVENT SEGMENTATION APPROACHES – Overview – TextTiling & peak scoring – Thresholding – Post-processing boundary gap • EXPERIMENTAL SETUP • RESULTS • CONCLUSIONS – Future Segmentation Work – Importance of segmentation to other lifelogging processing stages -9-
    • DUBLIN CITY UNIVERSITY ADAPTIVE INFORMATION CLUSTER CENTRE FOR DIGITAL VIDEO PROCESSING Event Segmentation Breakfast Work Car Talking to colleague Airplane Talking to friend - 10 -
    • DUBLIN CITY UNIVERSITY ADAPTIVE INFORMATION CLUSTER CENTRE FOR DIGITAL VIDEO PROCESSING Event Segmentation One Day’s Images1. Raw data For each image... For each sensor reading... • Scalable Colour • Accelerometer X/Y/Z Extract MPEG-7 • Colour Structure Sensor values... • Light descriptors... • Colour Layout • Temperature • Edge Histogram • Passive Infra Red Shot Boundary Detection OR TextTiling ... adjacent images/sensor vals ... adjacent blocks of 10 images/sensor vals2. Similarity matching ...... ...... 80 65 70 15 120 149 289 …3. Normalisation & Data fusion4. Thresholding5. Events Event-segmented images of a day - 11 -
    • DUBLIN CITY UNIVERSITY ADAPTIVE INFORMATION CLUSTER CENTRE FOR DIGITAL VIDEO PROCESSINGTextTiling vs Shot Bound Detection One Day’s Images1. Raw data For each image... For each sensor reading... • Scalable Colour TT WINDOW SIZE • Accelerometer X/Y/Z Extract MPEG-7 • Colour Structure • Light descriptors... • Colour Layout Sensor values... TO USE? • Temperature • Edge Histogram • Passive Infra Red Shot Boundary Detection OR TextTiling ... adjacent images/sensor vals ... adjacent blocks of 10 images/sensor vals2. Similarity matching ...... ...... 80 65 70 15 120 149 289 …3. Normalisation & Data fusion4. Thresholding5. Events Event-segmented images of a day - 12 -
    • DUBLIN CITY UNIVERSITY ADAPTIVE INFORMATION CLUSTER CENTRE FOR DIGITAL VIDEO PROCESSING Peak Scoring • Higher dissimilarity scores = greater likelihood of event boundary • Therefore wish to emphasise dissimilarity scores • We use a method that we refer to as “peak scoring” - 13 -
    • DUBLIN CITY UNIVERSITY ADAPTIVE INFORMATION CLUSTER CENTRE FOR DIGITAL VIDEO PROCESSING Peak Scoring Method Lowest value to left and right BEFORE while values are successively smaller AFTER - 14 -
    • DUBLIN CITY UNIVERSITY ADAPTIVE INFORMATION CLUSTER CENTRE FOR DIGITAL VIDEO PROCESSING Thresholding One Day’s Images1. Raw data For each image... For each sensor reading... • Scalable Colour • Accelerometer X/Y/Z Extract MPEG-7 • Colour Structure Sensor values... • Light Mean Thresholding descriptors... • Colour Layout • Edge Histogram • Temperature • Passive Infra Red Shot Boundary Detection OR TextTiling • Parametric ... adjacent images/sensor vals Kapur Thresholding ... adjacent blocks of 10 images/sensor vals2. • k * mean Similarity of values • Non-parametric matching ...... 80 65 70 15 ...... 120 149 • Entropy based 289 …3. Normalisation & Data fusion4. Thresholding5. Events Event-segmented images of a day - 15 -
    • DUBLIN CITY UNIVERSITY ADAPTIVE INFORMATION CLUSTER CENTRE FOR DIGITAL VIDEO PROCESSINGPost-Processing Boundary Gap • Sometimes there may be hectic activities with noisy data where many event boundaries are proposed • Just take the first and ignore the rest WINDOW SIZE TO USE? - 16 -
    • DUBLIN CITY UNIVERSITY ADAPTIVE INFORMATION CLUSTER CENTRE FOR DIGITAL VIDEO PROCESSING Overview • INTRODUCTION – Introduction to lifelogging & the challenges – Event segmentation and previous work • EVENT SEGMENTATION APPROACHES – Overview – TextTiling & peak scoring – Thresholding – Post-processing boundary gap • EXPERIMENTAL SETUP • RESULTS • CONCLUSIONS – Future Segmentation Work – Importance of segmentation to other lifelogging processing stages - 17 -
    • DUBLIN CITY UNIVERSITY ADAPTIVE INFORMATION CLUSTER CENTRE FOR DIGITAL VIDEO PROCESSING Experimental Setup • 5 users collected SenseCam data for 1 month each • 271,163 images collected • Groundtruth -> Users reviewed their images and marked event boundaries • 2,986 boundaries identified - 18 -
    • DUBLIN CITY UNIVERSITY ADAPTIVE INFORMATION CLUSTER CENTRE FOR DIGITAL VIDEO PROCESSING User marks change in activity - 19 -
    • DUBLIN CITY UNIVERSITY ADAPTIVE INFORMATION CLUSTER CENTRE FOR DIGITAL VIDEO PROCESSING Experimental Setup • Data split up into training set and test set • MPEG-7 processing time of 30 minutes for a busy day of 2,500 images • Some interesting user statistics – SenseCam worn an average of 10h 03m per day – 1,786 images per day – 19 events per day – 95 images per event - 20 -
    • DUBLIN CITY UNIVERSITY ADAPTIVE INFORMATION CLUSTER CENTRE FOR DIGITAL VIDEO PROCESSING Overview • INTRODUCTION – Introduction to lifelogging & the challenges – Event segmentation and previous work • EVENT SEGMENTATION APPROACHES – Overview – TextTiling & peak scoring – Thresholding – Post-processing boundary gap • EXPERIMENTAL SETUP • RESULTS • CONCLUSIONS – Future Segmentation Work – Importance of segmentation to other lifelogging processing stages - 21 -
    • DUBLIN CITY UNIVERSITY ADAPTIVE INFORMATION CLUSTER CENTRE FOR DIGITAL VIDEO PROCESSING Metrics Used• Hit = If system boundary is within 5 minutes of user defined boundary• Precision judges % of correct system boundaries• Recall judges % of user boundaries identified• F1 measure is a balance between precision and recall - 22 -
    • DUBLIN CITY UNIVERSITY ADAPTIVE INFORMATION CLUSTER CENTRE FOR DIGITAL VIDEO PROCESSINGOptimal Fusion Boundary Detection1 TextTiling vs Shot Technique Precision precision Recall recall F-Measure f_score • TextTiling better for sources with slowly changing values0.9 - Results3)Overview • 1) trained using <8>, Temperature <8> Best MPEG7 <5>, 2) PIR2 sources0.8 One Day’s Images0.7 Optimal Normalisation= 0.65) - MPEG7 (weighting Technique 1. Raw data Optimal MPEG7 Distance Metric0.6 - Accelerometer (0.35) • SBD better for sources with highly variable readings For each image... For each sensor reading...0.5 •1. Accelerometer, 2) Light 1) Sum Colour • Scalable (0.62) • Accelerometer X/Y/Z Extract MPEG-7 • Colour Structure Sensor values... • Light 1. CombMin (0.62) 2. Min-MaxShot Boundary Detection OR TextTiling (0.62) 1. Histogram Intersection descriptors... (0.58) • Colour Layout • Edge Histogram • Temperature0.4 Post-Processing Boundary Gap 2. CombSUM (0.58) 3. Max2. Euclidean (0.62) (0.58) ... adjacent images/sensor vals ... adjacent blocks of 10 images/sensor vals • Passive Infra Red0.3Optimal Thresholding Technique 2. 3. Similarity Mean-Shift – problem with negative values CombANZ (0.54) 4. 3. Manhattan (0.61)0.2 4. matching CombMAX (0.53) 4. Optimal Chord (0.60) with - Squared gap to deal ...... ......0.1 1. Mean with Jeffrey Mod KL (0.60) -> Best balance 5. k=3.4 (0.62) found to be 3 80 65 70 15 120 149 289 … noisy data 3. 0 2. Data fusion (0.58) Kapur 6. Scoring vs No(0.60)Scoring Normalisation & Peak Bray Curtis Peak -> High recall minutes 0 3 6 0.3 0.6 0.9 1.2 1.5 1.8 2.1 2.4 2.7 3.3 3.6 3.9 4.2 4.5 4.8 5.1 5.4 5.7 6.3 6.6 6.9 7.2 7.5 7.8 3. Select top3 Square Chi5Squared (0.59) 8precision 1 2 7. 20 (RIAO) (0.55) 6 4 -> High 7 9 10 • Peak Scoring is much better 8. X2 Statistics (0.59) • Better on 60 out of 63 days 4. Thresholding 9. Kullback Leiber (0.58) • Avg. F1-Measure of 0.62 vs 0.53 10. Canberra (0.56) 5. Events Event-segmented images of a day - 23 -
    • DUBLIN CITY UNIVERSITY ADAPTIVE INFORMATION CLUSTER CENTRE FOR DIGITAL VIDEO PROCESSING MPEG7 vs Sensor Only • Segmenting on sensors only is almost as good and much quicker 1 MPEG-7 + Sensors Sensors Only0.90.80.70.60.50.40.30.20.1 0 - 24 -
    • DUBLIN CITY UNIVERSITY ADAPTIVE INFORMATION CLUSTER CENTRE FOR DIGITAL VIDEO PROCESSING Comparison to Other Methods• 29.2% better than Yeung & Yeo video adaptation• 42.3% better than previous best in lifelogging (RIAO) 1 Sensors Only Yeung & Yeo Adaptation RIAO Princeton0.90.80.70.60.50.40.30.20.1 0 - 25 -
    • DUBLIN CITY UNIVERSITY ADAPTIVE INFORMATION CLUSTER CENTRE FOR DIGITAL VIDEO PROCESSING Overview • INTRODUCTION – Introduction to lifelogging & the challenges – Event segmentation and previous work • EVENT SEGMENTATION APPROACHES – Overview – TextTiling & peak scoring – Thresholding – Post-processing boundary gap • EXPERIMENTAL SETUP • RESULTS • CONCLUSIONS – Future Segmentation Work – Importance of segmentation to other lifelogging processing stages - 26 -
    • DUBLIN CITY UNIVERSITY ADAPTIVE INFORMATION CLUSTER CENTRE FOR DIGITAL VIDEO PROCESSING Conclusions • Introduced largest visual lifelogging dataset reported in literature • Segmentation on sensor values is instant • Using only sensor values is almost as good as using MPEG-7 + sensor values • Our approach works 42.3% better than any other used in lifelogging domain, and 29.2% better than any adaptation used - 27 -
    • DUBLIN CITY UNIVERSITY ADAPTIVE INFORMATION CLUSTER CENTRE FOR DIGITAL VIDEO PROCESSING Future Work SenseCam Images of a day (~ 2,000) Event Segmentation 2 Sept 06 Interactive Browser Retrieving Lifelog Events Event AugmentationDay -1Day -2Day -3 Day -4Day -5 Select Keyframe ImageDay -6 Determine 0.1 0.7 0.1 0.1 0.3 0.4 0.8 0.1 0.9 Last 7 Event Uniqueness days’ Events - 28 -
    • DUBLIN CITY UNIVERSITY ADAPTIVE INFORMATION CLUSTER CENTRE FOR DIGITAL VIDEO PROCESSING Danke further information: http://www.computing.dcu.ie/~adoherty http://www.cdvp.dcu.ie/SenseCam - 29 -