Smart Photo Selection: Interpret Gaze as Personal Interest

497 views

Published on

Manually selecting subsets of photos from large collections in order to present them to friends or colleagues or to print them as photo books can be a tedious task. Today, fully automatic approaches are at hand for supporting users. They make use of pixel information extracted from the images, analyze contextual information such as capture time and focal aperture, or use both to determine a proper subset of photos. However, these approaches miss the most important factor in the photo selection process: the user. The goal of our approach is to consider individual interests. By recording and analyzing gaze information from the user's viewing photo collections, we obtain information on user's interests and use this information in the creation of personal photo selections. In a controlled experiment with 33 participants, we show that the selections can be significantly improved over a baseline approach by up to 22% when taking individual viewing behavior into account. We also obtained significantly better results for photos taken at an event participants were involved in compared with photos from another event.

Published in: Science
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
497
On SlideShare
0
From Embeds
0
Number of Embeds
35
Actions
Shares
0
Downloads
3
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Smart Photo Selection: Interpret Gaze as Personal Interest

  1. 1. 1 Smart Photo Selection: Interpret Gaze as Personal Interest Tina Walber1 , Ansgar Scherp2,3 , Steffen Staab1 1 Institute WeST, University of Koblenz, Germany 2 Kiel University, Germany 3 Leibniz Information Center for Economics, Kiel, Germany
  2. 2. Walber, Scherp, Staab ● Smart Photo Selection: Interpret Gaze as Personal Interest 2 Managment of Digital Photos ● Its a mess! ● We take a lot of photos ● Manually selecting photos is cumbersome ● Like to have photo selections for – Sharing photos online – Creating photo products like photo books – Creating presentations
  3. 3. Walber, Scherp, Staab ● Smart Photo Selection: Interpret Gaze as Personal Interest 3 State of the Art: Automatic Creation of Photo Selections ● Content-based approaches – Analysis of low-level features ● Context-based approaches – Analysis of context information ● What about individual aestetics, personal preferences, user interests, ….?
  4. 4. Walber, Scherp, Staab ● Smart Photo Selection: Interpret Gaze as Personal Interest 4 Interpret Gaze as Personal Interest ● Gaze delivers information on user's interest ● Useful for creating individual photo selections? ● Principal approach – Merely observe what users are doing anyway – Do not ask to perfom additional tasks
  5. 5. Walber, Scherp, Staab ● Smart Photo Selection: Interpret Gaze as Personal Interest 5 ● Starting from $99
  6. 6. Walber, Scherp, Staab ● Smart Photo Selection: Interpret Gaze as Personal Interest 6 Research Questions 1. Is there a need for individual photo selections? 2. Does a gaze-based selection outperform selections based on content and context analysis when comparing to those created manually? 3. Does the personal interest in a viewed photo set have an impact on the obtained selection results?
  7. 7. Walber, Scherp, Staab ● Smart Photo Selection: Interpret Gaze as Personal Interest 7 Experiment Setup Photo Viewing Task: „get an overview“ Step 1 Recording of the eye tracking data
  8. 8. Walber, Scherp, Staab ● Smart Photo Selection: Interpret Gaze as Personal Interest 8 Photo Viewing 32 pages with 9 photos each
  9. 9. Walber, Scherp, Staab ● Smart Photo Selection: Interpret Gaze as Personal Interest 9 Experiment Setup Photo Viewing Task: „get an overview“ Step 1 Photo Selection Task: „select photos for your private photo collection“ Step 2 Recording of the eye tracking data Creation of Ground Truth Sm
  10. 10. Walber, Scherp, Staab ● Smart Photo Selection: Interpret Gaze as Personal Interest 10 Manual Selection Creator:LibreOffice 3.5 LanguageLevel:2
  11. 11. Walber, Scherp, Staab ● Smart Photo Selection: Interpret Gaze as Personal Interest 11 Collection CA Collection CB 162 photos 126 photos Experiment Data Set C = CA CB ∩ Two Data Sets and Two User Groups Institute A Institute B Home collectionHome collection Foreign collection ● Taken during social events of the research institutes
  12. 12. Walber, Scherp, Staab ● Smart Photo Selection: Interpret Gaze as Personal Interest 12 Participants ● 33 participants (12 of them female) ● 21 associated to Institute A, 12 to Institute B ● Aged between 25 and 62 (Ø 33.5 ± 9.57) ● 20 graduate students, 4 postdocs, 9 other professions
  13. 13. Walber, Scherp, Staab ● Smart Photo Selection: Interpret Gaze as Personal Interest 13 Overview Analysis and Evaluation Se Collection C Gaze Based Selection Calculation of Precision P Manual Selection Content and Context Based Selection Sb+e Sb Ground Truth Sm
  14. 14. Walber, Scherp, Staab ● Smart Photo Selection: Interpret Gaze as Personal Interest 14 Baseline Measures # Name Description 1 concentration Time Photo was taken with other photos in a short period of time 2 sharpness Sharpness score from related work 3 numberOfFaces Number of faces 4 faceGaussian Size and position of faces 5 personsPopula rity Popularity of the depicted persons 6 faceArea Areas in pixels covered by faces Selection of photos based on: Calculated for each photo
  15. 15. Walber, Scherp, Staab ● Smart Photo Selection: Interpret Gaze as Personal Interest 15 Eye Tracking Data ● Fixations and saccades ● Analysed gaze data with eye tracking measures Creator:LibreOffice 3.5 LanguageLevel:2 ● Viewing duration / page: M = 12.6 s ● Number of fixations / photo: M = 3.25
  16. 16. Walber, Scherp, Staab ● Smart Photo Selection: Interpret Gaze as Personal Interest 16 Eye Tracking Measures # Name Description 7 fixated Was the photo was fixated? Creator:LibreOffice 3.5 LanguageLevel:2
  17. 17. Walber, Scherp, Staab ● Smart Photo Selection: Interpret Gaze as Personal Interest 17 Eye Tracking Measures # Name Description 7 fixated Was the photo was fixated? 8 fixationCount Counts the number of fixations Creator:LibreOffice 3.5 LanguageLevel:2
  18. 18. Walber, Scherp, Staab ● Smart Photo Selection: Interpret Gaze as Personal Interest 18 Eye Tracking Measures # Name Description 7 fixated Was the photo was fixated? 8 fixationCount Counts the number of fixations 9 fixationDuration Sum of duration of all fixations Creator:LibreOffice 3.5 LanguageLevel:2
  19. 19. Walber, Scherp, Staab ● Smart Photo Selection: Interpret Gaze as Personal Interest 19 Eye Tracking Measures # Name Description 7 fixated Was the photo was fixated? 8 fixationCount Counts the number of fixations 9 fixationDuration Sum of duration of all fixations 10 firstFixationDuration Duration of the first fixation 11 lastFixationDuration Duration of the last fixation 12 avgFixationDuration Average fixation duration 13 maxVisitDuration Maximum visit length 14 meanVisitDuration Average visit length 15 visitCount Number of visits 16 saccLength Mean length of the saccades 17 pupilMax Maximum pupil diameter 18 pupilMaxChange Maximum pupil diameter change 19 pupilAvg Average pupil diameter
  20. 20. Walber, Scherp, Staab ● Smart Photo Selection: Interpret Gaze as Personal Interest 20 Combination of Measures ● Using a model learned from logistic regression ● Assigns each image a probability of being selected ● 30 random splits for training and test data
  21. 21. Walber, Scherp, Staab ● Smart Photo Selection: Interpret Gaze as Personal Interest 21 1. Is there a need for individual photo selections? 1 21 41 61 81 101121141161181201221241261281 0 5 10 15 20 25 30 Photo with the highest number of selections Photos with no selections Photos in data set C 10 40 70 100 130 160 190 220 250 280 SelectionFrequency ● Manually created photo selections are diverse
  22. 22. Walber, Scherp, Staab ● Smart Photo Selection: Interpret Gaze as Personal Interest 22 2. Evaluation of the Photo Selections PrecisionP Sb Sb+e Se * * Random Selection P = 0.428P = 0.365 P = 0.426 ● Improvement of 17% over baseline
  23. 23. Walber, Scherp, Staab ● Smart Photo Selection: Interpret Gaze as Personal Interest 23 3. Impact of personal interest? PrecisionP Results for Sb+e Foreign Collection Home Collection P = 0.446P = 0.404 *
  24. 24. Walber, Scherp, Staab ● Smart Photo Selection: Interpret Gaze as Personal Interest 24 Conclusion ● Photo selection behavior is individual ● Gaze helps capture personal preferences ● Results are better for photos with personal interest ● Might work even better for real personal photos ● Potential application in photo book authoring Thank you for your attention!

×