Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

EgoMon Gaze and Video Dataset for Visual Saliency Prediction

1,137 views

Published on

http://imatge-upc.github.io/egocentric-2016-saliency/
This project focuses on the creation of a new type of egocentric (first person) vision dataset. For that purpose, the EgoMon Gaze & Video Dataset is presented. This EgoMon dataset was recorded using the eye gaze tracking technology that studies the movement and position of the eyes. The Tobii glasses (wearable, eye tracker and head-mounted device) were the main tool used to record and extract the gaze data for this dataset. The dataset consists in 7 videos of 34 minutes each one of average, 13428 frames extracted from each video (with a frequency of 1 fps), and 7 files with the gaze data (fixations points of the wearer of the glasses) for each frame and video. The videos were recorded in the city of Dublin (Ireland) both indoor and outdoor. The generated dataset has been used to evaluate the performance of a state of art model for visual saliency prediction on egocentric video.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

EgoMon Gaze and Video Dataset for Visual Saliency Prediction

  1. 1. Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016. EgoMon Gaze and Video Dataset for Visual Saliency Prediction Mònica Chertó Sarret Supervised by: Cathal Gurrin and Xavier Giró
  2. 2. Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016. Outline 1. Introduction 2. State of the art 3. EgoMon Gaze & Video Dataset 4. Visual Saliency Prediction 5. Conclusions and Future Works 2
  3. 3. Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016. 1. Introduction 3
  4. 4. Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016. Introduction. Main goals and project planning 4 Goals February March April May June Construct the Dataset Run state of the art saliency estimator with a single image Frames extraction Run saliency estimator with the extracted frames Compare Results
  5. 5. Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016. Equipment and Software. Eye tracker, Tobii Glasses 5
  6. 6. Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016. Equipment and Software. Tobii studio Software 6
  7. 7. Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016. Equipment and Software. 7
  8. 8. Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016. Equipment and Software. 8
  9. 9. Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016. Publication 9 Repositori of Egocentric-saliency in GitHub [online] Available: https://github.com/imatge-upc/egocentric-saliency EgoMon Dataset [online] Available: https://imatge.upc.edu/web/sites/default/files/resources/1720/saliency/2016-egomon/
  10. 10. Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016. Outline 1. Introduction 2. State of the art 3. EgoMon Gaze & Video Dataset 4. Visual Saliency Prediction 5. Conclusions and Future Works 10
  11. 11. Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016. 2. State of the art 11 GTEA Dataset UT Ego Dataset GTEA (Georgia Tech Egocentric Activities) – Gaze Dataset [online] Available: http://ai.stanford.edu/~alireza/GTEA_Gaze_Website/ UT (University of Texas) Ego Dataset [online] Available: http://vision.cs.utexas.edu/projects/egocentric_data/UT_Egocentric_Dataset.html
  12. 12. Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016. Outline 1. Introduction 2. State of the art 3. EgoMon Gaze & Video Dataset 4. Visual Saliency Prediction 5. Conclusions and Future Works 12
  13. 13. Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016. Acquisition. Calibration process of the Tobii Glasses 13 Video tutorial uploaded on YouTube.
  14. 14. Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016. Acquisition. Results of the calibration process of the Tobii Glasses 14
  15. 15. Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016. EgoMon Gaze & Video Dataset 15 ... 7 x text files (gaze data) 7 x RAW (videos) 7 x Gaze (videos with the gaze information plotted) 13428 x frames extracted 75 x narrative images
  16. 16. Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016. EgoMon Gaze & Video Dataset 16
  17. 17. Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016. EgoMon Gaze & Video Dataset 17 INDOOR OUTDOOR
  18. 18. Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016. EgoMon. Oral Presentation 18
  19. 19. Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016. EgoMon. DCU and Albert College Park 19
  20. 20. Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016. EgoMon. Spanish Omelette 20
  21. 21. Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016. EgoMon. Playing cards 21
  22. 22. Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016. EgoMon. Botanic Gardens 22
  23. 23. Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016. EgoMon. Botanic Gardens (Narrative Clip) 23
  24. 24. Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016. EgoMon. Bus Ride 24
  25. 25. Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016. EgoMon. Walking to the Office 25
  26. 26. Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016. EgoMon. Privacy 26
  27. 27. Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016. EgoMon. Problems with the Gaze (Losses) 27 static non-static
  28. 28. Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016. EgoMon. Processing, Eye Gaze data 28
  29. 29. Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016. EgoMon. Frame extraction 29 DURATION FRAMES EXTRACTED TOTAL 3:43:41 13428 AVERAGE: 0:34:30 1918 1 fps
  30. 30. Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016. Outline 1. Introduction 2. State of the art 3. EgoMon Gaze & Video Dataset 4. Visual Saliency Prediction 5. Conclusions and Future Works 30
  31. 31. Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016. 4. Visual Saliency Predictor. 31
  32. 32. Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016. Saliency Predictor. SalNet 32
  33. 33. Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016. EgoMon Gaze & Video Dataset 33 ... 7 x text files (gaze data) 7 x RAW (videos) 7 x Gaze (videos with the gaze information plotted) 13428 x frames extracted 75 x narrative images ...13428 x saliency models
  34. 34. Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016. Results of the Dataset 34
  35. 35. Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016. Quantitative Evaluation. Comparison Metric 35 Location-based Distribution-based AUC-Judd, sAUC, NSS SIM, CC, EMD, KL NORMALIZED SCANPATH SALIENCY MIT Saliency Benchmark [online] Available: http://saliency.mit.edu/results_mit300.html
  36. 36. Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016. Results. Quantitative Evaluation 36
  37. 37. Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016. Results. Qualitative Evaluation 37 Example of GOOD results
  38. 38. Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016. Results. Qualitative Evaluation 38 Example of BAD results
  39. 39. Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016. Outline 1. Introduction 2. State of the art 3. EgoMon Gaze & Video Dataset 4. Visual Saliency Prediction 5. Conclusions and Future Works 39
  40. 40. Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016. 40 Conclusions Dataset Amount of Data Recorded Device Environment Number of participants GTEA 17 sequences Tobii eye-tracker Glasses Indoor 14 UT Ego 4 videos of 4 hours (16 h) Looxcie wearable camera Indoor + Outdoor 4 EgoMon 7 clean videos (4 h) 7 gaze videos 13428 extracted frames 13428 saliency maps 7 files with eye gaze data 75 Narrative images Tobii eye tracker glasses + Narrative Cip Indoor + Outdoor 3
  41. 41. Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016. Future Works Fine-tuning of saliency estimator based on the comparison metric 41
  42. 42. Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016. Publication 42 http://imatge-upc.github.io/egocentric-2016-saliency/
  43. 43. Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016. 43

×