SlideShare a Scribd company logo
Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.
EgoMon Gaze and Video
Dataset for Visual Saliency
Prediction
Mònica Chertó Sarret Supervised by: Cathal Gurrin and Xavier Giró
Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.
Outline
1. Introduction
2. State of the art
3. EgoMon Gaze & Video Dataset
4. Visual Saliency Prediction
5. Conclusions and Future Works
2
Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.
1. Introduction
3
Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.
Introduction. Main goals and project planning
4
Goals February March April May June
Construct the Dataset
Run state of the art saliency estimator
with a single image
Frames extraction
Run saliency estimator with the
extracted frames
Compare Results
Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.
Equipment and Software. Eye tracker, Tobii Glasses
5
Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.
Equipment and Software. Tobii studio Software
6
Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.
Equipment and Software.
7
Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.
Equipment and Software.
8
Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.
Publication
9
Repositori of Egocentric-saliency in GitHub [online] Available: https://github.com/imatge-upc/egocentric-saliency
EgoMon Dataset [online] Available: https://imatge.upc.edu/web/sites/default/files/resources/1720/saliency/2016-egomon/
Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.
Outline
1. Introduction
2. State of the art
3. EgoMon Gaze & Video Dataset
4. Visual Saliency Prediction
5. Conclusions and Future Works
10
Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.
2. State of the art
11
GTEA Dataset UT Ego Dataset
GTEA (Georgia Tech Egocentric Activities) – Gaze Dataset [online] Available: http://ai.stanford.edu/~alireza/GTEA_Gaze_Website/
UT (University of Texas) Ego Dataset [online] Available: http://vision.cs.utexas.edu/projects/egocentric_data/UT_Egocentric_Dataset.html
Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.
Outline
1. Introduction
2. State of the art
3. EgoMon Gaze & Video Dataset
4. Visual Saliency Prediction
5. Conclusions and Future Works
12
Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.
Acquisition. Calibration process of the Tobii Glasses
13
Video tutorial uploaded on YouTube.
Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.
Acquisition. Results of the calibration process of the Tobii Glasses
14
Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.
EgoMon Gaze & Video Dataset
15
...
7 x text files
(gaze data)
7 x RAW (videos)
7 x Gaze (videos with
the gaze information
plotted)
13428 x frames extracted
75 x
narrative
images
Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.
EgoMon Gaze & Video Dataset
16
Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.
EgoMon Gaze & Video Dataset
17
INDOOR OUTDOOR
Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.
EgoMon. Oral Presentation
18
Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.
EgoMon. DCU and Albert College Park
19
Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.
EgoMon. Spanish Omelette
20
Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.
EgoMon. Playing cards
21
Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.
EgoMon. Botanic Gardens
22
Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.
EgoMon. Botanic Gardens (Narrative Clip)
23
Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.
EgoMon. Bus Ride
24
Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.
EgoMon. Walking to the Office
25
Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.
EgoMon. Privacy
26
Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.
EgoMon. Problems with the Gaze (Losses)
27
static
non-static
Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.
EgoMon. Processing, Eye Gaze data
28
Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.
EgoMon. Frame extraction
29
DURATION FRAMES EXTRACTED
TOTAL 3:43:41 13428
AVERAGE: 0:34:30 1918
1 fps
Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.
Outline
1. Introduction
2. State of the art
3. EgoMon Gaze & Video Dataset
4. Visual Saliency Prediction
5. Conclusions and Future Works
30
Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.
4. Visual Saliency Predictor.
31
Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.
Saliency Predictor. SalNet
32
Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.
EgoMon Gaze & Video Dataset
33
...
7 x text files
(gaze data)
7 x RAW (videos)
7 x Gaze (videos with
the gaze information
plotted)
13428 x frames extracted
75 x
narrative
images
...13428 x saliency models
Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.
Results of the Dataset
34
Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.
Quantitative Evaluation. Comparison Metric
35
Location-based Distribution-based
AUC-Judd, sAUC, NSS SIM, CC, EMD, KL
NORMALIZED SCANPATH SALIENCY
MIT Saliency Benchmark [online] Available: http://saliency.mit.edu/results_mit300.html
Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.
Results. Quantitative Evaluation
36
Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.
Results. Qualitative Evaluation
37
Example of GOOD results
Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.
Results. Qualitative Evaluation
38
Example of BAD results
Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.
Outline
1. Introduction
2. State of the art
3. EgoMon Gaze & Video Dataset
4. Visual Saliency Prediction
5. Conclusions and Future Works
39
Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.
40
Conclusions
Dataset Amount of Data Recorded
Device
Environment Number of
participants
GTEA 17 sequences Tobii eye-tracker
Glasses
Indoor 14
UT Ego 4 videos of 4 hours (16
h)
Looxcie
wearable camera
Indoor + Outdoor 4
EgoMon 7 clean videos (4 h)
7 gaze videos
13428 extracted frames
13428 saliency maps
7 files with eye gaze data
75 Narrative images
Tobii eye tracker
glasses +
Narrative Cip
Indoor + Outdoor 3
Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.
Future Works
Fine-tuning of saliency estimator based on the
comparison metric
41
Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.
Publication
42
http://imatge-upc.github.io/egocentric-2016-saliency/
Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.
43

More Related Content

Viewers also liked

Strategy Instruction in writing
Strategy Instruction in writingStrategy Instruction in writing
Strategy Instruction in writing
mystiquemel
 
Quand lecture rime avec plaisir
Quand lecture rime avec plaisirQuand lecture rime avec plaisir
Quand lecture rime avec plaisir
Soumia EL Yaacoubi
 
Ppt eng y4
Ppt eng y4Ppt eng y4
Ppt eng y4
azura272
 
P7 e2 josemariabarrio
P7 e2 josemariabarrio P7 e2 josemariabarrio
P7 e2 josemariabarrio
Jose Maria Barrio Giron
 
538df1cdf0b7f
538df1cdf0b7f538df1cdf0b7f
538df1cdf0b7f
Mourad Karoudi
 
Zentangle Animals
Zentangle AnimalsZentangle Animals
Zentangle Animals
quicarroll
 
Musicas cifradas mpb 5
Musicas cifradas mpb 5Musicas cifradas mpb 5
Musicas cifradas mpb 5
Nome Sobrenome
 
(Nunca) perder la esperanza.
(Nunca) perder la esperanza.(Nunca) perder la esperanza.
(Nunca) perder la esperanza.
José María
 

Viewers also liked (8)

Strategy Instruction in writing
Strategy Instruction in writingStrategy Instruction in writing
Strategy Instruction in writing
 
Quand lecture rime avec plaisir
Quand lecture rime avec plaisirQuand lecture rime avec plaisir
Quand lecture rime avec plaisir
 
Ppt eng y4
Ppt eng y4Ppt eng y4
Ppt eng y4
 
P7 e2 josemariabarrio
P7 e2 josemariabarrio P7 e2 josemariabarrio
P7 e2 josemariabarrio
 
538df1cdf0b7f
538df1cdf0b7f538df1cdf0b7f
538df1cdf0b7f
 
Zentangle Animals
Zentangle AnimalsZentangle Animals
Zentangle Animals
 
Musicas cifradas mpb 5
Musicas cifradas mpb 5Musicas cifradas mpb 5
Musicas cifradas mpb 5
 
(Nunca) perder la esperanza.
(Nunca) perder la esperanza.(Nunca) perder la esperanza.
(Nunca) perder la esperanza.
 

More from Universitat Politècnica de Catalunya

Deep Generative Learning for All - The Gen AI Hype (Spring 2024)
Deep Generative Learning for All - The Gen AI Hype (Spring 2024)Deep Generative Learning for All - The Gen AI Hype (Spring 2024)
Deep Generative Learning for All - The Gen AI Hype (Spring 2024)
Universitat Politècnica de Catalunya
 
Deep Generative Learning for All
Deep Generative Learning for AllDeep Generative Learning for All
Deep Generative Learning for All
Universitat Politècnica de Catalunya
 
The Transformer in Vision | Xavier Giro | Master in Computer Vision Barcelona...
The Transformer in Vision | Xavier Giro | Master in Computer Vision Barcelona...The Transformer in Vision | Xavier Giro | Master in Computer Vision Barcelona...
The Transformer in Vision | Xavier Giro | Master in Computer Vision Barcelona...
Universitat Politècnica de Catalunya
 
Towards Sign Language Translation & Production | Xavier Giro-i-Nieto
Towards Sign Language Translation & Production | Xavier Giro-i-NietoTowards Sign Language Translation & Production | Xavier Giro-i-Nieto
Towards Sign Language Translation & Production | Xavier Giro-i-Nieto
Universitat Politècnica de Catalunya
 
The Transformer - Xavier Giró - UPC Barcelona 2021
The Transformer - Xavier Giró - UPC Barcelona 2021The Transformer - Xavier Giró - UPC Barcelona 2021
The Transformer - Xavier Giró - UPC Barcelona 2021
Universitat Politècnica de Catalunya
 
Learning Representations for Sign Language Videos - Xavier Giro - NIST TRECVI...
Learning Representations for Sign Language Videos - Xavier Giro - NIST TRECVI...Learning Representations for Sign Language Videos - Xavier Giro - NIST TRECVI...
Learning Representations for Sign Language Videos - Xavier Giro - NIST TRECVI...
Universitat Politècnica de Catalunya
 
Open challenges in sign language translation and production
Open challenges in sign language translation and productionOpen challenges in sign language translation and production
Open challenges in sign language translation and production
Universitat Politècnica de Catalunya
 
Generation of Synthetic Referring Expressions for Object Segmentation in Videos
Generation of Synthetic Referring Expressions for Object Segmentation in VideosGeneration of Synthetic Referring Expressions for Object Segmentation in Videos
Generation of Synthetic Referring Expressions for Object Segmentation in Videos
Universitat Politècnica de Catalunya
 
Discovery and Learning of Navigation Goals from Pixels in Minecraft
Discovery and Learning of Navigation Goals from Pixels in MinecraftDiscovery and Learning of Navigation Goals from Pixels in Minecraft
Discovery and Learning of Navigation Goals from Pixels in Minecraft
Universitat Politècnica de Catalunya
 
Learn2Sign : Sign language recognition and translation using human keypoint e...
Learn2Sign : Sign language recognition and translation using human keypoint e...Learn2Sign : Sign language recognition and translation using human keypoint e...
Learn2Sign : Sign language recognition and translation using human keypoint e...
Universitat Politècnica de Catalunya
 
Intepretability / Explainable AI for Deep Neural Networks
Intepretability / Explainable AI for Deep Neural NetworksIntepretability / Explainable AI for Deep Neural Networks
Intepretability / Explainable AI for Deep Neural Networks
Universitat Politècnica de Catalunya
 
Convolutional Neural Networks - Xavier Giro - UPC TelecomBCN Barcelona 2020
Convolutional Neural Networks - Xavier Giro - UPC TelecomBCN Barcelona 2020Convolutional Neural Networks - Xavier Giro - UPC TelecomBCN Barcelona 2020
Convolutional Neural Networks - Xavier Giro - UPC TelecomBCN Barcelona 2020
Universitat Politècnica de Catalunya
 
Self-Supervised Audio-Visual Learning - Xavier Giro - UPC TelecomBCN Barcelon...
Self-Supervised Audio-Visual Learning - Xavier Giro - UPC TelecomBCN Barcelon...Self-Supervised Audio-Visual Learning - Xavier Giro - UPC TelecomBCN Barcelon...
Self-Supervised Audio-Visual Learning - Xavier Giro - UPC TelecomBCN Barcelon...
Universitat Politècnica de Catalunya
 
Attention for Deep Learning - Xavier Giro - UPC TelecomBCN Barcelona 2020
Attention for Deep Learning - Xavier Giro - UPC TelecomBCN Barcelona 2020Attention for Deep Learning - Xavier Giro - UPC TelecomBCN Barcelona 2020
Attention for Deep Learning - Xavier Giro - UPC TelecomBCN Barcelona 2020
Universitat Politècnica de Catalunya
 
Generative Adversarial Networks GAN - Xavier Giro - UPC TelecomBCN Barcelona ...
Generative Adversarial Networks GAN - Xavier Giro - UPC TelecomBCN Barcelona ...Generative Adversarial Networks GAN - Xavier Giro - UPC TelecomBCN Barcelona ...
Generative Adversarial Networks GAN - Xavier Giro - UPC TelecomBCN Barcelona ...
Universitat Politècnica de Catalunya
 
Q-Learning with a Neural Network - Xavier Giró - UPC Barcelona 2020
Q-Learning with a Neural Network - Xavier Giró - UPC Barcelona 2020Q-Learning with a Neural Network - Xavier Giró - UPC Barcelona 2020
Q-Learning with a Neural Network - Xavier Giró - UPC Barcelona 2020
Universitat Politècnica de Catalunya
 
Language and Vision with Deep Learning - Xavier Giró - ACM ICMR 2020 (Tutorial)
Language and Vision with Deep Learning - Xavier Giró - ACM ICMR 2020 (Tutorial)Language and Vision with Deep Learning - Xavier Giró - ACM ICMR 2020 (Tutorial)
Language and Vision with Deep Learning - Xavier Giró - ACM ICMR 2020 (Tutorial)
Universitat Politècnica de Catalunya
 
Image Segmentation with Deep Learning - Xavier Giro & Carles Ventura - ISSonD...
Image Segmentation with Deep Learning - Xavier Giro & Carles Ventura - ISSonD...Image Segmentation with Deep Learning - Xavier Giro & Carles Ventura - ISSonD...
Image Segmentation with Deep Learning - Xavier Giro & Carles Ventura - ISSonD...
Universitat Politècnica de Catalunya
 
Curriculum Learning for Recurrent Video Object Segmentation
Curriculum Learning for Recurrent Video Object SegmentationCurriculum Learning for Recurrent Video Object Segmentation
Curriculum Learning for Recurrent Video Object Segmentation
Universitat Politècnica de Catalunya
 
Deep Self-supervised Learning for All - Xavier Giro - X-Europe 2020
Deep Self-supervised Learning for All - Xavier Giro - X-Europe 2020Deep Self-supervised Learning for All - Xavier Giro - X-Europe 2020
Deep Self-supervised Learning for All - Xavier Giro - X-Europe 2020
Universitat Politècnica de Catalunya
 

More from Universitat Politècnica de Catalunya (20)

Deep Generative Learning for All - The Gen AI Hype (Spring 2024)
Deep Generative Learning for All - The Gen AI Hype (Spring 2024)Deep Generative Learning for All - The Gen AI Hype (Spring 2024)
Deep Generative Learning for All - The Gen AI Hype (Spring 2024)
 
Deep Generative Learning for All
Deep Generative Learning for AllDeep Generative Learning for All
Deep Generative Learning for All
 
The Transformer in Vision | Xavier Giro | Master in Computer Vision Barcelona...
The Transformer in Vision | Xavier Giro | Master in Computer Vision Barcelona...The Transformer in Vision | Xavier Giro | Master in Computer Vision Barcelona...
The Transformer in Vision | Xavier Giro | Master in Computer Vision Barcelona...
 
Towards Sign Language Translation & Production | Xavier Giro-i-Nieto
Towards Sign Language Translation & Production | Xavier Giro-i-NietoTowards Sign Language Translation & Production | Xavier Giro-i-Nieto
Towards Sign Language Translation & Production | Xavier Giro-i-Nieto
 
The Transformer - Xavier Giró - UPC Barcelona 2021
The Transformer - Xavier Giró - UPC Barcelona 2021The Transformer - Xavier Giró - UPC Barcelona 2021
The Transformer - Xavier Giró - UPC Barcelona 2021
 
Learning Representations for Sign Language Videos - Xavier Giro - NIST TRECVI...
Learning Representations for Sign Language Videos - Xavier Giro - NIST TRECVI...Learning Representations for Sign Language Videos - Xavier Giro - NIST TRECVI...
Learning Representations for Sign Language Videos - Xavier Giro - NIST TRECVI...
 
Open challenges in sign language translation and production
Open challenges in sign language translation and productionOpen challenges in sign language translation and production
Open challenges in sign language translation and production
 
Generation of Synthetic Referring Expressions for Object Segmentation in Videos
Generation of Synthetic Referring Expressions for Object Segmentation in VideosGeneration of Synthetic Referring Expressions for Object Segmentation in Videos
Generation of Synthetic Referring Expressions for Object Segmentation in Videos
 
Discovery and Learning of Navigation Goals from Pixels in Minecraft
Discovery and Learning of Navigation Goals from Pixels in MinecraftDiscovery and Learning of Navigation Goals from Pixels in Minecraft
Discovery and Learning of Navigation Goals from Pixels in Minecraft
 
Learn2Sign : Sign language recognition and translation using human keypoint e...
Learn2Sign : Sign language recognition and translation using human keypoint e...Learn2Sign : Sign language recognition and translation using human keypoint e...
Learn2Sign : Sign language recognition and translation using human keypoint e...
 
Intepretability / Explainable AI for Deep Neural Networks
Intepretability / Explainable AI for Deep Neural NetworksIntepretability / Explainable AI for Deep Neural Networks
Intepretability / Explainable AI for Deep Neural Networks
 
Convolutional Neural Networks - Xavier Giro - UPC TelecomBCN Barcelona 2020
Convolutional Neural Networks - Xavier Giro - UPC TelecomBCN Barcelona 2020Convolutional Neural Networks - Xavier Giro - UPC TelecomBCN Barcelona 2020
Convolutional Neural Networks - Xavier Giro - UPC TelecomBCN Barcelona 2020
 
Self-Supervised Audio-Visual Learning - Xavier Giro - UPC TelecomBCN Barcelon...
Self-Supervised Audio-Visual Learning - Xavier Giro - UPC TelecomBCN Barcelon...Self-Supervised Audio-Visual Learning - Xavier Giro - UPC TelecomBCN Barcelon...
Self-Supervised Audio-Visual Learning - Xavier Giro - UPC TelecomBCN Barcelon...
 
Attention for Deep Learning - Xavier Giro - UPC TelecomBCN Barcelona 2020
Attention for Deep Learning - Xavier Giro - UPC TelecomBCN Barcelona 2020Attention for Deep Learning - Xavier Giro - UPC TelecomBCN Barcelona 2020
Attention for Deep Learning - Xavier Giro - UPC TelecomBCN Barcelona 2020
 
Generative Adversarial Networks GAN - Xavier Giro - UPC TelecomBCN Barcelona ...
Generative Adversarial Networks GAN - Xavier Giro - UPC TelecomBCN Barcelona ...Generative Adversarial Networks GAN - Xavier Giro - UPC TelecomBCN Barcelona ...
Generative Adversarial Networks GAN - Xavier Giro - UPC TelecomBCN Barcelona ...
 
Q-Learning with a Neural Network - Xavier Giró - UPC Barcelona 2020
Q-Learning with a Neural Network - Xavier Giró - UPC Barcelona 2020Q-Learning with a Neural Network - Xavier Giró - UPC Barcelona 2020
Q-Learning with a Neural Network - Xavier Giró - UPC Barcelona 2020
 
Language and Vision with Deep Learning - Xavier Giró - ACM ICMR 2020 (Tutorial)
Language and Vision with Deep Learning - Xavier Giró - ACM ICMR 2020 (Tutorial)Language and Vision with Deep Learning - Xavier Giró - ACM ICMR 2020 (Tutorial)
Language and Vision with Deep Learning - Xavier Giró - ACM ICMR 2020 (Tutorial)
 
Image Segmentation with Deep Learning - Xavier Giro & Carles Ventura - ISSonD...
Image Segmentation with Deep Learning - Xavier Giro & Carles Ventura - ISSonD...Image Segmentation with Deep Learning - Xavier Giro & Carles Ventura - ISSonD...
Image Segmentation with Deep Learning - Xavier Giro & Carles Ventura - ISSonD...
 
Curriculum Learning for Recurrent Video Object Segmentation
Curriculum Learning for Recurrent Video Object SegmentationCurriculum Learning for Recurrent Video Object Segmentation
Curriculum Learning for Recurrent Video Object Segmentation
 
Deep Self-supervised Learning for All - Xavier Giro - X-Europe 2020
Deep Self-supervised Learning for All - Xavier Giro - X-Europe 2020Deep Self-supervised Learning for All - Xavier Giro - X-Europe 2020
Deep Self-supervised Learning for All - Xavier Giro - X-Europe 2020
 

Recently uploaded

AWS Certified Solutions Architect Associate (SAA-C03)
AWS Certified Solutions Architect Associate (SAA-C03)AWS Certified Solutions Architect Associate (SAA-C03)
AWS Certified Solutions Architect Associate (SAA-C03)
HarpalGohil4
 
Apps Break Data
Apps Break DataApps Break Data
Apps Break Data
Ivo Velitchkov
 
"Choosing proper type of scaling", Olena Syrota
"Choosing proper type of scaling", Olena Syrota"Choosing proper type of scaling", Olena Syrota
"Choosing proper type of scaling", Olena Syrota
Fwdays
 
What is an RPA CoE? Session 2 – CoE Roles
What is an RPA CoE?  Session 2 – CoE RolesWhat is an RPA CoE?  Session 2 – CoE Roles
What is an RPA CoE? Session 2 – CoE Roles
DianaGray10
 
inQuba Webinar Mastering Customer Journey Management with Dr Graham Hill
inQuba Webinar Mastering Customer Journey Management with Dr Graham HillinQuba Webinar Mastering Customer Journey Management with Dr Graham Hill
inQuba Webinar Mastering Customer Journey Management with Dr Graham Hill
LizaNolte
 
GraphRAG for LifeSciences Hands-On with the Clinical Knowledge Graph
GraphRAG for LifeSciences Hands-On with the Clinical Knowledge GraphGraphRAG for LifeSciences Hands-On with the Clinical Knowledge Graph
GraphRAG for LifeSciences Hands-On with the Clinical Knowledge Graph
Neo4j
 
Getting the Most Out of ScyllaDB Monitoring: ShareChat's Tips
Getting the Most Out of ScyllaDB Monitoring: ShareChat's TipsGetting the Most Out of ScyllaDB Monitoring: ShareChat's Tips
Getting the Most Out of ScyllaDB Monitoring: ShareChat's Tips
ScyllaDB
 
"Frontline Battles with DDoS: Best practices and Lessons Learned", Igor Ivaniuk
"Frontline Battles with DDoS: Best practices and Lessons Learned",  Igor Ivaniuk"Frontline Battles with DDoS: Best practices and Lessons Learned",  Igor Ivaniuk
"Frontline Battles with DDoS: Best practices and Lessons Learned", Igor Ivaniuk
Fwdays
 
Christine's Supplier Sourcing Presentaion.pptx
Christine's Supplier Sourcing Presentaion.pptxChristine's Supplier Sourcing Presentaion.pptx
Christine's Supplier Sourcing Presentaion.pptx
christinelarrosa
 
AI in the Workplace Reskilling, Upskilling, and Future Work.pptx
AI in the Workplace Reskilling, Upskilling, and Future Work.pptxAI in the Workplace Reskilling, Upskilling, and Future Work.pptx
AI in the Workplace Reskilling, Upskilling, and Future Work.pptx
Sunil Jagani
 
What is an RPA CoE? Session 1 – CoE Vision
What is an RPA CoE?  Session 1 – CoE VisionWhat is an RPA CoE?  Session 1 – CoE Vision
What is an RPA CoE? Session 1 – CoE Vision
DianaGray10
 
From Natural Language to Structured Solr Queries using LLMs
From Natural Language to Structured Solr Queries using LLMsFrom Natural Language to Structured Solr Queries using LLMs
From Natural Language to Structured Solr Queries using LLMs
Sease
 
Mutation Testing for Task-Oriented Chatbots
Mutation Testing for Task-Oriented ChatbotsMutation Testing for Task-Oriented Chatbots
Mutation Testing for Task-Oriented Chatbots
Pablo Gómez Abajo
 
Harnessing the Power of NLP and Knowledge Graphs for Opioid Research
Harnessing the Power of NLP and Knowledge Graphs for Opioid ResearchHarnessing the Power of NLP and Knowledge Graphs for Opioid Research
Harnessing the Power of NLP and Knowledge Graphs for Opioid Research
Neo4j
 
Northern Engraving | Modern Metal Trim, Nameplates and Appliance Panels
Northern Engraving | Modern Metal Trim, Nameplates and Appliance PanelsNorthern Engraving | Modern Metal Trim, Nameplates and Appliance Panels
Northern Engraving | Modern Metal Trim, Nameplates and Appliance Panels
Northern Engraving
 
PRODUCT LISTING OPTIMIZATION PRESENTATION.pptx
PRODUCT LISTING OPTIMIZATION PRESENTATION.pptxPRODUCT LISTING OPTIMIZATION PRESENTATION.pptx
PRODUCT LISTING OPTIMIZATION PRESENTATION.pptx
christinelarrosa
 
Discover the Unseen: Tailored Recommendation of Unwatched Content
Discover the Unseen: Tailored Recommendation of Unwatched ContentDiscover the Unseen: Tailored Recommendation of Unwatched Content
Discover the Unseen: Tailored Recommendation of Unwatched Content
ScyllaDB
 
Biomedical Knowledge Graphs for Data Scientists and Bioinformaticians
Biomedical Knowledge Graphs for Data Scientists and BioinformaticiansBiomedical Knowledge Graphs for Data Scientists and Bioinformaticians
Biomedical Knowledge Graphs for Data Scientists and Bioinformaticians
Neo4j
 
ScyllaDB Tablets: Rethinking Replication
ScyllaDB Tablets: Rethinking ReplicationScyllaDB Tablets: Rethinking Replication
ScyllaDB Tablets: Rethinking Replication
ScyllaDB
 
Containers & AI - Beauty and the Beast!?!
Containers & AI - Beauty and the Beast!?!Containers & AI - Beauty and the Beast!?!
Containers & AI - Beauty and the Beast!?!
Tobias Schneck
 

Recently uploaded (20)

AWS Certified Solutions Architect Associate (SAA-C03)
AWS Certified Solutions Architect Associate (SAA-C03)AWS Certified Solutions Architect Associate (SAA-C03)
AWS Certified Solutions Architect Associate (SAA-C03)
 
Apps Break Data
Apps Break DataApps Break Data
Apps Break Data
 
"Choosing proper type of scaling", Olena Syrota
"Choosing proper type of scaling", Olena Syrota"Choosing proper type of scaling", Olena Syrota
"Choosing proper type of scaling", Olena Syrota
 
What is an RPA CoE? Session 2 – CoE Roles
What is an RPA CoE?  Session 2 – CoE RolesWhat is an RPA CoE?  Session 2 – CoE Roles
What is an RPA CoE? Session 2 – CoE Roles
 
inQuba Webinar Mastering Customer Journey Management with Dr Graham Hill
inQuba Webinar Mastering Customer Journey Management with Dr Graham HillinQuba Webinar Mastering Customer Journey Management with Dr Graham Hill
inQuba Webinar Mastering Customer Journey Management with Dr Graham Hill
 
GraphRAG for LifeSciences Hands-On with the Clinical Knowledge Graph
GraphRAG for LifeSciences Hands-On with the Clinical Knowledge GraphGraphRAG for LifeSciences Hands-On with the Clinical Knowledge Graph
GraphRAG for LifeSciences Hands-On with the Clinical Knowledge Graph
 
Getting the Most Out of ScyllaDB Monitoring: ShareChat's Tips
Getting the Most Out of ScyllaDB Monitoring: ShareChat's TipsGetting the Most Out of ScyllaDB Monitoring: ShareChat's Tips
Getting the Most Out of ScyllaDB Monitoring: ShareChat's Tips
 
"Frontline Battles with DDoS: Best practices and Lessons Learned", Igor Ivaniuk
"Frontline Battles with DDoS: Best practices and Lessons Learned",  Igor Ivaniuk"Frontline Battles with DDoS: Best practices and Lessons Learned",  Igor Ivaniuk
"Frontline Battles with DDoS: Best practices and Lessons Learned", Igor Ivaniuk
 
Christine's Supplier Sourcing Presentaion.pptx
Christine's Supplier Sourcing Presentaion.pptxChristine's Supplier Sourcing Presentaion.pptx
Christine's Supplier Sourcing Presentaion.pptx
 
AI in the Workplace Reskilling, Upskilling, and Future Work.pptx
AI in the Workplace Reskilling, Upskilling, and Future Work.pptxAI in the Workplace Reskilling, Upskilling, and Future Work.pptx
AI in the Workplace Reskilling, Upskilling, and Future Work.pptx
 
What is an RPA CoE? Session 1 – CoE Vision
What is an RPA CoE?  Session 1 – CoE VisionWhat is an RPA CoE?  Session 1 – CoE Vision
What is an RPA CoE? Session 1 – CoE Vision
 
From Natural Language to Structured Solr Queries using LLMs
From Natural Language to Structured Solr Queries using LLMsFrom Natural Language to Structured Solr Queries using LLMs
From Natural Language to Structured Solr Queries using LLMs
 
Mutation Testing for Task-Oriented Chatbots
Mutation Testing for Task-Oriented ChatbotsMutation Testing for Task-Oriented Chatbots
Mutation Testing for Task-Oriented Chatbots
 
Harnessing the Power of NLP and Knowledge Graphs for Opioid Research
Harnessing the Power of NLP and Knowledge Graphs for Opioid ResearchHarnessing the Power of NLP and Knowledge Graphs for Opioid Research
Harnessing the Power of NLP and Knowledge Graphs for Opioid Research
 
Northern Engraving | Modern Metal Trim, Nameplates and Appliance Panels
Northern Engraving | Modern Metal Trim, Nameplates and Appliance PanelsNorthern Engraving | Modern Metal Trim, Nameplates and Appliance Panels
Northern Engraving | Modern Metal Trim, Nameplates and Appliance Panels
 
PRODUCT LISTING OPTIMIZATION PRESENTATION.pptx
PRODUCT LISTING OPTIMIZATION PRESENTATION.pptxPRODUCT LISTING OPTIMIZATION PRESENTATION.pptx
PRODUCT LISTING OPTIMIZATION PRESENTATION.pptx
 
Discover the Unseen: Tailored Recommendation of Unwatched Content
Discover the Unseen: Tailored Recommendation of Unwatched ContentDiscover the Unseen: Tailored Recommendation of Unwatched Content
Discover the Unseen: Tailored Recommendation of Unwatched Content
 
Biomedical Knowledge Graphs for Data Scientists and Bioinformaticians
Biomedical Knowledge Graphs for Data Scientists and BioinformaticiansBiomedical Knowledge Graphs for Data Scientists and Bioinformaticians
Biomedical Knowledge Graphs for Data Scientists and Bioinformaticians
 
ScyllaDB Tablets: Rethinking Replication
ScyllaDB Tablets: Rethinking ReplicationScyllaDB Tablets: Rethinking Replication
ScyllaDB Tablets: Rethinking Replication
 
Containers & AI - Beauty and the Beast!?!
Containers & AI - Beauty and the Beast!?!Containers & AI - Beauty and the Beast!?!
Containers & AI - Beauty and the Beast!?!
 

EgoMon Gaze and Video Dataset for Visual Saliency Prediction

  • 1. Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016. EgoMon Gaze and Video Dataset for Visual Saliency Prediction Mònica Chertó Sarret Supervised by: Cathal Gurrin and Xavier Giró
  • 2. Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016. Outline 1. Introduction 2. State of the art 3. EgoMon Gaze & Video Dataset 4. Visual Saliency Prediction 5. Conclusions and Future Works 2
  • 3. Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016. 1. Introduction 3
  • 4. Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016. Introduction. Main goals and project planning 4 Goals February March April May June Construct the Dataset Run state of the art saliency estimator with a single image Frames extraction Run saliency estimator with the extracted frames Compare Results
  • 5. Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016. Equipment and Software. Eye tracker, Tobii Glasses 5
  • 6. Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016. Equipment and Software. Tobii studio Software 6
  • 7. Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016. Equipment and Software. 7
  • 8. Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016. Equipment and Software. 8
  • 9. Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016. Publication 9 Repositori of Egocentric-saliency in GitHub [online] Available: https://github.com/imatge-upc/egocentric-saliency EgoMon Dataset [online] Available: https://imatge.upc.edu/web/sites/default/files/resources/1720/saliency/2016-egomon/
  • 10. Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016. Outline 1. Introduction 2. State of the art 3. EgoMon Gaze & Video Dataset 4. Visual Saliency Prediction 5. Conclusions and Future Works 10
  • 11. Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016. 2. State of the art 11 GTEA Dataset UT Ego Dataset GTEA (Georgia Tech Egocentric Activities) – Gaze Dataset [online] Available: http://ai.stanford.edu/~alireza/GTEA_Gaze_Website/ UT (University of Texas) Ego Dataset [online] Available: http://vision.cs.utexas.edu/projects/egocentric_data/UT_Egocentric_Dataset.html
  • 12. Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016. Outline 1. Introduction 2. State of the art 3. EgoMon Gaze & Video Dataset 4. Visual Saliency Prediction 5. Conclusions and Future Works 12
  • 13. Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016. Acquisition. Calibration process of the Tobii Glasses 13 Video tutorial uploaded on YouTube.
  • 14. Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016. Acquisition. Results of the calibration process of the Tobii Glasses 14
  • 15. Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016. EgoMon Gaze & Video Dataset 15 ... 7 x text files (gaze data) 7 x RAW (videos) 7 x Gaze (videos with the gaze information plotted) 13428 x frames extracted 75 x narrative images
  • 16. Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016. EgoMon Gaze & Video Dataset 16
  • 17. Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016. EgoMon Gaze & Video Dataset 17 INDOOR OUTDOOR
  • 18. Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016. EgoMon. Oral Presentation 18
  • 19. Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016. EgoMon. DCU and Albert College Park 19
  • 20. Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016. EgoMon. Spanish Omelette 20
  • 21. Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016. EgoMon. Playing cards 21
  • 22. Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016. EgoMon. Botanic Gardens 22
  • 23. Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016. EgoMon. Botanic Gardens (Narrative Clip) 23
  • 24. Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016. EgoMon. Bus Ride 24
  • 25. Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016. EgoMon. Walking to the Office 25
  • 26. Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016. EgoMon. Privacy 26
  • 27. Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016. EgoMon. Problems with the Gaze (Losses) 27 static non-static
  • 28. Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016. EgoMon. Processing, Eye Gaze data 28
  • 29. Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016. EgoMon. Frame extraction 29 DURATION FRAMES EXTRACTED TOTAL 3:43:41 13428 AVERAGE: 0:34:30 1918 1 fps
  • 30. Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016. Outline 1. Introduction 2. State of the art 3. EgoMon Gaze & Video Dataset 4. Visual Saliency Prediction 5. Conclusions and Future Works 30
  • 31. Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016. 4. Visual Saliency Predictor. 31
  • 32. Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016. Saliency Predictor. SalNet 32
  • 33. Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016. EgoMon Gaze & Video Dataset 33 ... 7 x text files (gaze data) 7 x RAW (videos) 7 x Gaze (videos with the gaze information plotted) 13428 x frames extracted 75 x narrative images ...13428 x saliency models
  • 34. Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016. Results of the Dataset 34
  • 35. Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016. Quantitative Evaluation. Comparison Metric 35 Location-based Distribution-based AUC-Judd, sAUC, NSS SIM, CC, EMD, KL NORMALIZED SCANPATH SALIENCY MIT Saliency Benchmark [online] Available: http://saliency.mit.edu/results_mit300.html
  • 36. Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016. Results. Quantitative Evaluation 36
  • 37. Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016. Results. Qualitative Evaluation 37 Example of GOOD results
  • 38. Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016. Results. Qualitative Evaluation 38 Example of BAD results
  • 39. Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016. Outline 1. Introduction 2. State of the art 3. EgoMon Gaze & Video Dataset 4. Visual Saliency Prediction 5. Conclusions and Future Works 39
  • 40. Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016. 40 Conclusions Dataset Amount of Data Recorded Device Environment Number of participants GTEA 17 sequences Tobii eye-tracker Glasses Indoor 14 UT Ego 4 videos of 4 hours (16 h) Looxcie wearable camera Indoor + Outdoor 4 EgoMon 7 clean videos (4 h) 7 gaze videos 13428 extracted frames 13428 saliency maps 7 files with eye gaze data 75 Narrative images Tobii eye tracker glasses + Narrative Cip Indoor + Outdoor 3
  • 41. Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016. Future Works Fine-tuning of saliency estimator based on the comparison metric 41
  • 42. Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016. Publication 42 http://imatge-upc.github.io/egocentric-2016-saliency/
  • 43. Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016. 43