Learning Where and When to Look #reworkRETAIL 2018
Report
Mar. 17, 2018•0 likes•628 views
2 likes
Be the first to like this
Show More
views
Total views
0
On Slideshare
0
From embeds
0
Number of embeds
0
Check these out next
Learning Where and When to Look #reworkRETAIL 2018
Mar. 17, 2018•0 likes•628 views
2 likes
Be the first to like this
Show More
views
Total views
0
On Slideshare
0
From embeds
0
Number of embeds
0
Download to read offline
Report
Data & Analytics
Learning Where and When to Look Deep learning models do not only achieve superior performances in image recognition tasks, but also in predicting where and when users focus their attention. This talk will provide an overview of how convolutional neural networks have been trained to predict saliency maps that describe the probability of fixing the gaze on each image location. Different solution have been proposed for this task, and our recent work has added a temporal dimension by predicting the gaze scanpath over 360 degree images for VR/AR. These techniques allow simulating eye tracker data with no need of user data collection. https://www.re-work.co/events/deep-learning-in-retail-summit-london-2018
Recommended
The Transformer in Vision | Xavier Giro | Master in Computer Vision Barcelona...Universitat Politècnica de Catalunya
Towards Sign Language Translation & Production | Xavier Giro-i-NietoUniversitat Politècnica de Catalunya
The Transformer - Xavier Giró - UPC Barcelona 2021Universitat Politècnica de Catalunya
Learning Representations for Sign Language Videos - Xavier Giro - NIST TRECVI...Universitat Politècnica de Catalunya
Open challenges in sign language translation and productionUniversitat Politècnica de Catalunya
Generation of Synthetic Referring Expressions for Object Segmentation in VideosUniversitat Politècnica de Catalunya
Learning Where and When to Look #reworkRETAIL 2018
- DEEP LEARNING IN RETAIL & ADVERTISING SUMMIT London, UK 16th March 2018 Xavier Giro-i-Nieto xavier.giro@upc.edu Associate Professor Universitat Politecnica de Catalunya Technical University of Catalonia Learning Where and When to Look @DocXavi[Slides on GDrive] #reworkRETAIL
- A salient team Kevin McGuinness Noel E. O’Connor Cristian Canton Junting Pan Marc Assens Marta Coll Elisa Sayrol Jordi Torres
- 3 Click to learn more
- 4 Let’s play a game!
- The importance of visual attention
- The importance of visual attention
- The importance of visual attention
- The importance of visual attention
- The importance of visual attention
- Why don’t we see the changes? We don’t really see the whole image We only focus on small specific regions: the salient parts Human beings reliably attend to the same regions of images when shown
- What we perceive
- Where we look
- What we actually see
- Can we predict WHEREhumans will look? Yes! Computational models of visual saliency Why might this be useful?
- 17Johanna Närväinen, Janne Laine, “Looking through their eyes” VTT Impulse 2016.
- 18Johanna Närväinen, Janne Laine, “Looking through their eyes” VTT Impulse 2016.
- 19Johanna Närväinen, Janne Laine, “Looking through their eyes” VTT Impulse 2016.
- Image (input) Saliency map (ouput)
- 21John Markoff, “Scientists see promise in deep learning Programs”, The New York Times (2012). Photo: Keith Penner
- 22Yazmin How, “Interview: Yoshua Bengio, Yann LeCun, Geoffrey Hinton” RE·WORK (2017).
- 23A Krizhevsky, I Sutskever, GE Hinton “Imagenet classification with deep convolutional neural networks” NIPS 2012 Orange AlexNet Photo: Keith Penner
- 24 DATA AlexNet ALGORITHMS CNNs COMPUTATION
- 25 JuntingNet ALGORITHMS CNNs DATA COMPUTATION
- 26J. Pan, X. Giró-i-Nieto, “End-to-end Convolutional Network for Saliency Prediction” (CVPRW 2015) End to end CNN for Saliency Prediction
- 27 Upsample + filter 2D map J. Pan, X. Giró-i-Nieto, “End-to-end Convolutional Network for Saliency Prediction” (CVPRW 2015) JuntingNet
- 28
- 29 JuntingNet ALGORITHMS CNNs DATA COMPUTATION
- 30 SalNet ALGORITHMS CNNs DATA COMPUTATION
- 31 Convolution 1 (7 × 7 × 96) Input (size 320 × 240 × 3) Local response normalization Max pool (kernel 3 × 3 stride 2) Convolution 2 (5 × 5 × 256) Max pool (kernel 3 × 3 stride 2) Convolution 3 (3 × 3 × 512) Convolution 4 (5 × 5 × 512) Convolution 5 (5 × 5 × 512) Convolution 6 (7 × 7 × 256) Convolution 7 (11 × 11 × 128) Convolution 8 (13 × 13 × 1) Deconvolution 1 (8 × 8 × 1, stride 4) VGG-M Pan, Junting, Kevin McGuinness, Elisa Sayrol, Xavier Giro-i-Nieto, and Noel E. O'Connor. "Shallow and deep convolutional networks for saliency prediction." CVPR 2016. SalNet
- 32 SalNet ALGORITHMS CNNs DATA COMPUTATION
- 33 DATA COMPUTATION ALGORITHMS CNNs ALGORITHMS Adversarial SalGAN
- 34 SalGAN DATA COMPUTATION ALGORITHMS CNNs ALGORITHMS Adversarial
- 35Credit: Santiago Pascual [slides] [video]
- 36 Goodfellow, Ian, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. "Generative adversarial nets." NIPS 2014 Interview to Ian Goodfellow at Re-Work Deep Learning Summit San Francisco, 2018. Adversarial Training
- 37 Junting Pan, Cristian Canton, Kevin McGuinness, Noel E. O’Connor, Jordi Torres, Elisa Sayrol and Xavier Giro-i-Nieto. “SalGAN: Visual Saliency Prediction with Generative Adversarial Networks.” CVPRW 2017. Generator SalGAN Discriminator
- Can we predict WHERE & WHEN humans will look?
- Challenge: Create a model able to predict visual scanpaths
- 40 SalTiNet ALGORITHMS CNNs DATA COMPUTATION
- 41 SalTiNet ALGORITHMS CNNs DATA COMPUTATION
- 42 Yashas Rai, Jesús Gutiérrez, and Patrick Le Callet. 2017. “A Dataset of Head and Eye Movements for 360 Degree Images“. MMSys 2017
- 43 Yashas Rai, Jesús Gutiérrez, and Patrick Le Callet. 2017. “A Dataset of Head and Eye Movements for 360 Degree Images“. MMSys 2017
- 44 SalTiNet ALGORITHMS CNNs DATA COMPUTATION
- 45 Create a model able to predict visual scanpaths SalTiNet solution: Add a TIME dimension to saliency maps with saliency volumes and sample from them. Challenge:
- 46 Marc Assens, Kevin McGuinness, Xavier Giro-i-Nieto and Noel E. O’Connor. “SaltiNet: Scan-Path Prediction on 360 Degree Images Using Saliency Volumes.” ICCVW 2017. width height timeSaliency Volume
- 47 00000000000000000 00000000000000000 00000000000000000 00000000000000000 00000000000000000 00000000000000000 00000000000000000 width height time Marc Assens, Kevin McGuinness, Xavier Giro-i-Nieto and Noel E. O’Connor. “SaltiNet: Scan-Path Prediction on 360 Degree Images Using Saliency Volumes.” ICCVW 2017. Saliency Volume
- 48 00000000000000000 00000000000000000 00010000000010000 00000000000000000 00000000000000000 00000000010000000 00000000000000000 Fixation points : (x, y, t) width height time Marc Assens, Kevin McGuinness, Xavier Giro-i-Nieto and Noel E. O’Connor. “SaltiNet: Scan-Path Prediction on 360 Degree Images Using Saliency Volumes.” ICCVW 2017. Saliency Volume
- 49 00000000000000000 00000000000000000 00010000000010000 00000000000000000 00000000000000000 00000000010000000 00000000000000000 Convolution Fixation points : (x, y, t) width height timeSaliency Volume Marc Assens, Kevin McGuinness, Xavier Giro-i-Nieto and Noel E. O’Connor. “SaltiNet: Scan-Path Prediction on 360 Degree Images Using Saliency Volumes.” ICCVW 2017.
- 50 Predicted saliency volume SalTiNet Marc Assens, Kevin McGuinness, Xavier Giro-i-Nieto and Noel E. O’Connor. “SaltiNet: Scan-Path Prediction on 360 Degree Images Using Saliency Volumes.” ICCVW 2017.
- 51 Stochastic sampling SalTiNet Marc Assens, Kevin McGuinness, Xavier Giro-i-Nieto and Noel E. O’Connor. “SaltiNet: Scan-Path Prediction on 360 Degree Images Using Saliency Volumes.” ICCVW 2017.
- 52 Winners of IEEE ICME 2017 Challenge Marc Assens, Kevin McGuinness, Xavier Giro-i-Nieto and Noel E. O’Connor. “SaltiNet: Scan-Path Prediction on 360 Degree Images Using Saliency Volumes.” ICCVW 2017.
- 53 SalTiNet ALGORITHMS CNNs DATA COMPUTATION
- 54 PathGAN DATA COMPUTATION ALGORITHMS CNN & RNN ALGORITHMS Adversarial
- 55 PathGAN DATA COMPUTATION ALGORITHMS CNN & RNN ALGORITHMS Adversarial
- 56 Create a model able to predict visual scanpaths PathGAN solution: Generate a sequence of fixation points with recurrent neural networks (RNNs). Challenge:
- 57 Recurrent Neural Network (RNN) Hochreiter, Sepp, and Jürgen Schmidhuber. "Long short-term memory." Neural computation 9, no. 8 (1997): 1735-1780.
- 58 Marc Assens, Kevin McGuinness, Xavier Giro-i-Nieto and Noel E. O’Connor. “PathGan: Visual Scan-path Prediction with Generative Adversarial Networks“ arXiv 2018. PathGAN
- 59 Marc Assens, Kevin McGuinness, Xavier Giro-i-Nieto and Noel E. O’Connor. “PathGan: Visual Scan-path Prediction with Generative Adversarial Networks“ arXiv 2018. PathGAN
- 60 PathGAN DATA COMPUTATION ALGORITHMS CNN & RNN ALGORITHMS Adversarial
- SalNetJuntingNet SalGAN SaltiNet PathGAN Saliency maps Scanpaths
- 2016 20172015 Convolutional Neural Networks (CNNs) Recurrent Networks Transfer Learning from ImageNet Adversarial Training 2018 Scanpath Prediction
- ● MSc course (2017) ● BSc course (2018) 63 Deep Learning courses UPC Barcelona ● 1st edition (2016) ● 2nd edition (2017) ● 3rd edition (2018) ● 1st edition (2017) ● 2nd edition (2018) Next edition Autumn 2018 Next edition Winter/Spring 2019Summer School (late June 2018)
- @DocXavi Xavier Giro-i-Nieto Slides available at: http://bit.ly/reworkRETAIL We are looking for both industrial & academic partners: xavier.giro@upc.edu #reworkRETAIL