Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Visual Memorability for Egocentric Cameras

428 views

Published on

https://imatge.upc.edu/web/publications/visual-memorability-egocentric-cameras

This project explores visual memorability of egocentric in different ways having three main contributions. The first and the main contribution of the project is a new tool visual memorability in egocentric images. This tool that consists in a web application that allows the annotation of the visual memorability associated to still images with an online game. The second contribution of this work is a convolutional neural network model for visual memorability prediction that adapts an off-the-shelf model to egocentric images. Moreover, a visualization study has been pursued to localize the regions of the images that are more memorable than others. With this maps a comparison with saliency maps and is explored. This part of the research opens a new branch in visual memorability that consists in use memorability maps for saliency prediction. Also the memorability of the images is related with a sentiment analysis applying a model that predicts that feature. The final contribution is related to join visual memorability of images with human behaviour and physical state, finding a relation between memory and some physiological signals as: heart rate, galvanic skin response and electroencephalographic signals.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Visual Memorability for Egocentric Cameras

  1. 1. Visual Memorability for Egocentric Cameras Marc Carné Herrera Advisors: Xavier Giró-i-Nieto and Cathal Gurrin
  2. 2. Acknowledgements 2 Petia Radeva Maite Garolera Albert Gil Josep Pujal
  3. 3. Outline ➔ Introduction ➔ Contributions ◆ Annotation tool for visual memorability ◆ EgoMemNet: visual memorability adaptation to egocentric images ◆ Visual memorability and physiological signals ➔ Conclusions 3
  4. 4. Outline ➔ Introduction ➔ Contributions ◆ Annotation tool for visual memorability ◆ EgoMemNet: visual memorability adaptation to egocentric images ◆ Visual memorability and physiological signals ➔ Conclusions 4
  5. 5. “Brain is designed to forget in order to survive” ● Lifelogger → person that captures his daily life in order to create a virtual and digital memory. ● Wearable cameras → capture first person vision. ● Big data → 1.400 - 2.000 images/day. ● Challenge → retrieval! 5 Introduction
  6. 6. “Brain is designed to forget in order to survive” ● Lifelogger → person that captures his daily life in order to create a virtual and digital memory. ● Wearable cameras → capture first person vision. ● Big data → 1.400 - 2.000 images/day. ● Challenge → retrieval! 6 Introduction What we want to remember?
  7. 7. ● Cognitive therapy → Alzheimer patients, reminiscence therapy. 7 Motivation
  8. 8. 8 Image set Relevant images with low level feature from a CNN Relevant images with object detection, faces detction… (based on content)
  9. 9. Visual memorability 9 [Isola, CVPR 2011]
  10. 10. Visual memorability 10 [Isola, CVPR 2011] More memorable Less memorable
  11. 11. Domain adaptation 11 Human-taken Egocentric [Khosla, ICCV 2015]
  12. 12. Outline ➔ Introduction ➔ Contributions ◆ Annotation tool for visual memorability ◆ EgoMemNet: visual memorability adaptation to egocentric images ◆ Visual memorability and physiological signals ➔ Conclusions 12
  13. 13. Why an annotation tool? 13 Convolutional Neural Network Input (image) Output (label) Image + label to train the model
  14. 14. ● Inspired by MIT research work [1] ● Visual memory game: ○ Simple task → press ‘d’ when a repeated image is found ○ Duration: 9 minutes ○ Output: text file with detections 14 [1] Understanding and Predicting Image Memorability at a Large Scale, A. Khosla, A. S. Raju, A. Torralba and A. Oliva. ICCV 2015 Annotation tool for visual memorability UTEgocentric Insight Center for Data Analytics
  15. 15. Annotation 15 [Khosla, ICCV 2015]
  16. 16. Annotation tool 16
  17. 17. ● Docker: ○ Container with an operating system and software required. ○ Always run the same in any environment. ● Simple implementation → dockerfile 17 Annotation tool implementation Why to use a Docker? First docker implementation in GPI for research
  18. 18. 18 ● Memorability score → [0,1] ● Result: ○ Dataset → 50 annotated images (25 users) Annotation tool results
  19. 19. Outline ➔ Introduction ➔ Contributions ◆ Annotation tool for visual memorability ◆ EgoMemNet: visual memorability adaptation to egocentric images ◆ Visual memorability and physiological signals ➔ Contributions 19
  20. 20. Convolutional neural network: definition ● Automatic learning paradigm based by how human brain works ● Neuron interconnection that work together to generate an output stimulus or activation 20
  21. 21. 21 Convolutional neural network: layers Convolutional layer Fully connected layer
  22. 22. ● MemNet → CNN for memorability prediction ○ 5 conv layers + 2 fully connected layers + linear regression 22 EgoMemNet: visual memorability adaptation to egocentric images MemNet CNN [Koshla, ICCV 2015] 1 Structure: AlexNet
  23. 23. CNN fine-tuning 23 MemNet [Koshla, ICCV 2015]Insight dataset (egocentric dataset) EgoMemNet 1
  24. 24. ● No augmentation ● Spatial data augmentation → common method ● Temporal data augmentation → egocentric feature 24 Data augmentation strategies Spatial data augmentation Temporal data augmentation
  25. 25. 25 Quantitative results Spearman’s rank correlation Compute the similarity between positions between two different ranked lists. Memorability rank Ground truth rank
  26. 26. 26 Quantitative results Spearman’s rank correlation
  27. 27. 27 Qualitatives results
  28. 28. 28 Memorability maps ● Heat maps that highlight most memorable regions. ● Methods: ○ Grid-and-forward → obtain a memorability score per patch ○ EgoMemNet → fully convolutional version Grid-and-forward EgoMemNet
  29. 29. Memorability maps: grid-and-forward pass 29
  30. 30. Memorability maps: EgoMemNet 30 [Zhou, CVPR 2016]
  31. 31. 31 Memorability vs. saliency maps Original image Saliency map (SalNet CNN) [Pan, CVPR 2016] Memorability map (EgoMemNet* CNN) In green, parts shared between saliency and memorability maps. In blue, memorability regions non-salient. In red, salient regions non-memorability Binarized maps with learned threshold
  32. 32. Outline ➔ Introduction ➔ Contributions ◆ Annotation tool for visual memorability ◆ EgoMemNet: visual memorability adaptation to egocentric images ◆ Visual memorability and physiological signals ➔ Conclusions 32
  33. 33. The Insight dataset ● Multimodal homemade dataset: ○ Images ○ Memorability score ○ Heart rate value (during image acquisition) ○ Galvanic skin response (during image acquisition) 33 Publicly available!
  34. 34. Heart rate correlation 34 Memory scores quantized in 8 bins Mean heart rate in the bin
  35. 35. Galvanic skin response 35 Memory scores quantized in 8 bins Mean GSR in the bin
  36. 36. Physiological signals for memorability prediction 36 SNAP !
  37. 37. Detect snap points ● Prior approach → efficient capture without image processing 37 Linear Regression
  38. 38. Adding physiological signals for memorability prediction ● Post approach 38 Linear Regression EgoMemNet score
  39. 39. New feature: EEG signals ● EEG → electroencephalographic signals ● Hands free visual memory game 39
  40. 40. EEG data extraction 40 Peak at 400 ms P3@Pz → average 350-600 ms
  41. 41. 41 P3@PzPeak 400ms
  42. 42. Outline ➔ Introduction ➔ Contributions ◆ Annotation tool for visual memorability ◆ EgoMemNet: visual memorability adaptation to egocentric images ◆ Visual memorability and physiological signals ➔ Conclusions 42
  43. 43. Conclusions ● New annotation tool allows to create novel dataset for egocentric memorability. ● Egocentric (first person vision) dataset containing 50 annotated images. ● EgoMemNet, a model adapted for memorability prediction to egocentric images, presents a perform over MemNet, a convolutional neural network model trained with human-taken images. ● Physiological signals for memorability prediction. 43
  44. 44. Extended abstract Carné-Herrera M, Giró-i-Nieto X, Gurrin C. EgoMemNet: Visual Memorability Adaptation to Egocentric Images. Las Vegas, NV, USA: 4th Workshop on Egocentric (First-Person) Vision, CVPR 2016; 44
  45. 45. Spotlight Full spotlight in youtube! https://www.youtube.com/watch?v=qwM5NNW37YE
  46. 46. Poster presentation 464646
  47. 47. 47 Open Research Dataset Model Annotation tool http://imatge-upc.github.io/memory-2016-fpv/
  48. 48. 48 Hope this presentation has been memorable! Thanks for your attention!

×