Successfully reported this slideshow.
Your SlideShare is downloading. ×

Modeling the Shape of the Scene

Ad

Modeling the Shape of the
     Scene: A Holistic
Representation of the Spatial
        Envelope

     Oliva. A., Torralba ...

Ad

Scene Recognition
 - What is a Scene?
 - S.R. Studies
Spatial Envelope
 - Holistic
     Representation
                   ...

Ad

Scene Recognition
 - What is a Scene?
 - S.R. Studies
Spatial Envelope
 - Holistic
     Representation
                   ...

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Check these out next

1 of 22 Ad
1 of 22 Ad
Advertisement

More Related Content

More from Dimitris Agorastos

Advertisement

Modeling the Shape of the Scene

  1. 1. Modeling the Shape of the Scene: A Holistic Representation of the Spatial Envelope Oliva. A., Torralba A. (2001). International Journal of Computer Vision 42(3), 145-175.
  2. 2. Scene Recognition - What is a Scene? - S.R. Studies Spatial Envelope - Holistic Representation Scene Recognition - Definition - Spatial Properties Scene Structure 2 different approaches: - DFT - WFT  Details  General Picture - Mean Spectrograms  General Picture  Details Sp.E. Properties - DST - WDST - Naturalness  Scene Recognition is… - Natural Scenes’ Properties a reconstruction of the properties of a 3D - Urban Scenes’ Properties Scene that are important for tasks such as - Correlations - Semantic movement or grasping Categories  the providing of information about the Conclusions semantic category and the function of the environment.
  3. 3. Scene Recognition - What is a Scene? - S.R. Studies Spatial Envelope - Holistic Representation Scene Recognition - Definition - Spatial Properties Scene Structure  Object  Scene - DFT - WFT 1 – 2 meters  >5meters - Mean Spectrograms  “Hand distance”  “Movement” Sp.E. Properties - DST - WDST - Naturalness - Natural Scenes’ Properties - Urban Scenes’ Properties - Correlations - Semantic Categories Conclusions
  4. 4. Scene Recognition - What is a Scene? - S.R. Studies Spatial Envelope - Holistic Representation Scene Recognition - Definition - Spatial Properties Scene Structure  “Gist” Recognition Speed: 200ms - DFT - WFT - Mean  View of a Scene for short time Spectrograms Sp.E. Properties - DST  Recognition of Functional and - WDST - Naturalness Categorical Properties - Natural Scenes’ Properties  Object information – ignored - Urban Scenes’ Properties - Correlations  Environmental pictures – rapid - Semantic Categories Conclusions categorization
  5. 5. Scene Recognition - What is a Scene? - S.R. Studies Spatial Envelope - Holistic Representation Spatial Envelope - Definition - Spatial Properties Scene Structure Scene is not just an arrangement of objects and - DFT - WFT it’s something more than the relationship - Mean between them. Spectrograms Sp.E. Properties - DST - WDST - Naturalness - Natural Scenes’ Properties - Urban Scenes’ Properties - Correlations - Semantic Categories Conclusions Scenes of the same category have a stable spatial structure.
  6. 6. Scene Recognition - What is a Scene? - S.R. Studies Spatial Envelope - Holistic Representation Spatial Envelope - Definition - Spatial Properties Scene Structure Spatial Envelope: boundaries that give a shape to the Scene. - DFT - WFT e.g. ground, horizon, elevations, slants - Mean Spectrograms Sp.E. Properties - DST - WDST - Naturalness - Natural Scenes’ Properties - Urban Scenes’ Properties - Correlations - Semantic Categories Conclusions
  7. 7. Scene Recognition - What is a Scene? - S.R. Studies Spatial Envelope - Holistic Representation Spatial Envelope - Definition - Spatial Properties Scene Structure 81 pictures - DFT - WFT Experiment A: - Mean Spectrograms 17 subjects - 81 pictures Sp.E. Properties Criteria: A B - DST 1)Global Aspect - WDST - Naturalness 2)Global Structure - Natural Scenes’ 3)Elements A1 A2 B1 B2 Properties - Urban Scenes’ Properties a b a b a b a b - Correlations - Semantic Categories 8 subgroups Conclusions
  8. 8. Scene Recognition - What is a Scene? - S.R. Studies Spatial Envelope - Holistic Representation Spatial Envelope - Definition 5 Main Properties - Spatial Properties Scene Structure 1. Naturalness - DFT sharp VS smooth edges - WFT - Mean Spectrograms 1. Openness Sp.E. Properties - DST No Horizon / Visual references - WDST VS Horizon / lack of Visual - Naturalness References - Natural Scenes’ Properties - Urban Scenes’ 1. Roughness Properties - Correlations Complexity VS No complexity - Semantic Categories Conclusions 1. Expansion Depth VS No depth 1. Ruggedness Deviation VS No deviation
  9. 9. Scene Recognition - What is a Scene? - S.R. Studies Spatial Envelope Modeling the Scene Structure Image-based representations - Holistic Representation - Definition Hanning - Spatial Properties Discrete Fourier Transform Window Scene Structure Intensity - DFT - WFT - Mean Spectrograms Sp.E. Properties - DST - WDST Amplitude Phase - Naturalness Spectrum Function - Natural Scenes’ Properties - Urban Scenes’ Properties Windowed Fourier Transform - Correlations - Semantic Categories Hamming Conclusions Intensity window with radius r
  10. 10. Scene Recognition - What is a Scene? - S.R. Studies Spatial Envelope - Holistic Representation Modeling the Scene Structure - Definition Energy Spectrum - Spatial Properties Scene Structure - DFT - WFT - Mean Spectrograms Sp.E. Properties - DST - WDST - Naturalness Energy Spectrum Principal Components of Energy Spectrum - Natural Scenes’ Properties - Urban Scenes’ Spectrogram Properties - Correlations - Semantic Categories Conclusions Spectrogram Principal Components of the Spectrogram
  11. 11. Scene Recognition - What is a Scene? - S.R. Studies Spatial Envelope Modeling the Scene Structure - Holistic Representation - Definition - Spatial Properties Scene Structure - DFT - WFT - Mean Spectrograms Sp.E. Properties Man-made open urban | Streets City Center Buildings close-up - DST Urban Str. - WDST - Naturalness - Natural Scenes’ Properties - Urban Scenes’ Properties - Correlations - Semantic Categories Natural open natural enclosed mountains enclosed forest natural, Conclusions non-textured
  12. 12. Scene Recognition - What is a Scene? - S.R. Studies Spatial Envelope Estimation of the Spatial Envelope - Holistic Representation Properties - Definition - Spatial Properties Scene Scene Structure attribute s - DFT - WFT - Mean Spectrograms Sp.E. Properties - DST - WDST - Naturalness - Natural Scenes’ Properties - Urban Scenes’ Properties - Correlations - Semantic Categories Conclusions Discriminant Spectral Template - DST
  13. 13. Scene Recognition - What is a Scene? - S.R. Studies Spatial Envelope Estimation of the Spatial Envelope - Holistic Representation Properties - Definition - Spatial Properties Scene Structure - DFT - WFT - Mean Spectrograms Sp.E. Properties - DST - WDST - Naturalness - Natural Scenes’ Properties - Urban Scenes’ Properties - Correlations - Semantic Categories Conclusions Windowed Discriminant Spectral Template - WDST
  14. 14. Scene Recognition - What is a Scene? - S.R. Studies Spatial Envelope Spatial Envelope Properties - Holistic Representation ~Naturalness~ - Definition - Spatial Properties Scene Structure 2000 images of natural and man-made scenes - DFT - WFT - Mean Spectrograms Sp.E. Properties - DST - WDST - Naturalness DST(fx,fy) WDST(fx,fy) - Natural Scenes’ Properties - Urban Scenes’ 93.5% Correct! Properties - Correlations - Semantic Categories Conclusions Low Naturalness High Naturalness
  15. 15. Scene Recognition - What is a Scene? - S.R. Studies Spatial Envelope Spatial Envelope Properties - Holistic Representation ~Natural Images~ - Definition - Spatial Properties 500 images of natural scenes Scene Structure Openness Ruggedness Roughness - DFT - WFT - Mean Spectrograms Sp.E. Properties DST(fx,fy) - DST - WDST - Naturalness - Natural Scenes’ Properties - Urban Scenes’ Properties - Correlations WDST(fx,fy) - Semantic Categories Conclusions DST: 93% DST: 89% DST: 92% WDST: 96,5% WDST: 91% WDST: 94%
  16. 16. Scene Recognition - What is a Scene? - S.R. Studies Spatial Envelope Spatial Envelope Properties - Holistic Representation ~Natural Images~ - Definition - Spatial Properties Scene Structure - DFT - WFT - Mean Spectrograms Sp.E. Properties - DST - WDST - Naturalness - Natural Scenes’ Properties - Urban Scenes’ Properties - Correlations - Semantic Categories Conclusions
  17. 17. Scene Recognition - What is a Scene? - S.R. Studies Spatial Envelope Spatial Envelope Properties - Holistic Representation ~Urban Environments~ - Definition - Spatial Properties 500 urban scenes Scene Structure Openness Expansion Roughness - DFT - WFT - Mean Spectrograms Sp.E. Properties DST(fx,fy) - DST - WDST - Naturalness - Natural Scenes’ Properties - Urban Scenes’ Properties - Correlations WDST(fx,fy) - Semantic Categories Conclusions DST: 94% DST: 90% DST: 91% WDST: 96% WDST: 94% WDST: 92%%
  18. 18. Scene Recognition - What is a Scene? - S.R. Studies Spatial Envelope Spatial Envelope Properties - Holistic Representation ~Urban Environments~ - Definition - Spatial Properties Scene Structure - DFT - WFT - Mean Spectrograms Sp.E. Properties - DST - WDST - Naturalness - Natural Scenes’ Properties - Urban Scenes’ Properties - Correlations - Semantic Categories Conclusions
  19. 19. Scene Recognition - What is a Scene? - S.R. Studies Spatial Envelope - Holistic Spatial Envelope Properties Representation - Definition - Spatial Properties Natural Scenes Scene Structure - DFT - WFT - Mean Spectrograms Sp.E. Properties - DST - WDST - Naturalness - Natural Scenes’ Properties - Urban Scenes’ Properties Urban Scenes - Correlations - Semantic Categories Conclusions
  20. 20. Scene Recognition - What is a Scene? - S.R. Studies Spatial Envelope - Holistic Spatial Envelope Properties Representation - Definition - Spatial Properties Scene Structure - DFT - WFT - Mean Spectrograms Sp.E. Properties - DST - WDST - Naturalness - Natural Scenes’ Properties - Urban Scenes’ Properties - Correlations - Semantic Categories Conclusions
  21. 21. Scene Recognition - What is a Scene? - S.R. Studies Spatial Envelope - Holistic Spatial Envelope Properties Representation - Definition - Spatial Properties Scene Structure - DFT - WFT - Mean Spectrograms Sp.E. Properties - DST - WDST - Naturalness - Natural Scenes’ Properties - Urban Scenes’ Properties - Correlations 400 “target” images - Semantic Categories XXX + XXXX Conclusions XXXOXOO + OOXOXOX XX+XX DST: 86% WDST: 92%
  22. 22. Scene Recognition - What is a Scene? - S.R. Studies Spatial Envelope - Holistic Conclusions / Important Points Representation - Definition - Spatial Properties Scene Structure •Introduction of the “Spatial Envelope” model - DFT •A set of important perceptual properties (Naturalness, openness, - WFT ruggedness, expansion, roughness) - Mean Spectrograms •Provides a holistic description of the scene Sp.E. Properties - DST •Spatial Envelope properties are strongly correlated with - WDST the templates of DST and WDST - Naturalness - Natural Scenes’ •The S.E. model organizes scenes as humans subjects do Properties - Urban Scenes’ •S.E. model is able to retrieve images sharing the same Properties semantic category - Correlations - Semantic •Object identification is not a necessary stage for Categories identification a scene Conclusions •S.E. model provides a meaningful representation of complex scenes

×