Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Modeling the Shape of the
     Scene: A Holistic
Representation of the Spatial
        Envelope

     Oliva. A., Torralba ...
Scene Recognition
 - What is a Scene?
 - S.R. Studies
Spatial Envelope
 - Holistic
     Representation
                   ...
Scene Recognition
 - What is a Scene?
 - S.R. Studies
Spatial Envelope
 - Holistic
     Representation
                   ...
Scene Recognition
 - What is a Scene?
 - S.R. Studies
Spatial Envelope
 - Holistic
     Representation
                   ...
Scene Recognition
 - What is a Scene?
 - S.R. Studies
Spatial Envelope
 - Holistic
     Representation
                   ...
Scene Recognition
 - What is a Scene?
 - S.R. Studies
Spatial Envelope
 - Holistic
     Representation
                   ...
Scene Recognition
 - What is a Scene?
 - S.R. Studies
Spatial Envelope
 - Holistic
     Representation
                   ...
Scene Recognition
 - What is a Scene?
 - S.R. Studies
Spatial Envelope
 - Holistic
     Representation
                   ...
Scene Recognition
 - What is a Scene?
 - S.R. Studies
Spatial Envelope           Modeling the Scene Structure
            ...
Scene Recognition
 - What is a Scene?
 - S.R. Studies
Spatial Envelope
 - Holistic
     Representation
                   ...
Scene Recognition
 - What is a Scene?
 - S.R. Studies
Spatial Envelope
                            Modeling the Scene Stru...
Scene Recognition
 - What is a Scene?
 - S.R. Studies
Spatial Envelope
                           Estimation of the Spatia...
Scene Recognition
 - What is a Scene?
 - S.R. Studies
Spatial Envelope
                        Estimation of the Spatial E...
Scene Recognition
 - What is a Scene?
 - S.R. Studies
Spatial Envelope
                           Spatial Envelope Propert...
Scene Recognition
 - What is a Scene?
 - S.R. Studies
Spatial Envelope
                         Spatial Envelope Propertie...
Scene Recognition
 - What is a Scene?
 - S.R. Studies
Spatial Envelope
                        Spatial Envelope Properties...
Scene Recognition
 - What is a Scene?
 - S.R. Studies
Spatial Envelope
                         Spatial Envelope Propertie...
Scene Recognition
 - What is a Scene?
 - S.R. Studies
Spatial Envelope
                        Spatial Envelope Properties...
Scene Recognition
 - What is a Scene?
 - S.R. Studies
Spatial Envelope
 - Holistic             Spatial Envelope Properties...
Scene Recognition
 - What is a Scene?
 - S.R. Studies
Spatial Envelope
 - Holistic             Spatial Envelope Properties...
Scene Recognition
 - What is a Scene?
 - S.R. Studies
Spatial Envelope
 - Holistic             Spatial Envelope Properties...
Scene Recognition
 - What is a Scene?
 - S.R. Studies
Spatial Envelope
 - Holistic             Conclusions / Important Poi...
Upcoming SlideShare
Loading in …5
×

Modeling the Shape of the Scene

1,724 views

Published on

Description of the main features of a scene which are extracted by most computational models. Comparison of two different computational models.

Published in: Education, Art & Photos, Business
  • Be the first to comment

  • Be the first to like this

Modeling the Shape of the Scene

  1. 1. Modeling the Shape of the Scene: A Holistic Representation of the Spatial Envelope Oliva. A., Torralba A. (2001). International Journal of Computer Vision 42(3), 145-175.
  2. 2. Scene Recognition - What is a Scene? - S.R. Studies Spatial Envelope - Holistic Representation Scene Recognition - Definition - Spatial Properties Scene Structure 2 different approaches: - DFT - WFT  Details  General Picture - Mean Spectrograms  General Picture  Details Sp.E. Properties - DST - WDST - Naturalness  Scene Recognition is… - Natural Scenes’ Properties a reconstruction of the properties of a 3D - Urban Scenes’ Properties Scene that are important for tasks such as - Correlations - Semantic movement or grasping Categories  the providing of information about the Conclusions semantic category and the function of the environment.
  3. 3. Scene Recognition - What is a Scene? - S.R. Studies Spatial Envelope - Holistic Representation Scene Recognition - Definition - Spatial Properties Scene Structure  Object  Scene - DFT - WFT 1 – 2 meters  >5meters - Mean Spectrograms  “Hand distance”  “Movement” Sp.E. Properties - DST - WDST - Naturalness - Natural Scenes’ Properties - Urban Scenes’ Properties - Correlations - Semantic Categories Conclusions
  4. 4. Scene Recognition - What is a Scene? - S.R. Studies Spatial Envelope - Holistic Representation Scene Recognition - Definition - Spatial Properties Scene Structure  “Gist” Recognition Speed: 200ms - DFT - WFT - Mean  View of a Scene for short time Spectrograms Sp.E. Properties - DST  Recognition of Functional and - WDST - Naturalness Categorical Properties - Natural Scenes’ Properties  Object information – ignored - Urban Scenes’ Properties - Correlations  Environmental pictures – rapid - Semantic Categories Conclusions categorization
  5. 5. Scene Recognition - What is a Scene? - S.R. Studies Spatial Envelope - Holistic Representation Spatial Envelope - Definition - Spatial Properties Scene Structure Scene is not just an arrangement of objects and - DFT - WFT it’s something more than the relationship - Mean between them. Spectrograms Sp.E. Properties - DST - WDST - Naturalness - Natural Scenes’ Properties - Urban Scenes’ Properties - Correlations - Semantic Categories Conclusions Scenes of the same category have a stable spatial structure.
  6. 6. Scene Recognition - What is a Scene? - S.R. Studies Spatial Envelope - Holistic Representation Spatial Envelope - Definition - Spatial Properties Scene Structure Spatial Envelope: boundaries that give a shape to the Scene. - DFT - WFT e.g. ground, horizon, elevations, slants - Mean Spectrograms Sp.E. Properties - DST - WDST - Naturalness - Natural Scenes’ Properties - Urban Scenes’ Properties - Correlations - Semantic Categories Conclusions
  7. 7. Scene Recognition - What is a Scene? - S.R. Studies Spatial Envelope - Holistic Representation Spatial Envelope - Definition - Spatial Properties Scene Structure 81 pictures - DFT - WFT Experiment A: - Mean Spectrograms 17 subjects - 81 pictures Sp.E. Properties Criteria: A B - DST 1)Global Aspect - WDST - Naturalness 2)Global Structure - Natural Scenes’ 3)Elements A1 A2 B1 B2 Properties - Urban Scenes’ Properties a b a b a b a b - Correlations - Semantic Categories 8 subgroups Conclusions
  8. 8. Scene Recognition - What is a Scene? - S.R. Studies Spatial Envelope - Holistic Representation Spatial Envelope - Definition 5 Main Properties - Spatial Properties Scene Structure 1. Naturalness - DFT sharp VS smooth edges - WFT - Mean Spectrograms 1. Openness Sp.E. Properties - DST No Horizon / Visual references - WDST VS Horizon / lack of Visual - Naturalness References - Natural Scenes’ Properties - Urban Scenes’ 1. Roughness Properties - Correlations Complexity VS No complexity - Semantic Categories Conclusions 1. Expansion Depth VS No depth 1. Ruggedness Deviation VS No deviation
  9. 9. Scene Recognition - What is a Scene? - S.R. Studies Spatial Envelope Modeling the Scene Structure Image-based representations - Holistic Representation - Definition Hanning - Spatial Properties Discrete Fourier Transform Window Scene Structure Intensity - DFT - WFT - Mean Spectrograms Sp.E. Properties - DST - WDST Amplitude Phase - Naturalness Spectrum Function - Natural Scenes’ Properties - Urban Scenes’ Properties Windowed Fourier Transform - Correlations - Semantic Categories Hamming Conclusions Intensity window with radius r
  10. 10. Scene Recognition - What is a Scene? - S.R. Studies Spatial Envelope - Holistic Representation Modeling the Scene Structure - Definition Energy Spectrum - Spatial Properties Scene Structure - DFT - WFT - Mean Spectrograms Sp.E. Properties - DST - WDST - Naturalness Energy Spectrum Principal Components of Energy Spectrum - Natural Scenes’ Properties - Urban Scenes’ Spectrogram Properties - Correlations - Semantic Categories Conclusions Spectrogram Principal Components of the Spectrogram
  11. 11. Scene Recognition - What is a Scene? - S.R. Studies Spatial Envelope Modeling the Scene Structure - Holistic Representation - Definition - Spatial Properties Scene Structure - DFT - WFT - Mean Spectrograms Sp.E. Properties Man-made open urban | Streets City Center Buildings close-up - DST Urban Str. - WDST - Naturalness - Natural Scenes’ Properties - Urban Scenes’ Properties - Correlations - Semantic Categories Natural open natural enclosed mountains enclosed forest natural, Conclusions non-textured
  12. 12. Scene Recognition - What is a Scene? - S.R. Studies Spatial Envelope Estimation of the Spatial Envelope - Holistic Representation Properties - Definition - Spatial Properties Scene Scene Structure attribute s - DFT - WFT - Mean Spectrograms Sp.E. Properties - DST - WDST - Naturalness - Natural Scenes’ Properties - Urban Scenes’ Properties - Correlations - Semantic Categories Conclusions Discriminant Spectral Template - DST
  13. 13. Scene Recognition - What is a Scene? - S.R. Studies Spatial Envelope Estimation of the Spatial Envelope - Holistic Representation Properties - Definition - Spatial Properties Scene Structure - DFT - WFT - Mean Spectrograms Sp.E. Properties - DST - WDST - Naturalness - Natural Scenes’ Properties - Urban Scenes’ Properties - Correlations - Semantic Categories Conclusions Windowed Discriminant Spectral Template - WDST
  14. 14. Scene Recognition - What is a Scene? - S.R. Studies Spatial Envelope Spatial Envelope Properties - Holistic Representation ~Naturalness~ - Definition - Spatial Properties Scene Structure 2000 images of natural and man-made scenes - DFT - WFT - Mean Spectrograms Sp.E. Properties - DST - WDST - Naturalness DST(fx,fy) WDST(fx,fy) - Natural Scenes’ Properties - Urban Scenes’ 93.5% Correct! Properties - Correlations - Semantic Categories Conclusions Low Naturalness High Naturalness
  15. 15. Scene Recognition - What is a Scene? - S.R. Studies Spatial Envelope Spatial Envelope Properties - Holistic Representation ~Natural Images~ - Definition - Spatial Properties 500 images of natural scenes Scene Structure Openness Ruggedness Roughness - DFT - WFT - Mean Spectrograms Sp.E. Properties DST(fx,fy) - DST - WDST - Naturalness - Natural Scenes’ Properties - Urban Scenes’ Properties - Correlations WDST(fx,fy) - Semantic Categories Conclusions DST: 93% DST: 89% DST: 92% WDST: 96,5% WDST: 91% WDST: 94%
  16. 16. Scene Recognition - What is a Scene? - S.R. Studies Spatial Envelope Spatial Envelope Properties - Holistic Representation ~Natural Images~ - Definition - Spatial Properties Scene Structure - DFT - WFT - Mean Spectrograms Sp.E. Properties - DST - WDST - Naturalness - Natural Scenes’ Properties - Urban Scenes’ Properties - Correlations - Semantic Categories Conclusions
  17. 17. Scene Recognition - What is a Scene? - S.R. Studies Spatial Envelope Spatial Envelope Properties - Holistic Representation ~Urban Environments~ - Definition - Spatial Properties 500 urban scenes Scene Structure Openness Expansion Roughness - DFT - WFT - Mean Spectrograms Sp.E. Properties DST(fx,fy) - DST - WDST - Naturalness - Natural Scenes’ Properties - Urban Scenes’ Properties - Correlations WDST(fx,fy) - Semantic Categories Conclusions DST: 94% DST: 90% DST: 91% WDST: 96% WDST: 94% WDST: 92%%
  18. 18. Scene Recognition - What is a Scene? - S.R. Studies Spatial Envelope Spatial Envelope Properties - Holistic Representation ~Urban Environments~ - Definition - Spatial Properties Scene Structure - DFT - WFT - Mean Spectrograms Sp.E. Properties - DST - WDST - Naturalness - Natural Scenes’ Properties - Urban Scenes’ Properties - Correlations - Semantic Categories Conclusions
  19. 19. Scene Recognition - What is a Scene? - S.R. Studies Spatial Envelope - Holistic Spatial Envelope Properties Representation - Definition - Spatial Properties Natural Scenes Scene Structure - DFT - WFT - Mean Spectrograms Sp.E. Properties - DST - WDST - Naturalness - Natural Scenes’ Properties - Urban Scenes’ Properties Urban Scenes - Correlations - Semantic Categories Conclusions
  20. 20. Scene Recognition - What is a Scene? - S.R. Studies Spatial Envelope - Holistic Spatial Envelope Properties Representation - Definition - Spatial Properties Scene Structure - DFT - WFT - Mean Spectrograms Sp.E. Properties - DST - WDST - Naturalness - Natural Scenes’ Properties - Urban Scenes’ Properties - Correlations - Semantic Categories Conclusions
  21. 21. Scene Recognition - What is a Scene? - S.R. Studies Spatial Envelope - Holistic Spatial Envelope Properties Representation - Definition - Spatial Properties Scene Structure - DFT - WFT - Mean Spectrograms Sp.E. Properties - DST - WDST - Naturalness - Natural Scenes’ Properties - Urban Scenes’ Properties - Correlations 400 “target” images - Semantic Categories XXX + XXXX Conclusions XXXOXOO + OOXOXOX XX+XX DST: 86% WDST: 92%
  22. 22. Scene Recognition - What is a Scene? - S.R. Studies Spatial Envelope - Holistic Conclusions / Important Points Representation - Definition - Spatial Properties Scene Structure •Introduction of the “Spatial Envelope” model - DFT •A set of important perceptual properties (Naturalness, openness, - WFT ruggedness, expansion, roughness) - Mean Spectrograms •Provides a holistic description of the scene Sp.E. Properties - DST •Spatial Envelope properties are strongly correlated with - WDST the templates of DST and WDST - Naturalness - Natural Scenes’ •The S.E. model organizes scenes as humans subjects do Properties - Urban Scenes’ •S.E. model is able to retrieve images sharing the same Properties semantic category - Correlations - Semantic •Object identification is not a necessary stage for Categories identification a scene Conclusions •S.E. model provides a meaningful representation of complex scenes

×