Search

Mechanisms of bottom-up and top-down processing in visual perception

Loading...

Flash Player 9 (or above) is needed to view presentations.
We have detected that you do not have it on your computer. To install it, go here.

0 comments

Post a comment

    Post a comment
    Embed Video
    Edit your comment Cancel

    Notes on slide 1

    Thank you very much Charles for inviting me. I am delighted to be here and enjoying a weather that we could never hope for in the Spring in Boston...

    Here is the problem I am trying to solve: You give me an image and I tell you for instance whether or not it contains an animal.
    Object recognition is a very hard computational problem. The reason for that is that despite the fact that all of these are images of a giraffe, they look quite different at the pixel level. Objects in the real-world and these animal images in particular can vary drastically in terms of their appearance, shape, texture.
    In particular, changes in position and scale can create very large changes in the pattern of activity that they elicit on the retina... Think about that: even just a small shift in position of 2 deg of visual angle corresponds to shifting of the image on the retina of more than 120 photoreceptors!
    This is an extremely difficult task and today, no artificial computer vision system can do this task as robustly and accurately as the primate visual system.
    However as primates we are extremely good at solving this task despite all these variations...

    A classical paradigm that has been extensively used to study object recognition and visual perception is what I would call the rapid recognition paradigms.
    Here I am flashing images in rapid succession. This paradigm is called RSVP and was introduced by Molly Potter in the 70’s. Images are being presented at a rate of 7/s. At this speed you probably don’t get every details in the image but at the very least you are able to build a coarse description of the scene. For instance most of you should be able to recognize and perhaps memorize objects in these images...
    While these types of task do not necessarily reflect natural everyday vision when the visual world moves continuously and you are free to move your eyes and shift your attention. However they are able to isolate the first 100-150 ms of visual processing during which a base representation for images is being formed before more complex visual routines can come into play...

    A classical paradigm that has been extensively used to study object recognition and visual perception is what I would call the rapid recognition paradigms.
    Here I am flashing images in rapid succession. This paradigm is called RSVP and was introduced by Molly Potter in the 70’s. Images are being presented at a rate of 7/s. At this speed you probably don’t get every details in the image but at the very least you are able to build a coarse description of the scene. For instance most of you should be able to recognize and perhaps memorize objects in these images...
    While these types of task do not necessarily reflect natural everyday vision when the visual world moves continuously and you are free to move your eyes and shift your attention. However they are able to isolate the first 100-150 ms of visual processing during which a base representation for images is being formed before more complex visual routines can come into play...

    A classical paradigm that has been extensively used to study object recognition and visual perception is what I would call the rapid recognition paradigms.
    Here I am flashing images in rapid succession. This paradigm is called RSVP and was introduced by Molly Potter in the 70’s. Images are being presented at a rate of 7/s. At this speed you probably don’t get every details in the image but at the very least you are able to build a coarse description of the scene. For instance most of you should be able to recognize and perhaps memorize objects in these images...
    While these types of task do not necessarily reflect natural everyday vision when the visual world moves continuously and you are free to move your eyes and shift your attention. However they are able to isolate the first 100-150 ms of visual processing during which a base representation for images is being formed before more complex visual routines can come into play...

    In this talk I will argue that this base representation corresponds to the activation of a hierarchy of image fragments following a single feedforward sweep through the visual system. This bottom-up feedforward sweep rapidly activates specific sub-population of neurons in the ventral stream of the visual cortex that are tuned to image fragments with different levels of selectivity and invariance.

    I will show you that consistent with human psychophysics, a key limitation of this architecture is that it is susceptible to clutter. While it does relatively well on images that contains a single object and limited clutter (like the ones I just showed you), we found that the performance decreases significantly with increased amount of clutter.


    In this talk I will argue that this base representation corresponds to the activation of a hierarchy of image fragments following a single feedforward sweep through the visual system. This bottom-up feedforward sweep rapidly activates specific sub-population of neurons in the ventral stream of the visual cortex that are tuned to image fragments with different levels of selectivity and invariance.

    I will show you that consistent with human psychophysics, a key limitation of this architecture is that it is susceptible to clutter. While it does relatively well on images that contains a single object and limited clutter (like the ones I just showed you), we found that the performance decreases significantly with increased amount of clutter.


    In the second part of my talk I will argue that the way the visual system solves this clutter problem is via cortical feedback and shifts of attention. I will outline an integrated model of object recognition and attention. I will show that the object recognition performance of the model increases significantly when used in conjunction with attentional mechanisms. Using eye movements as a proxy for attention, I will show that the resulting model can account for a significant fraction of human eye movements during search tasks in complex natural images.










    We have implemented a computational model (shown on the right) that implement these sets of principles.

    Van Essen on the left. We do not try to account for the whole visual cortex, only the ventral stream of the visual cortex...

    The model is hierarchical with only feedforward connections.

    We have implemented a computational model (shown on the right) that implement these sets of principles.

    Van Essen on the left. We do not try to account for the whole visual cortex, only the ventral stream of the visual cortex...

    The model is hierarchical with only feedforward connections.

    We have implemented a computational model (shown on the right) that implement these sets of principles.

    Van Essen on the left. We do not try to account for the whole visual cortex, only the ventral stream of the visual cortex...

    The model is hierarchical with only feedforward connections.

    Computational considerations suggest that you need two types of operations and therefore functional classes of cells to explain those data.

    By analogy with H&B hierarchical model of processing in the visual cortex, we have called these two classes of cells simple and complex. The scheme that I am going to describe essentially extend their proposal from striate to extra-striate visual areas.

    We have assumed that these two types of functional units implement two types of computations or mathematical operations: Gaussian-like or bell-shape tuning and a max-like operation.

    The gaussian-bell tuning was motivated by a learning algorithm called Radial Basis Function while the max operation was motivated by the standard scanning approach in computer vision and theoretical arguments from signal processing.

    The goal of the simple units is to increase the complexity of the representation. Here on this example by pooling together the activity of afferent units with different orientations via this Gaussian-like tuning. This Gaussian tuning is ubiquitous in the visual cortex from orientation tuning in V1 to tuning for complex objects around certain poses in IT.

    The complex units pool together afferent units with the same preferred stimuli eg vertical bar but slightly different positions and scales. At the complex unit level we thus build some tolerance with respect to the exact position and scale of the stimulus within the receptive field of the unit.

    We have implemented a computational model (shown on the right) that implement these sets of principles.

    Van Essen on the left. We do not try to account for the whole visual cortex, only the ventral stream of the visual cortex...

    The model is hierarchical with only feedforward connections.

    EMPHASIZE AFTER TRAINING: NO DATA FITTING
    MENTION CHARLES

    It builds a simple-to-complex cells hierarchies.

    Mimic as closely as possible the tuning properties of neurons in various areas of the ventral stream
    Builds on earlier work in the lab by Max Riesenhuber



    -- I would argue that a key aspect of this model is the learning of a large dictionary of reusable features (I would call them shape components) from V1 to IT. These features represent a basic vocabulary of shape components that can be used to represent any visual input. These features correspond to patches of images which appear with high probability in the natural world. We argue that learning of this dictionary is done UNSUPERVISED during a developmental period.

    -- In this model, the goal of the ventral stream of the visual cortex from V1 to IT is to build a good representation for images, i.e. a representation which is compact and invariant with respect to 2D transformations such as translation and scale.

    -- With a good image representation, learning a new image category is relatively easy. We speculate that this can be done from a handful of labeling examples by training task-specific circuits running from IT to the PFC.

    We showed that it worked well on multiple object categories on standard computer vision databases.

    -- I would argue that a key aspect of this model is the learning of a large dictionary of reusable features (I would call them shape components) from V1 to IT. These features represent a basic vocabulary of shape components that can be used to represent any visual input. These features correspond to patches of images which appear with high probability in the natural world. We argue that learning of this dictionary is done UNSUPERVISED during a developmental period.

    -- In this model, the goal of the ventral stream of the visual cortex from V1 to IT is to build a good representation for images, i.e. a representation which is compact and invariant with respect to 2D transformations such as translation and scale.

    -- With a good image representation, learning a new image category is relatively easy. We speculate that this can be done from a handful of labeling examples by training task-specific circuits running from IT to the PFC.

    We showed that it worked well on multiple object categories on standard computer vision databases.

    -- I would argue that a key aspect of this model is the learning of a large dictionary of reusable features (I would call them shape components) from V1 to IT. These features represent a basic vocabulary of shape components that can be used to represent any visual input. These features correspond to patches of images which appear with high probability in the natural world. We argue that learning of this dictionary is done UNSUPERVISED during a developmental period.

    -- In this model, the goal of the ventral stream of the visual cortex from V1 to IT is to build a good representation for images, i.e. a representation which is compact and invariant with respect to 2D transformations such as translation and scale.

    -- With a good image representation, learning a new image category is relatively easy. We speculate that this can be done from a handful of labeling examples by training task-specific circuits running from IT to the PFC.

    We showed that it worked well on multiple object categories on standard computer vision databases.

    -- I would argue that a key aspect of this model is the learning of a large dictionary of reusable features (I would call them shape components) from V1 to IT. These features represent a basic vocabulary of shape components that can be used to represent any visual input. These features correspond to patches of images which appear with high probability in the natural world. We argue that learning of this dictionary is done UNSUPERVISED during a developmental period.

    -- In this model, the goal of the ventral stream of the visual cortex from V1 to IT is to build a good representation for images, i.e. a representation which is compact and invariant with respect to 2D transformations such as translation and scale.

    -- With a good image representation, learning a new image category is relatively easy. We speculate that this can be done from a handful of labeling examples by training task-specific circuits running from IT to the PFC.

    We showed that it worked well on multiple object categories on standard computer vision databases.

    -- I would argue that a key aspect of this model is the learning of a large dictionary of reusable features (I would call them shape components) from V1 to IT. These features represent a basic vocabulary of shape components that can be used to represent any visual input. These features correspond to patches of images which appear with high probability in the natural world. We argue that learning of this dictionary is done UNSUPERVISED during a developmental period.

    -- In this model, the goal of the ventral stream of the visual cortex from V1 to IT is to build a good representation for images, i.e. a representation which is compact and invariant with respect to 2D transformations such as translation and scale.

    -- With a good image representation, learning a new image category is relatively easy. We speculate that this can be done from a handful of labeling examples by training task-specific circuits running from IT to the PFC.

    We showed that it worked well on multiple object categories on standard computer vision databases.

    -- I would argue that a key aspect of this model is the learning of a large dictionary of reusable features (I would call them shape components) from V1 to IT. These features represent a basic vocabulary of shape components that can be used to represent any visual input. These features correspond to patches of images which appear with high probability in the natural world. We argue that learning of this dictionary is done UNSUPERVISED during a developmental period.

    -- In this model, the goal of the ventral stream of the visual cortex from V1 to IT is to build a good representation for images, i.e. a representation which is compact and invariant with respect to 2D transformations such as translation and scale.

    -- With a good image representation, learning a new image category is relatively easy. We speculate that this can be done from a handful of labeling examples by training task-specific circuits running from IT to the PFC.

    We showed that it worked well on multiple object categories on standard computer vision databases.

    -- I would argue that a key aspect of this model is the learning of a large dictionary of reusable features (I would call them shape components) from V1 to IT. These features represent a basic vocabulary of shape components that can be used to represent any visual input. These features correspond to patches of images which appear with high probability in the natural world. We argue that learning of this dictionary is done UNSUPERVISED during a developmental period.

    -- In this model, the goal of the ventral stream of the visual cortex from V1 to IT is to build a good representation for images, i.e. a representation which is compact and invariant with respect to 2D transformations such as translation and scale.

    -- With a good image representation, learning a new image category is relatively easy. We speculate that this can be done from a handful of labeling examples by training task-specific circuits running from IT to the PFC.

    We showed that it worked well on multiple object categories on standard computer vision databases.

    for the sake of time I am only going to show you that you can simulate a neurophysiology experiment with this model. You can record from population of random neurons and perform the same exact analysis as in a real experiment. On the bar plot shown here we performed the same exact readout experiment as in the study by Hung et al. What is shown here the classification performance when training in a specific position and scale and evaluating the generalization capability of the classifier to positions and scales not presented during training. This measures the built-in invariance inherited from the response properties of population of neurons and you can see that the fit is quite good.

    In parallel we have used this model in real-world computer vision applications. For instance we have developed a computer vision system for the automatic parsing of street scene images. Here are examples of automatic parsing by the system overlaid over the original images. The colors and bounding boxes indicate predictions from the model (eg green for trees etc).


    The computer vision system shown here is based exclusively on the response properties

    More recently we have extended the approach for the recognition of human actions such as running, walking, jogging, jumping, waving etc...

    In all cases we have shown that the resulting biologically motivated computer vision systems were performing on par or better than state-of-the-art computer vision systems.


    The goal of the model was not to explain natural every day vision when you are free to move your eyes and shift your attention but rather was is often called rapid recognition or immediate recognition which corresponds to the first 100-150 ms of visual processing (when an image is briefly presented) ie when the visual system is forced to operate in a feedforward mode before eye movements and shifts of attention.

    Here is an example on the left. Here I flash an image for a couple of ms, you probably don’t have time to get every fine details of this image but most people are able to say whether they contain an animal or not.

    Here we had divided our dataset in 4 subcategories: head... overall both the model and human do about 80% on this very difficult task and you can see that they agree quite well in turns of how they perform for these 4 subcategories...

    The goal of the model was not to explain natural every day vision when you are free to move your eyes and shift your attention but rather was is often called rapid recognition or immediate recognition which corresponds to the first 100-150 ms of visual processing (when an image is briefly presented) ie when the visual system is forced to operate in a feedforward mode before eye movements and shifts of attention.

    Here is an example on the left. Here I flash an image for a couple of ms, you probably don’t have time to get every fine details of this image but most people are able to say whether they contain an animal or not.

    Here we had divided our dataset in 4 subcategories: head... overall both the model and human do about 80% on this very difficult task and you can see that they agree quite well in turns of how they perform for these 4 subcategories...







    We have seen that in the model and in the visual cortex, when two stimuli fall within the receptive field of a neuron, the two stimuli “compete”, that is they reduce the selectivity of the neurons. I just showed you that at the psychophysical level, the amount of clutter in an image largely determines the performance of the model and of human observers during rapid categorization tasks.

    We have seen that in the model and in the visual cortex, when two stimuli fall within the receptive field of a neuron, the two stimuli “compete”, that is they reduce the selectivity of the neurons. I just showed you that at the psychophysical level, the amount of clutter in an image largely determines the performance of the model and of human observers during rapid categorization tasks.

    We have seen that in the model and in the visual cortex, when two stimuli fall within the receptive field of a neuron, the two stimuli “compete”, that is they reduce the selectivity of the neurons. I just showed you that at the psychophysical level, the amount of clutter in an image largely determines the performance of the model and of human observers during rapid categorization tasks.

    We have seen that in the model and in the visual cortex, when two stimuli fall within the receptive field of a neuron, the two stimuli “compete”, that is they reduce the selectivity of the neurons. I just showed you that at the psychophysical level, the amount of clutter in an image largely determines the performance of the model and of human observers during rapid categorization tasks.

    We have seen that in the model and in the visual cortex, when two stimuli fall within the receptive field of a neuron, the two stimuli “compete”, that is they reduce the selectivity of the neurons. I just showed you that at the psychophysical level, the amount of clutter in an image largely determines the performance of the model and of human observers during rapid categorization tasks.



    Using eye movements as correlate of attention. Assumption is that attention gets to an item just before eye moves so if eyes move we an assume that just before that attention was there

    Here is the original model: we added back-projections to account for these attentional modulations
    we assume that feature-based attention acts through a cascade of top-down connections though the ventral stream originating in the PFC where a template of the target object is held in memory all the way down to V4 and possibly lower areas. We also assume a spatial attention modulation originating from the parietal cortex (here I am assuming LIP based on limited experimental evidence).

    This attentional mechanisms can be casted in a probabilistic Bayesian framework whereby the parietal cortex represents Location variables, the ventral stream represents feature variables. These are our image fragments. Variables for the target object are encoded in higher areas such as PFC...
    This framework is inspired by an earlier model by Rao to explain spatial attention and is a special case of the computational model of the visual cortex described by David Mumford and that probably most of you know...


    Here is the original model: we added back-projections to account for these attentional modulations
    we assume that feature-based attention acts through a cascade of top-down connections though the ventral stream originating in the PFC where a template of the target object is held in memory all the way down to V4 and possibly lower areas. We also assume a spatial attention modulation originating from the parietal cortex (here I am assuming LIP based on limited experimental evidence).

    This attentional mechanisms can be casted in a probabilistic Bayesian framework whereby the parietal cortex represents Location variables, the ventral stream represents feature variables. These are our image fragments. Variables for the target object are encoded in higher areas such as PFC...
    This framework is inspired by an earlier model by Rao to explain spatial attention and is a special case of the computational model of the visual cortex described by David Mumford and that probably most of you know...


    Here is the original model: we added back-projections to account for these attentional modulations
    we assume that feature-based attention acts through a cascade of top-down connections though the ventral stream originating in the PFC where a template of the target object is held in memory all the way down to V4 and possibly lower areas. We also assume a spatial attention modulation originating from the parietal cortex (here I am assuming LIP based on limited experimental evidence).

    This attentional mechanisms can be casted in a probabilistic Bayesian framework whereby the parietal cortex represents Location variables, the ventral stream represents feature variables. These are our image fragments. Variables for the target object are encoded in higher areas such as PFC...
    This framework is inspired by an earlier model by Rao to explain spatial attention and is a special case of the computational model of the visual cortex described by David Mumford and that probably most of you know...



    here the way we implemented that is via belief propagation in polytrees (here the messages are shown for the simplified case of a single feature for clarity).
    Within framework, spatial attention can be described as a series of msgs from L to Fil to Fi to O while feature-based attention goes the opposite way.
    Thus the model makes specific predictions about the timing of visual areas in the ventral stream and the parietal cortex depending on the task at end.

    Obviously I am leaving a lot of details open unfortunately...


    We have implemented the approach in the context of our animal search
    model mostly improves on medium and far conditions

    We have implemented the approach in the context of our animal search
    model mostly improves on medium and far conditions

    We have implemented the approach in the context of our animal search
    model mostly improves on medium and far conditions

    We have implemented the approach in the context of our animal search
    model mostly improves on medium and far conditions











    here the way we implemented that is via belief propagation in polytrees (here the messages are shown for the simplified case of a single feature for clarity).
    Within framework, spatial attention can be described as a series of msgs from L to Fil to Fi to O while feature-based attention goes the opposite way.
    Thus the model makes specific predictions about the timing of visual areas in the ventral stream and the parietal cortex depending on the task at end.

    Obviously I am leaving a lot of details open unfortunately...







    Unlike artificial search arrays were arbitrary objects are simply randomly placed on a display, natural scenes are highly structured.
    This is a point that has been made by Antonio Torralba and Aude Oliva and the fact that global features could provide a good representation of the gist of the scene which is sufficient to associate contextual information from the visual scene to actual object locations like here for instance where you would expect people to be most in these darker regions...









    2 Favorites

    Mechanisms of bottom-up and top-down processing in visual perception - Presentation Transcript

    1. Mechanisms of bottom-up and top-down processing in visual perception Thomas Serre McGovern Institute for Brain Research Department of Brain & Cognitive Sciences Massachusetts Institute of Technology
    2. The problem: recognition in natural scenes
    3. Rapid recognition: human behavior Potter 1971, 1975 see also Biederman 1972; Thorpe 1996 movie courtesy of Jim DiCarlo
    4. Rapid recognition: human behavior Potter 1971, 1975 see also Biederman 1972; Thorpe 1996 movie courtesy of Jim DiCarlo
    5. Rapid recognition: human behavior Gist of the scene at 7 images/s from unpredictable random sequence of images No time for eye movements No top-down / expectations Potter 1971, 1975 see also Biederman 1972; Thorpe 1996 movie courtesy of Jim DiCarlo
    6. Rapid recognition: human behavior Gist of the scene at 7 images/s from unpredictable random sequence of images No time for eye movements No top-down / expectations Feedforward processing: Coarse / base image representation Potter 1971, 1975 see also Biederman 1972; Thorpe 1996 movie courtesy of Jim DiCarlo
    7. Outline 1.Rapid recognition and feedforward processing: Loose hierarchy of image fragments “Clutter problem”
    8. Outline 1.Rapid recognition and feedforward processing: Loose hierarchy of image fragments “Clutter problem”
    9. Outline 1.Rapid recognition and feedforward processing: Loose hierarchy of image fragments “Clutter problem”
    10. Outline 1.Rapid recognition and feedforward processing: Loose hierarchy of image fragments “Clutter problem” 2.Beyond feedforward processing: X X Top-down cortical feedback and attention to solve the “clutter problem” XX Predicting human eye movements
    11. Outline 1.Rapid recognition and feedforward processing: Loose hierarchy of image fragments “Clutter problem” 2.Beyond feedforward processing: Top-down cortical feedback and attention to solve the “clutter problem” Predicting human eye movements
    12. Object recognition in the visual cortex source: Jim DiCarlo
    13. Object recognition in the visual cortex Ventral visual stream source: Jim DiCarlo
    14. Object recognition in the visual cortex Hierarchical architecture: Ventral visual stream source: Jim DiCarlo
    15. Object recognition in the visual cortex Hierarchical architecture: Latencies Ventral visual stream source: Jim DiCarlo
    16. Object recognition in the visual cortex Hierarchical architecture: Latencies Ventral visual stream Anatomy source: Jim DiCarlo
    17. Object recognition in the visual cortex Hierarchical architecture: Latencies Ventral visual stream Anatomy Function source: Jim DiCarlo
    18. Object recognition in the visual cortex Nobel prize 1981 Hubel & Wiesel 1959, 1962, 1965, 1968
    19. Object recognition in the visual cortex gradual increase in complexity of preferred stimulus Kobatake & Tanaka 1994 see also Oram & Perrett 1993; Sheinberg & Logothetis 1996; Gallant et al 1996; Riesenhuber & Poggio 1999
    20. Object recognition in the visual cortex Parallel increase in invariance properties (position and scale) of neurons Kobatake & Tanaka 1994 see also Oram & Perrett 1993; Sheinberg & Logothetis 1996; Gallant et al 1996; Riesenhuber & Poggio 1999
    21. Model RF sizes Num. layers units Animal vs. Prefrontal 11, task-dependent learning classification 8 46 45 12 10 0 non-animal Cortex 13 units Supervised Increase in complexity (number of subunits), RF size and invariance PG V2,V3,V4,MT,MST LIP,VIP,DP,7a V1 AIT,36,35 PIT, AIT TE 2 o S4 7 10 STP Rostral STS } 36 35 TG o 10 3 C3 7 TPO PGa IPa TEa TEm PG Cortex task-independent learning AIT o 10 3 7 C2b Unsupervised o o 10 4 1.2 - 3.2 S3 PIT VIP LIP 7a PP MSTcMSTp DP FST o o TF 10 7 0.9 - 4.4 S2b o o 10 5 1.1 - 3.0 C2 o o 10 7 0.6 - 2.4 V4 PO V3A MT S2 o o 10 4 0.4 - 1.6 V3 C1 V2 o 0.2o- 1.1 10 6 V1 S1 dorsal stream ventral stream 'where' pathway 'what' pathway Simple cells Complex cells Main routes Tuning Serre Kouh Cadieu Knoblich Kreiman & Poggio 2005 MAX Bypass routes
    22. Model RF sizes Num. layers units Animal vs. Prefrontal 11, task-dependent learning classification 8 46 45 12 10 0 non-animal Cortex 13 Large-scale (108 units Supervised Increase in complexity (number of subunits), RF size and invariance PG V2,V3,V4,MT,MST units), spans LIP,VIP,DP,7a V1 AIT,36,35 PIT, AIT several areas of TE 2 o S4 7 10 the visual cortex STP Rostral STS } 36 35 TG o 10 3 C3 7 TPO PGa IPa TEa TEm PG Cortex task-independent learning AIT o 10 3 7 C2b Unsupervised o o 10 4 1.2 - 3.2 S3 PIT VIP LIP 7a PP MSTcMSTp DP FST o o TF 10 7 0.9 - 4.4 S2b o o 10 5 1.1 - 3.0 C2 o o 10 7 0.6 - 2.4 V4 PO V3A MT S2 o o 10 4 0.4 - 1.6 V3 C1 V2 o 0.2o- 1.1 10 6 V1 S1 dorsal stream ventral stream 'where' pathway 'what' pathway Simple cells Complex cells Main routes Tuning Serre Kouh Cadieu Knoblich Kreiman & Poggio 2005 MAX Bypass routes
    23. Model RF sizes Num. layers units Animal vs. Prefrontal 11, task-dependent learning classification 8 46 45 12 10 0 non-animal Cortex 13 Large-scale (108 units Supervised Increase in complexity (number of subunits), RF size and invariance PG V2,V3,V4,MT,MST units), spans LIP,VIP,DP,7a V1 AIT,36,35 PIT, AIT several areas of TE 2 o S4 7 10 the visual cortex STP Rostral STS } 36 35 TG o 10 3 C3 7 TPO PGa IPa TEa TEm PG Cortex task-independent learning Combination of AIT o 3 7 10 C2b Unsupervised forward 10 and o o 4 1.2 - 3.2 S3 reverse PIT VIP LIP 7a PP MSTcMSTp DP FST o o TF 7 0.9 - 4.4 10 engineering S2b o o 10 5 1.1 - 3.0 C2 o o 10 7 0.6 - 2.4 V4 PO V3A MT S2 o o 10 4 0.4 - 1.6 V3 C1 V2 o 0.2o- 1.1 10 6 V1 S1 dorsal stream ventral stream 'where' pathway 'what' pathway Simple cells Complex cells Main routes Tuning Serre Kouh Cadieu Knoblich Kreiman & Poggio 2005 MAX Bypass routes
    24. Model RF sizes Num. layers units Animal vs. Prefrontal 11, task-dependent learning classification 8 46 45 12 10 0 non-animal Cortex 13 Large-scale (108 units Supervised Increase in complexity (number of subunits), RF size and invariance PG V2,V3,V4,MT,MST units), spans LIP,VIP,DP,7a V1 AIT,36,35 PIT, AIT several areas of TE 2 o S4 7 10 the visual cortex STP Rostral STS } 36 35 TG o 10 3 C3 7 TPO PGa IPa TEa TEm PG Cortex task-independent learning Combination of AIT o 3 7 10 C2b Unsupervised forward 10 and o o 4 1.2 - 3.2 S3 reverse PIT VIP LIP 7a PP MSTcMSTp DP FST o o TF 7 0.9 - 4.4 10 engineering S2b o o 10 5 1.1 - 3.0 C2 Shown to be o o 7 0.6 - 2.4 10 V4 PO V3A MT S2 consistent with o o 4 0.4 - 1.6 10 V3 C1 V2 many1.1 10 experimental o o 6 0.2 - V1 data across areas S1 of visual cortex dorsal stream ventral stream 'where' pathway 'what' pathway (V1, V2, V4, MT and IT) Simple cells Complex cells Main routes Tuning Serre Kouh Cadieu Knoblich Kreiman & Poggio 2005 MAX Bypass routes
    25. Two functional classes of cells Simple cells Complex cells Invariance Template matching max-like operation Gaussian-like tuning ~”OR” ~ “AND” Riesenhuber & Poggio 1999 (building on Fukushima 1980 and Hubel & Wiesel 1962)
    26. Model RF sizes Num. layers units Animal vs. Prefrontal 11, task-dependent learning classification 8 46 45 12 10 0 non-animal Cortex 13 units Supervised Increase in complexity (number of subunits), RF size and invariance PG V2,V3,V4,MT,MST LIP,VIP,DP,7a V1 AIT,36,35 PIT, AIT TE 2 o S4 7 10 STP Rostral STS } 36 35 TG o 10 3 C3 7 TPO PGa IPa TEa TEm PG Cortex task-independent learning AIT o 10 3 7 C2b Unsupervised o o 10 4 1.2 - 3.2 S3 PIT VIP LIP 7a PP MSTcMSTp DP FST o o TF 10 7 0.9 - 4.4 S2b o o 10 5 1.1 - 3.0 C2 o o 10 7 0.6 - 2.4 V4 PO V3A MT S2 o o 10 4 0.4 - 1.6 V3 C1 V2 o 0.2o- 1.1 10 6 V1 S1 dorsal stream ventral stream 'where' pathway 'what' pathway Simple cells Complex cells Main routes Tuning Serre Kouh Cadieu Knoblich Kreiman & Poggio 2005 MAX Bypass routes
    27. Hierarchy of image fragments see also Ullman et al 2002
    28. Hierarchy of image fragments Unsupervised learning of frequent image fragments during development see also Ullman et al 2002
    29. Hierarchy of image fragments Unsupervised learning of frequent image fragments during development Reusable fragments shared across categories see also Ullman et al 2002
    30. Hierarchy of image fragments Unsupervised learning of frequent image fragments during development Reusable fragments shared across categories Large redundant vocabulary for implicit geometry see also Ullman et al 2002
    31. Hierarchy of image fragments Unsupervised learning of frequent image fragments IT during development Reusable fragments shared across categories Large redundant vocabulary for implicit geometry V1 see also Ullman et al 2002
    32. Hierarchy of image fragments Unsupervised learning of frequent image fragments IT during development Reusable fragments shared across categories Large redundant vocabulary for implicit geometry V1 see also Ullman et al 2002
    33. Hierarchy of image fragments Unsupervised learning of frequent image fragments IT during development Reusable fragments shared across categories Large redundant vocabulary for implicit geometry V1 see also Ullman et al 2002
    34. Hierarchy of image fragments category selective units linear perceptron Unsupervised learning of frequent image fragments IT during development Reusable fragments shared across categories Large redundant vocabulary for implicit geometry V1 see also Ullman et al 2002
    35. Model vs. IT 1 IT Model 0.8 Classification performance 0.6 0.4 0.2 0 Size: 3.4o 3.4o 1.7o 6.8o 3.4o 3.4o center 2ohorz. 4ohorz. Position: center center center TRAIN Model data: Serre Kouh Cadieu Knoblich Kreiman & Poggio 2005 Experimental data: Hung* Kreiman* Poggio & DiCarlo 2005
    36. Is this model sufficient to explain performance in rapid categorization tasks? Image Interval Image-Mask Mask 1/f noise 20 ms 30 ms ISI 80 ms Animal present or not ? Thorpe et al 1996; Van Rullen & Koch 2003; Bacon-Mace et al 2005
    37. Rapid categorization Serre Oliva & Poggio 2007
    38. Rapid categorization Head Close-body Medium-body Far-body Animals Natural distractors Artificial distractors Serre Oliva & Poggio 2007
    39. Rapid categorization Serre Oliva & Poggio 2007
    40. Rapid categorization Head Close-body Medium-body Far-body Animals Natural distractors Serre Oliva & Poggio 2007
    41. Rapid categorization 2.6 2.4 Performance (d') 1.8 1.4 Model (82% correct) Human observers (80% correct) 1.0 Head Close-body Medium-body Far-body Head Close- Medium- Far- body body body Animals Natural distractors Serre Oliva & Poggio 2007
    42. “Clutter effect” Limitation of feedforward model compatible with reduced selectivity in V4 (Reynolds et al 1999) and IT in the presence of clutter (Zoccolan et al 2005, 2007; Rolls et al 2003) Meyers Freiwald Embark Kreiman Serre Poggio in prep
    43. “Clutter effect” Recording site in monkey’s IT Limitation of feedforward model compatible with reduced selectivity in V4 Model (Reynolds et al 1999) and IT in the presence of clutter IT neurons (Zoccolan et al 2005, 2007; Rolls et al 2003) fMRI Meyers Freiwald Embark Kreiman Serre Poggio in prep
    44. Summary I Rapid categorization seems compatible with model based on feedforward hierarchy of image fragments Consistent with psychophysics, key limitation of architecture is recognition in clutter How does the visual system overcome such limitation?
    45. Outline 1.Rapid recognition and feedforward processing: Loose hierarchy of image fragments “Clutter problem” 2.Beyond feedforward processing: X X Top-down cortical feedback and attention to solve the “clutter problem” XX Predicting human eye movements
    46. Spatial attention solves the “clutter problem” see also Broadbent 1952 1954; Treisman 1960; Treisman & Gelade 1980; Duncan & Desimone 1995; Wolfe, 1997; and many others
    47. Spatial attention solves the “clutter problem” see also Broadbent 1952 1954; Treisman 1960; Treisman & Gelade 1980; Duncan & Desimone 1995; Wolfe, 1997; and many others foreground
    48. Spatial attention solves the “clutter problem” see also Broadbent 1952 1954; Treisman 1960; Treisman & Gelade 1980; Duncan & Desimone 1995; Wolfe, 1997; and many others background foreground
    49. Spatial attention solves the “clutter problem” see also Broadbent 1952 1954; Treisman 1960; Treisman & Gelade 1980; Duncan & Desimone 1995; Wolfe, 1997; and many others background foreground X X XX
    50. Spatial attention solves the “clutter problem” see also Broadbent 1952 1954; Treisman 1960; Treisman & Gelade 1980; Duncan & Desimone 1995; Wolfe, 1997; and many others background foreground X X XX Problem: How to know where to attend?
    51. Spatial attention solves X X XX the “clutter problem” see also Broadbent 1952 1954; Treisman 1960; Treisman & Gelade 1980; Duncan & Desimone 1995; Wolfe, 1997; and many others Science 22 April 2005: Vol. 308. no. 5721, pp. 529 - 534 Parallel and Serial Neural Mechanisms for Visual Search in Macaque Area V4 Narcisse P. Bichot, Andrew F. Rossi, Robert Desimone
    52. Spatial attention solves X X XX the “clutter problem” see also Broadbent 1952 1954; Treisman 1960; Treisman & Gelade 1980; Duncan & Desimone 1995; Wolfe, 1997; and many others Science 22 April 2005: Vol. 308. no. 5721, pp. 529 - 534 Parallel and Serial Neural Mechanisms for Visual Search in Macaque Area V4 Narcisse P. Bichot, Andrew F. Rossi, Robert Desimone Answer: Parallel feature-based attention
    53. Parallel feature-based X X XX attention modulation normalized spike activity 2 1 0 0 100 200 0 100 200 time from fixation (ms)
    54. Serial spatial attention X X XX modulation Test for serial (spatial) selection 2 attend within RF normalized spike activity 1 FIX attend away from RF RF 0 0 100 200 RF stimulus is SACCADE: target of saccade ruary 18, 2009 time from fixation (ms) vs. RF stimulus is not SACCADE: target of saccade Fig. 4. Illustration of the saccade enhancement analysis. We compared neuronal measures when the monkey made a saccade to an RF stimulus versus a saccade away from the RF. In this dis-
    55. Attention as Bayesian inference PFC IT V4/PIT V2 Chikkerur Serre & Poggio in prep see also Rao 2005; Lee & Mumford 2003
    56. Attention as Bayesian inference PFC feature-based attention IT V4/PIT V2 Chikkerur Serre & Poggio in prep see also Rao 2005; Lee & Mumford 2003
    57. Attention as Bayesian inference PFC feature-based attention IT FEF/LIP V4/PIT spatial attention V2 Chikkerur Serre & Poggio in prep see also Rao 2005; Lee & Mumford 2003
    58. Attention as Bayesian inference O PFC feature-based object priors attention Fi IT L FEF/LIP Fli V4/PIT location priors spatial attention N I V2 Chikkerur Serre & Poggio in prep see also Rao 2005; Lee & Mumford 2003
    59. Attention as Bayesian inference PFC O LIP IT Fi L V4 Fli N V2 I Chikkerur Serre & Poggio in prep
    60. Attention as Bayesian inference feature-based PFC O attention belief propagation: FEF/LIP = P (L) mLIP →V 4 IT Fi = P (F i |O) mIT →V 4 = P (Fli |F, L)P (L)P (I|Fli ) mV 4→IT L L Fli = P (Fli |F, L)P (F i |O)P (I|Fli ) mV 4→LIP V4 Fli Fi Fli N Where is at object O? V2 I Chikkerur Serre & Poggio in prep see also Rao 2005; Lee & Mumford 2003
    61. Attention as Bayesian inference spatial attention PFC O belief propagation: FEF/LIP = P (L) mLIP →V 4 IT Fi = P (F i |O) mIT →V 4 = P (Fli |F, L)P (L)P (I|Fli ) mV 4→IT L L Fli = P (Fli |F, L)P (F i |O)P (I|Fli ) mV 4→LIP V4 Fli Fi Fli N What is at location L? V2 I Chikkerur Serre & Poggio in prep see also Rao 2005; Lee & Mumford 2003
    62. Model performance improves with attention performance (d’) one shift of no attention attention Model Humans Chikkerur Serre & Poggio in prep
    63. Model performance improves with attention 3 performance (d’) 2 1 0 one shift of no attention attention Model Humans Chikkerur Serre & Poggio in prep
    64. Model performance improves with attention 3 performance (d’) 2 1 0 one shift of no attention attention Model Humans Chikkerur Serre & Poggio in prep
    65. Model performance improves with attention 3 performance (d’) 2 1 0 one shift of no attention attention Model Humans Chikkerur Serre & Poggio in prep
    66. Model performance improves with attention mask no mask 3 performance (d’) 2 1 0 one shift of no attention attention Model Humans Chikkerur Serre & Poggio in prep
    67. Agreement with neurophysiology data Feature-based attention: Differential modulation for preferred vs. non-preferred stimulus (Bichot et al’ 05) Spatial attention: Gain modulation on neuron’s tuning curves (McAdams & Maunsell’99) Competitive mechanisms in V2 and V4 (Reynolds et al’ 99) Improved readout in clutter (being tested in collaboration with the Desimone lab)
    68. IT readout improves with attention train readout classifier on + isolated object Zhang Meyers Serre Bichot Desimone Poggio in prep
    69. IT readout improves with attention + Zhang Meyers Serre Bichot Desimone Poggio in prep
    70. IT readout improves with attention + Zhang Meyers Serre Bichot Desimone Poggio in prep
    71. IT readout improves with attention + Zhang Meyers Serre Bichot Desimone Poggio in prep
    72. IT readout improves with attention cue transient change 7 attention on object Average rank attention away 8 + from object object not shown 9 0 500 1000 1500 2000 Time (ms) n=34 Zhang Meyers Serre Bichot Desimone Poggio in prep
    73. IT readout improves with attention cue transient change 7 attention on object Average rank attention away 8 + from object object not shown 9 0 500 1000 1500 2000 Time (ms) n=34 Zhang Meyers Serre Bichot Desimone Poggio in prep
    74. Could these attentional mechanisms also explain search strategies in complex natural images?
    75. Matching human eye movements Dataset: 100 street-scenes images with cars & pedestrians and 20 without Experiment 8 participants asked to count the number of cars/pedestrians Blocks/randomized presentations Each image presented twice Eye movements recorded using an infra-red eye tracker Eye movements as proxy for attention Chikkerur Tan Serre & Poggio in sub
    76. Matching human eye movements Car search Pedestrian search Chikkerur Tan Serre & Poggio in sub
    77. Matching human eye movements Car search Pedestrian search Chikkerur Tan Serre & Poggio in sub
    78. Attention as Bayesian inference PFC O FEF/LIP IT Fi L V4 Fli N V2 I Chikkerur Serre & Poggio in prep
    79. Matching human eye movements
    80. Matching human eye 100% movements fraction fixations 75% 50% 25% 10% 20% 30% % image covered by saliency maps
    81. Matching human eye 100% area movements fraction fixations 75% under 50% ROC 25% curve 10% 20% 30% % image covered by saliency maps
    82. Results ROC area Humans Bottom-up Top-down (feature-based) Chikkerur Tan Serre & Poggio in sub
    83. Results 1 ROC area 0.75 0.5 0.25 0 car pedestrian Humans Bottom-up Top-down (feature-based) Chikkerur Tan Serre & Poggio in sub
    84. Results 1 ROC area 0.75 0.5 0.25 0 car pedestrian Humans Bottom-up Top-down (feature-based) Chikkerur Tan Serre & Poggio in sub
    85. Results 1 ROC area 0.75 0.5 0.25 0 car pedestrian Humans Bottom-up Top-down (feature-based) Chikkerur Tan Serre & Poggio in sub
    86. Results 1 ROC area 0.75 0.5 0.25 0 car pedestrian Humans Bottom-up Top-down (feature-based) Chikkerur Tan Serre & Poggio in sub
    87. Figure 1. The structure of objects and their backgrounds. In this illustration, each image has been created by averaging hundreds of pictures containing a particular object Local vs. global contextual in the center (a face, keyboard and fire hydrant) at a fixed scale and pose. Images come from the LabelMe dataset [65]. Before averaging, each picture is translated and scaled so that the target object is in the center. No other transformations are applied. The averages reveal the regularities existing in the intensity patterns across all the images. The background of many objects does not average to a uniform field, showing that an object extends its influence beyond its own boundaries, and this property is heavily used by the visual system. recent work, these rules have been integrated into a pairing was inappropriate (e.g. a kitchen counter and bass cues common framework of contextual influences [11,18–20], drum). In a recent study, Davenport and Potter [3] in which context provides a robust estimate of the prob- observed that consistency information influences percep- ability of an object’s presence, position and scale. tion of both the object and the scene background if a scene The most documented effect of context on object is presented briefly (80 ms), which suggests a recurrent recognition is the scene consistency–inconsistency effect processing framework, in which objects and their settings [1,5,7,8]. Palmer [7] found that observers’ accuracy at an influence each other mutually [21]. object-categorization task was facilitated if the target (e.g. A natural way of representing the context of an object is a loaf of bread) was presented after an appropriate scene in terms of its relationship to other objects (Figure 3). (e.g. a kitchen counter) and impaired if the scene–object Learning statistical contingencies between objects can cause the perception of one object or scene to generate strong expectations about the probable presence and location of other objects (see Box 1 for a description of current theories about the mechanisms involved in con- textual inference). Chun and Jiang [22] showed that people can learn the contingencies between novel objects, predict- ing the presence of one object on the basis of another, over the course of only 30 min. Contextual interactions between objects can be sensitive to subtle visual aspects. Green and Hummel [23] found that mechanisms of object perception are sensitive to the relative pose of pairs of objects. In a priming design, observers were presented with a prime object (e.g. a pitcher) for 50 ms, followed by a target image (e.g. a glass) for 50 ms. Crucially, the accuracy of target recognition was significantly higher if the prime object was oriented to interact with the target object in a consistent manner (e.g. a pitcher facing a glass) than if the pair interacted in an inconsistent manner (e.g. a pitcher oriented away from a glass). In most of these studies, the participants did not learn any new contextual rules. All the experiments were designed to prove the effects of contextual rules previously learnt by the observers in the real world, and the tests were performed on realistic images or line drawings depicting real scenes. In the next section, we review recent work that Figure 2. The strength of context. The visual system makes assumptions regarding shows that human observers have a remarkable ability to object identities according to their size and location in the scene. In this picture, observers describe the scene as containing a car and pedestrian in the street. learn contextual associations. Observers do not need to be However, the pedestrian is in fact the same shape as the car, except for a 908 explicitly aware of contextual associations to benefit from rotation. The atypicality of this orientation for a car within the context defined by Torralba ’01; Torralba & Oliva ’02 ’03; Torralba ’03; them. the street scene causes the car to be recognized as a pedestrian. Torralba Oliva et al ’06; Oliva & Torralba ’06 ‘07 www.sciencedirect.com
    88. Figure 1. The structure of objects and their backgrounds. In this illustration, each image has been created by averaging hundreds of pictures containing a particular object Local vs. global contextual in the center (a face, keyboard and fire hydrant) at a fixed scale and pose. Images come from the LabelMe dataset [65]. Before averaging, each picture is translated and scaled so that the target object is in the center. No other transformations are applied. The averages reveal the regularities existing in the intensity patterns across all the images. The background of many objects does not average to a uniform field, showing that an object extends its influence beyond its own boundaries, and this property is heavily used by the visual system. recent work, these rules have been integrated into a pairing was inappropriate (e.g. a kitchen counter and bass cues common framework of contextual influences [11,18–20], drum). In a recent study, Davenport and Potter [3] in which context provides a robust estimate of the prob- observed that consistency information influences percep- ability of an object’s presence, position and scale. tion of both the object and the scene background if a scene The most documented effect of context on object is presented briefly (80 ms), which suggests a recurrent recognition is the scene consistency–inconsistency effect processing framework, in which objects and their settings [1,5,7,8]. Palmer [7] found that observers’ accuracy at an influence each other mutually [21]. object-categorization task was facilitated if the target (e.g. A natural way of representing the context of an object is a loaf of bread) was presented after an appropriate scene in terms of its relationship to other objects (Figure 3). (e.g. a kitchen counter) and impaired if the scene–object Learning statistical contingencies between objects can cause the perception of one object or scene to generate strong expectations about the probable presence and location of other objects (see Box 1 for a description of current theories about the mechanisms involved in con- textual inference). Chun and Jiang [22] showed that people can learn the contingencies between novel objects, predict- ing the presence of one object on the basis of another, over the course of only 30 min. Contextual interactions between objects can be sensitive to subtle visual aspects. Green and Hummel [23] found that mechanisms of object perception are sensitive to the relative pose of pairs of objects. In a priming design, observers were presented with a prime object (e.g. a pitcher) for 50 ms, followed by a target image (e.g. a glass) for 50 ms. Crucially, the accuracy of target recognition was significantly higher if the prime object was oriented to interact with the target object in a consistent manner (e.g. a pitcher facing a glass) than if the pair interacted in an inconsistent manner (e.g. a pitcher oriented away from a glass). In most of these studies, the participants did not learn any new contextual rules. All the experiments were designed to prove the effects of contextual rules previously learnt by the observers in the real world, and the tests were performed on realistic images or line drawings depicting real scenes. In the next section, we review recent work that Figure 2. The strength of context. The visual system makes assumptions regarding shows that human observers have a remarkable ability to object identities according to their size and location in the scene. In this picture, observers describe the scene as containing a car and pedestrian in the street. learn contextual associations. Observers do not need to be However, the pedestrian is in fact the same shape as the car, except for a 908 explicitly aware of contextual associations to benefit from rotation. The atypicality of this orientation for a car within the context defined by Torralba ’01; Torralba & Oliva ’02 ’03; Torralba ’03; them. the street scene causes the car to be recognized as a pedestrian. Torralba Oliva et al ’06; Oliva & Torralba ’06 ‘07 www.sciencedirect.com
    89. Local vs. global contextual cues O object priors Fi L Fli location priors N I Chikkerur Tan Serre & Poggio in sub
    90. Local vs. global contextual cues O object priors Fi L Fli location priors N I Chikkerur Tan Serre & Poggio in sub
    91. Local vs. global contextual cues O S object priors global scene priors Fi L Fli location priors N I Chikkerur Tan Serre & Poggio in sub
    92. Integrating (local) feature-based + (global) context-based cues Results accounts for 92% of inter- subject agreement! 1 ROC area 0.75 0.5 0.25 0 car pedestrian Humans Bottom-up Top-down (feature-based) Feaure-based + contextual cues similar (independent) results by Chikkerur Tan Serre & Poggio in sub Ehinger, Hidalgo Torralba & Oliva (in press)
    93. Summary II
    94. Summary II Attentional mechanisms may help solve the “clutter problem”
    95. Summary II Attentional mechanisms may help solve the “clutter problem” Model combining (local) feature-based attentional cues with (global) scene contextual priors for object locations accounts for significant amount of human eye movements during complex visual searches
    96. Acknowledgments Tomaso Poggio Aude Oliva (rapid categ. / psychophysics) Sharat Chikkerur (model of attention) Cheston Tan (eye tracking / psychophysics Ethan Meyers (attention neural data analysis) Other: Minjoon Kouh Gabriel Kreiman Narcisse Bichot Timothee Masquelier Stan Bileschi Leila Reddy Charles Cadieu David Sheinberg Robert Desimone Jed Singer Jim DiCarlo Andrew Steele Michelle Fabre-Thorpe Simon Thorpe Winrich Freiwald Nao Tsuchyia Estibaliz Garrote Lior Wolf Hueihan Jhuang Ying Zhang Ulf Knoblich Christof Koch

    + tserretserre, 6 months ago

    custom

    1779 views, 2 favs, 2 embeds more stats

    This is a talk given in April 2009 in the Redwood c more

    More info about this document

    © All Rights Reserved

    Go to text version

    • Total Views 1779
      • 1736 on SlideShare
      • 43 from embeds
    • Comments 0
    • Favorites 2
    • Downloads 34
    Most viewed embeds
    • 42 views on http://web.mit.edu
    • 1 views on http://static.slideshare.net

    more

    All embeds
    • 42 views on http://web.mit.edu
    • 1 views on http://static.slideshare.net

    less

    Flagged as inappropriate Flag as inappropriate
    Flag as inappropriate

    Select your reason for flagging this presentation as inappropriate. If needed, use the feedback form to let us know more details.

    Cancel
    File a copyright complaint
    Having problems? Go to our helpdesk?

    Categories

    Tags

    more
    -->
    Search
    RSS Feed
    What's new?

    © 2009 SlideShare Inc. All Rights Reserved