This document summarizes a talk on how cortical grid cells may represent the structure of objects. [1] Cortical grid cells in the entorhinal cortex and within cortical columns can represent locations within environments and on objects. As an animal or sensor moves, grid cell modules update locations using path integration. [2] A network model shows how grid cell representations of location can be used to build predictive models of objects by associating locations with sensory features. Simulations demonstrate the network can recognize objects through sequences of movements and sensations. [3] There is suggestive anatomical and physiological evidence mapping the proposed model to biology, including grid cell signatures in human cortex and sensorimotor prediction in primary sensory areas.

1. Workshop on Integrative Theories
of Cortical Function
July 18, 2018
Subutai Ahmad
sahmad@numenta.com
@SubutaiAhmad
Locations in the Neocortex:
A Theory of Sensorimotor Prediction Using
Cortical Grid Cells
Collaborators:
-Jeff Hawkins, Marcus Lewis,
Scott Purdy, Mirko Klukas

2. Observation:
The neocortex is constantly predicting its inputs.
“the most important and also the most neglected problem
of cerebral physiology” (Lashley, 1951)
How can networks of neurons learn predictive models of
the world and objects through movement?
Question:
The neocortex uses “cortical grid cells” and path
integration to model an object’s structure.
Proposal:

3. 1) Properties of grid cells
3) Mapping to biology
- Suggestive biological evidence
- Cortical columns revisited
- Model predictions
- Encoding location using grid cells
- Path integration and location spaces
- Sensory cues and recalling locations
- Learns predictive models of objects using locations and sensory cues
- Simulation results
2) A network model of sensorimotor prediction using cortical grid cells

4. Single Grid Cell
(Moser & Moser, 2013)

5. Grid Cell Module Contains Multiple Cells That Are Offset

6. Grid Cell Module Contains Multiple Cells That Are Offset

7. Multiple Modules At Different Scales and Orientations
Environment

8. Multiple Modules Can Represent Locations Uniquely
(Fiete et al, 2008; Sreenivasan and Fiete, 2011)

9. Path Integration
(Hafting et al, 2005;
McNaughton et al., 2006; Ocko et al., 2018)
- As animal moves, grid cells update their location
- This can happen in the dark, using efference copy of motion signals
- Path integration: regardless of path trajectory, the same location in
environment will activate a consistent grid code
- Imprecise, so sensory cues are used to “anchor” grid cells

10. Unique Location Spaces For Each Environment
- Each module randomly initialized in a new room, codes for each room will be unique
(e.g. 20 modules, 100 cells each = 10020 possible codes)
- Initial point implicitly defines a location space for each environment
- Modules update independently, so path integration still works
- Sensory cues can be used to anchor or re-activate the location space
(Rowland & Moser, 2014)

11. Summary: Grid Cells Represent the Structure of Environments
Entorhinal Cortex
Body in environments
Location
- Encoded by grid cells
- Unique to location in room AND room
- Location is updated by movement
A Room is:
- A set of locations that are connected
by movement (via path integration).
- Some locations have associated
features.

12. Proposal: Cortical Grid Cells Represent the Structure of Objects
Location
- Encoded by grid cells
- Unique to location in room AND room
- Location is updated by movement
A Room is:
- A set of locations that are connected
by movement (via path integration).
- Some locations have associated
features.
Entorhinal Cortex
Body in environments
Cortical Column
Sensor patch relative to objects
Location
- Encoded by grid-like cells
- Unique to location on object AND object
- Location is updated by movement
An Object is:
- A set of locations that are connected
by movement (via path integration).
- Some locations have associated
features.

13. 1) Properties of grid cells
3) Mapping to biology
- Suggestive biological evidence
- Cortical columns revisited
- Model predictions
- Encoding location using grid cells
- Path integration and location spaces
- Sensory cues and recalling locations
- Learns predictive models of objects using locations and sensory cues
- Simulation results
2) A network model of sensorimotor prediction using cortical grid cells

14. Network Model
Grid cell modules
use movement to
update their location
Sensory predictions
(forward model)Predictions + sensation =>
sensory representation
- Sequence of discrete time steps
- Each time step consists of 4 stages
representing a movement followed by a
sensation
New sensation updates
location

15. Updating Grid Cell Module Based On Motion
- Given a motion 𝒅 𝑡
, phase is shifted according to:
- Threshold activity to get a binary location representation
- Cells in each module represented by 2D phase Φ
- Activity at time t is a bump centered around a
phase Φ 𝑡
𝑖
- Each module has a different scale and orientation,
represented by a transform matrix:
𝑴𝑖 =
𝑠𝑖 cos 𝜃𝑖 − 𝑠𝑖 sin 𝜃𝑖
𝑠𝑖 sin 𝜃𝑖 𝑠𝑖 cos 𝜃𝑖
−1
Φ 𝑡,move
𝑖
= 𝜑 + 𝑴𝑖 𝒅 𝑡 mod 1.0 𝜑 ∈ Φ 𝑡−1
𝑖

16. Location Layer Forms Predictions in Sensory Layer
Sensory predictions
(forward model)

17. Sensory Layer is a Sequence Memory Layer
- Neurons in a mini-column learn same FF receptive field.
- Distal dendritic segments form connections to cells in location layer.
- Active segments act as predictions and bias cells.
- With sensory input these cells fire first, and inhibit other cells within
mini-column.
(Hawkins & Ahmad, 2016; Hawkins et al, 2017)
No prediction
t=0
t=1
Sensory input
Predicted input
t=0
Predicted cells
inhibit neighbors
t=1
Sensory input
Sensory layer
Very specific sparse representation
that encodes the current sensory
input at the current location.
Dense representation that activates
the codes for this sensory input at
any location.

18. Multiple Simultaneous Locations Represents Uncertainty
- Sensory representation activates grid cell locations
- If this sensory representation is not unique, we activate
a union of grid cells in each module
- With sufficiently large modules, the union can represent
several locations without confusion
- Movement shifts all active grid cells
(Ahmad & Hawkins, 2016)

19. Learning
- For a new object, we first activate a random
cell in each module and move to first feature
- Select random subset of active sensory cells for
this sensory input. Store location representation
on independent dendritic segment in sensory
cells.
- Store this sensory representation on an
independent dendritic segment in each active
location cell
- Move to next feature and repeat
- This process invokes a new unique location
space and sequentially stores specific location
representation with sensory cues, and specific
sensory code with locations.

20. An Example Recognition Walkthrough
Sense f1

21. An Example Recognition Walkthrough
Move to f2

22. An Example Recognition Walkthrough
Sense f2

23. An Example Recognition Walkthrough
Move to f1

24. Simulation: Network Convergence

25. Network Convergence Improves With More Modules
100 objects with 10 points each, pool of 100 unique features

26. Network Convergence vs Ideal Observer
100 objects, 10 unique features, 20 modules

27. Capacity of Single Cortical Column
100 unique features, 20 modules

28. Two Layer Network Model For Sensorimotor Inference
- Structure of objects are defined by the
relative locations of sensory features
- Grid cell code enables a powerful
predictive sensorimotor network
- Objects are recognized through a
sequence of movements and sensations
- Simulation results demonstrate
convergence and capacity

29. Sensorimotor Inference With Multiple Columns And Long-
Range Lateral Connections
(Hawkins et al, 2017)

30. Sensorimotor Inference With Multiple Columns And Long-
Range Lateral Connections
- Each column has partial knowledge of object.
- Columns use lateral context and vote through long range lateral connections.
- Inference is much faster with multiple columns.
(Hawkins et al, 2017)

31. 1) Properties of grid cells
3) Mapping to biology
- Suggestive biological evidence
- Cortical columns revisited
- Model predictions
- Encoding location using grid cells
- Path integration and location spaces
- Sensory cues and recalling locations
- Learns predictive models of objects using locations and sensory cues
- Simulation results
2) A network model of sensorimotor prediction using cortical grid cells

32. L2/3
L4
L6
L5
Input
Anatomy of a Cortical Column
1) Cortical columns are really complex! The function of a cortical
column must also be complex.
2) Large reciprocal projections from L6a to L4
3) Motor signals drive activity in L6
Simple
Output L2
L3a
L3b
L4
L6a
L6b
L6 ip
L6 mp
L6 bp
L5 tt
L5 cc
L5 cc-ns
L5: Calloway et. al, 2015
L6: Zhang and Deschenes, 1997
Binzegger et al., 2004
Nelson et al., 2013
Output, via thalamus
50%10%
Cortex
Thalamus
Output, direct
L5 CTC: Guillery, 1995
Constantinople and Bruno, 2013
Long range lateral connections

33. 1) Border ownership cells:
Cells fire only if feature is present at object-centric location on object.
Detected even in primary sensory areas (V1 and V2).
(Zhou et al., 2000; Willford & von der Heydt, 2015)
2) Grid cell signatures in cortex:
Cortical areas in humans show grid cell like signatures (fMRI and single cell recordings)
Seen while subjects navigate conceptual object spaces and virtual environments.
(Doeller et al., 2010; Jacobs et al. 2013; Constantinescu et al., 2016; )
3) Sensorimotor prediction in sensory regions:
Cells predict their activity before a saccade.
Predictions during saccades are important for invariant object recognition.
(Duhamel et al., 1992; Nakamura and Colby, 2002; Li and DiCarlo, 2008)
4) Hippocampal functionality may have been conserved in neocortex:
Six-layer neocortex evolved by stacking 3-layer hippocampus and piriform cortex
(Jarvis et al., 2005; Luzatti, 2015)
Other Biological Evidence
33

34. Experimentally Testable Predictions
34
1) Object coding:
Every sensory region will contain layers that are stable while sensing a familiar object.
The set of cells will be sparse but specific to object identity.
Ambiguous information will lead to denser activity in upper layers.
Each region will contain cells tuned to locations of features in the object’s reference frame.
(Zhou et al., 2000; Zheng & Kwon, 2018)
2) Cortical columns:
Cortical cols can learn complete object models
Complexity of objects tied to span of long-range lateral connections
Activity within stable layers will converge slower with long–range connections disabled
Subgranular layers of primary sensory regions (Layer 6) will be driven by motor signals
Grid-like cells in Layer 6a
(Nelson et al., 2013; Sutter et Shepherd, 2015; Lee et al, 2008; Leinweber et al, 2017)

35. Summary: Cortical Grid Cells Represent the Structure of Objects
Entorhinal Cortex
Body in environments
Cortical Column
Sensor patch relative to objects
1. Grid cell code provides a neural mechanism for representing locations
2. Cortical grid cells can build predictive sensorimotor models of objects
3. Suggestive anatomical and physiological mapping to biology

36. Numenta Team
Jeff Hawkins Marcus Lewis
Contact: sahmad@numenta.com
Scott Purdy
Mirko Klukas Luiz Scheinkman
Max Schwarzer