This was a presentation given on December 15, 2017 at the MIT Center for Brains, Minds + Machines as part of their Brains, Minds and Machines Seminar Series. You can watch the recording of the presentation after Slide 1. In this talk, Jeff describes a theory that sensory regions of the neocortex process two inputs. One input is the well-known sensory data arriving via thalamic relay cells. We propose the second input is a representation of allocentric location. The allocentric location represents where the sensed feature is relative to the object being sensed, in an object-centric reference frame. As the sensors move, cortical columns learn complete models of objects by integrating sensory features and location representations over time. Lateral projections allow columns to rapidly reach a consensus of what object is being sensed. We propose that the representation of allocentric location is derived locally, in layer 6 of each column, using the same tiling principles as grid cells in the entorhinal cortex. Because individual cortical columns are able to model complete complex objects, cortical regions are far more powerful than currently believed. The inclusion of allocentric location offers the possibility of rapid progress in understanding the function of numerous aspects of cortical anatomy. Jeff discusses material from these two papers. Others can be found at https://numenta.com/papers A Theory of How Columns in the Neocortex Enable Learning the Structure of the World URL: https://doi.org/10.3389/fncir.2017.00081 Why Neurons Have Thousands of Synapses, A Theory of Sequence Memory in the Neocortex URL: https://doi.org/10.3389/fncir.2016.00023

1. MIT
December 15, 2017
Jeff Hawkins
jhawkins@numenta.com
Have We Missed Half of What the Neocortex Does?
Allocentric Location as the Basis of Perception

2. 1) Reverse engineer the neocortex
- an ambitious but realizable goal
- seek biologically accurate theories
- test empirically and via simulation
2) Enable technology based on cortical theory
- active open source community
- basis for Machine Intelligence

4. L2
L3a
L3b
L4
L6a
L6b
L6 ip
L6 mp
L6 bp
L5 tt
L5 cc
L5 cc-ns
L2/3
L4
L6
L5
Input
The Cortical Column
1) Cortical columns are complex
- Twelve or more excitatory cellular layers
- Two parallel FF pathways
- Parallel FB pathways (not shown)
- Numerous intra- and inter-column connections (not shown)
- Inhibitory neurons/circuits are equally complex
2) The function of a cortical column must also be complex.
3) Whatever a column does applies to everything the cortex does.
L5: Calloway et. al, 2015
L6: Zhang and Deschenes, 1997
Simple
Output, via thalamus
50%10%
Cortex
Thalamus
Output, direct
L5 CTC: Guillery, 1995
Constantinople and Bruno, 2013
A Couple of Thoughts
Output

5. Observation:
The neocortex is constantly predicting its inputs.
How do networks of neurons, as seen in the neocortex,
learn predictive models of the world?
Research:

6. 1) How does the cortex learn predictive models of extrinsic sequences?
2) How does the cortex learn predictive models of sensorimotor sequences?
Current research: How do columns compute allocentric location?
- Grid cells in entorhinal cortex solve a similar problem
- Big Idea: cortical columns contain analogs of grid cells and head direction cells
- Starting to understand the function of numerous layers and connections
“Why Neurons Have Thousands of Synapses, a Theory of Sequence Memory in the Neocortex”
Hawkins and Ahmad, Frontiers in Neural Circuits, 2016/03/30
- Big Idea: Pyramidal neuron model for prediction
- A single layer network model for sequence memory
- Properties of sparse activations
“A Theory of How Columns in the Neocortex Learn the Structure of the World”
Hawkins, Ahmad, and Cui, Frontiers in Neural Circuits, 2017/10/25
- Extension of sequence memory model
- Big Idea: Columns compute “allocentric” location of input
- By moving sensor, columns learn models of complete objects

7. Proximal synapses: Cause somatic spikes
Define classic receptive field of neuron
Distal synapses: Cause dendritic spikes
Put the cell into a depolarized, or “predictive” state
Depolarized neurons fire sooner, inhibiting nearby neurons.
A neuron can predict its activity in hundreds of unique contexts.
5K to 30K excitatory synapses
- 10% proximal
- 90% distal
Distal dendrites are pattern detectors
- 8-15 co-active, co-located synapses
generate dendritic spike
- sustained depolarization of soma
HTM Neuron Model
Prediction Starts in the Neuron
Pyramidal Neuron
Major, Larkum and Schiller 2013

8. Properties of Sparse Activations
L2
L3a
L3b
L4
L6a
L6b
L6 ip
L6 mp
L6 bp
L5 tt
L5 cc
L5 cc-ns
Example: One layer of cells, 5,000 neurons, 2% (100) active
1) Representational capacity is virtually unlimited
(5,000 choose 100) = 3x10211
2) Randomly chosen representations have minimal overlap
3) A neuron can robustly recognize an activation pattern by forming 10 to 20 synapses
4) Unions of patterns do not cause errors in recognition
Hypothesis: Cellular layers use unions to represent uncertainty
Hawkins, Ahmad, 2016
Ahmad, Hawkins, 2015
Pattern 1 (100 active cells)
Cell robustly recognizes pattern1
by forming synapses to small sub-
sample of active cells
Union
Patterns 1-10 (1,000 active cells)
Cell still robustly recognizes pattern 1

9. A Single Layer Network Model for Sequence Memory
- Neurons in a mini-column learn same FF receptive field.
- Neurons forms distal connections to nearby cells.
No prediction Predicted input
(Hawkins & Ahmad, 2016)
(Cui et al, 2016)
- High capacity (learns up to 1M transitions)
- Learns high-order sequences: “ABCD” vs “XBCY”
- Makes simultaneous predictions: “BC…” predicts “D” and “Y”
- Extremely robust (tolerant to 40% noise and faults)
- Learning is unsupervised, continuous, and local
- Satisfies many biological constraints
- Multiple open source implementations (some commercial)
t=0
t=1
Predicted cells fire first
and inhibit neighbors
Next prediction t=2
t=0
t=1

10. 1) How does the cortex learn predictive models of extrinsic sequences?
2) How does the cortex learn predictive models of sensorimotor sequences?
Current research: How do columns compute allocentric location?
- Grid cells in entorhinal cortex solve a similar problem
- Hypothesis: cortical columns contain analogs of grid cells and head direction cells
- Starting to understand the function of numerous layers and connections
“Why Neurons Have Thousands of Synapses, a Theory of Sequence Memory in the Neocortex”
Hawkins and Ahmad, Frontiers in Neural Circuits, 2016/03/30
- Pyramidal neuron model
- A single layer network model for sequence memory
- Properties of sparse activations
“A Theory of How Columns in the Neocortex Learn the Structure of the World”
Hawkins, Ahmad, and Cui, Frontiers in Neural Circuits, 2017/10/25
- Extension of sequence memory model
- Big Idea: Columns compute “allocentric” location of input
- By moving sensor, columns learn models of complete objects

11. How Could a Layer of Neurons Learn a Predictive Model of
Sensorimotor Sequences?
Sequence memory
Sensorimotor sequences
SensorMotor-related context
Hypothesis:
By adding motor-related context, a cellular layer can predict
its input as the sensor moves.
What is the correct motor-related context?
L2
L3a
L3b
L4
L6a
L6b
L6 ip
L6 mp
L6 bp
L5 tt
L5 cc
L5 cc-ns
50%
Sensory
feature

13. Two Layer Model of Sensorimotor Sequence Memory
Feature @ location
Object Stable over movement of sensor
With allocentric location input, a column can learn models of
complete objects by sensing different locations on object over time.
Sensor
Feature
Allocentric
Location
Pooling
Seq Mem
Changes with each movement

14. Object
Feature @ Location
Location
on object
Column 1 Column 2 Column 3
Sensor
feature
Sensorimotor Inference With Multiple Columns
Each column has partial knowledge of object.
Long range connections in object layer allow columns to vote.
Inference is much faster with multiple columns.

15. FeatureFeatureFeatureLocationLocationLocation
Output
Input
Objects Recognized By Integrating Inputs Over Time

16. FeatureLocationFeatureLocationFeatureLocation
Column 1 Column 2 Column 3
Output
Input
Recognition is Faster with Multiple Columns

17. Yale-CMU-Berkeley (YCB) Object Benchmark (Calli et al, 2017)
- 80 objects designed for robotics grasping tasks
- Includes high-resolution 3D CAD files
YCB Object Benchmark
We created a virtual hand using the Unity game engine
Curvature based sensor on each fingertip
4096 neurons per layer per column
98.7% recall accuracy (77/78 uniquely classified)
Convergence time depends on object, sequence of
sensations, number of fingers.
Simulation using YCB Object Benchmark

18. Pairwise confusion between objects after 1 touch
Convergence 1 finger 1 touch

19. Pairwise confusion between objects after 2 touches
Convergence 1 finger 2 touches

20. Pairwise confusion between objects after 6 touches
Convergence 1 finger 6 touches

21. Pairwise confusion between objects after 10
touches
Convergence 1 finger 10 touches

22. Convergence Time vs. Number of Columns
This is why we can infer complex objects in a single grasp or single visual fixation.

23. 1) How does the cortex learn predictive models of extrinsic sequences?
2) How does the cortex learn predictive models of sensorimotor sequences?
Current research: How do columns compute allocentric location?
- Hypothesis: cortical columns contain analogs of grid cells and head direction cells
- Starting to understand the function of numerous layers and connections
“Why Neurons Have Thousands of Synapses, a Theory of Sequence Memory in the Neocortex”
Hawkins and Ahmad, Frontiers in Neural Circuits, 2016/03/30
- Pyramidal neuron model
- A single layer network model for sequence memory
- Properties of sparse activations
“A Theory of How Columns in the Neocortex Learn the Structure of the World”
Hawkins, Ahmad, and Cui, Frontiers in Neural Circuits, 2017/10/25
- Extension of sequence memory model
- Big Idea: Columns compute “allocentric” location of input
- By moving sensor, columns learn models of complete objects

24. Entorhinal Cortex
environments
A
B C
X
Y Z
R
S T
Room 3
Room 2Room 1
Location
- Encoded by Grid Cells
- Unique to location in room AND room
- Location is updated by movement
Orientation (of head to room)
- Encoded by Head Direction Cells
- Anchored to room
- Orientation is updated by movement
Location
- Unique to location on object AND object
- Location is updated by movement
Orientation (of sensor patch to object)
- Anchored to object
- Orientation is updated by movement
Cortical Column
objects
Hypothesis:
Cortical columns contain analogs of grid cells and head direction cells
A
C
B
X
Y
Z
Stensola, Solstad, Frøland, Moser, Moser: 2012
Location and Orientation are both necessary
to learn the structure of rooms and predict
sensory input.
Location and Orientation are both necessary
to learn the structure of objects and predict
sensory input.

25. L3
L4
L6a
L6b
L5a
L5b
Mapping Orientation and Location to a Cortical Column (most complex slide)
Sensation
Orientation
1) A column is a two-stage sensorimotor model for learning and inferring structure.
2) A column usually cannot infer a Feature or Object in one sensation.
- Integrate over time (sense, move, sense, move, sense..)
- Vote with neighboring columns
3) This system is most obvious for touch, but it applies to vision and other sensory modalities.
Because this architecture exists throughout the neocortex, it suggests we learn, infer,
and manipulate abstract concepts the same way we manipulate objects in the world.
Location
Sensation @ Orientation
Feature
Feature @ Location
Object
Motor updated (HD cell-like)
Motor updated (grid cell-like)
Seq mem
Pooling
Seq mem
Pooling
Meaning Operation

26. Rethinking Hierarchy
Every column learns complete models of objects. They operate in parallel.
Inputs project to multiple levels at once. Columns operate at different
scales of input.
Sense
Simple features
Complex features
Objects
Classic
Objects
Objects
Objects
Sensor array
Proposed
Region 3
Region 2
Region 1

27. Rethinking Hierarchy
Every column learns complete models of objects. They operate in parallel.
Inputs project to multiple levels at once. Columns operate at different
scales of input.
Non-hierarchical connections allow columns to vote on shared elements
such as “object” and “feature”.
Sense
Simple features
Complex features
Objects
Classic
Sensor array
Objects
Objects
Objects
Sensor array
vision touch
Proposed
Region 3
Region 2
Region 1

28. Summary
Goal: Understand the function and operation of the laminar circuits in the neocortex.
Method: Study how cortical columns make predictions of their inputs.
Proposals
1) Pyramidal neurons are the substrate of prediction.
Each neuron predicts its activity in hundreds of contexts.
2) A single layer of neurons forms a predictive memory of high-order sequences.
(sparse activations, mini-columns, fast inhibition, and lateral connections)
3) A two-layer network forms a predictive memory of sensorimotor sequences.
(add motor-derived context and a pooling layer)
4) Columns need motor-derived representations of location and orientation, of the
sensor relative to the object. These are analogous to grid and head direction cells.
5) A framework for the cortical column.
- Columns learn complete models of objects as “features at locations”, using two
sensorimotor inference stages.
6) The neocortex contains thousands of parallel models, that resolve uncertainty by
associative linking and/or movement of the sensors.

29. Open Issues
Behaviors: how are they learned, encoded, and applied to objects?
Detailed model of hierarchy including thalamus
How can the model be applied to “Where” pathways, and how do “What” and “Where”
pathways work together
Collaborations
There are many testable predictions in this model, a “green field”. We welcome
collaborations and discussions.
We are always interested in hosting visiting scholars and interns.

30. Numenta Team
Subutai Ahmad
VP Research
Marcus Lewis
Thank You