1.
Johns Hopkins APL
September 21, 2018
Jeff Hawkins
jhawkins@numenta.com
Location, Location, Location
A Framework for Intelligence and Cortical Computation

2.
The Human Neocortex
75% of brain
Organ of intelligence
How it works is a mystery
Solving this mystery is the most
important scientific problem of all time
Talk Outline
1) Background
2) Recent Advances
3) Implications

3.
Regions and Hierarchy
Retinal array
Simple
features
Complex
features
Objects
Region 3
Region 2
Region 1
Somatosensory
Visual
regions
Auditory
Felleman, van Essen, 1991
Hierarchy is complex…
40% of all possible connections
between regions exist.
However…
Local circuitry is remarkably similar
in all regions and in all species.
Macaque monkey

4.
Local Cortical Circuits
Dozens of neuron types
Organized in layers
Prototypical projections across all layers
Limited horizontal projections
All regions have a motor output
Similar circuitry in all regions
L3
L4
L6a
L6b
L5a
L5b
Sense Motor
L2
Cajal, 1899
2.5mm

5.
Vernon Mountcastle’s Big Idea
1) All areas of the neocortex look the same because they perform
the same basic function.
2) What makes one region a visual region and another an auditory
region is what it is connected to.
3) A small area of cortex, a 1mm2 “cortical column”, is the unit of
replication and contains the common cortical algorithm.
Mountcastle, 1978

6.
What Does the Neocortex Do?
The neocortex learns a model of the world
- Thousands of objects, how they appear on the sensors
- Where objects are located relative to other objects
- How objects behave
- Learned via movement of sensors
- Physical and abstract objects
Mountcastle corollary:
If the neocortex learns models of objects,
then each column learns models of objects
(including morphology, location relative to other objects, and behaviors)

7.
Talk Outline
1) Background
2) Recent Advances
3) Implications

8.
Thought Experiment

9.
L3
L4
L6a
Location
relative
to object
Object
A single column learns completes
models of objects by integrating
features and locations over time.
“A Theory of How Columns in the Neocortex Enable Learning the
Structure of the World” (Hawkins, et. al., 2017)
Multiple columns can infer objects in a single
sensation by “voting” on object identity.
?
Sensed
feature

10.
How Can Neurons Represent Object-centric Location?
“Grid Cell” neurons in entorhinal cortex represent the location of the
body relative to an environment.
The Big Idea:
Grid cell mechanisms were preserved, also exist in the neocortex.
Cortical grid cells define a location-based framework for understanding
how the neocortex functions.

11.
How Grid Cells Represent Location

12.
How Grid Cells Represent Location
A grid cell fires at multiple locations as the animal moves.
The locations are “anchored” by sensory input,
and “updated” by motor commands.
Firing locations of two grid cells
Grid cells cannot represent a unique location.
Grid cell “modules” differ by scale and orientation.
Module 1 Module 2 Unique location
If grid cell modules anchor independently, then location
is unique to position in room and to the room.
Stensola, Solstad, Frøland, Moser, Moser: 2012

13.
Grid cells represent location of body in
room
Representation of locations are unique to
each room
Location is updated by movement
Each room has its own “location space”
Cortical grid cells represent location sensor
input on object
Representation of locations are unique to
each object
Location is updated by movement
Each object has its own “location space”
Grid cells in older part of
the brain
Learns models of environments
Room 1
Room 2
Grid cells in the neocortex
Learns models of objects

14.
L3
L4
L6a
Object
Location
relative
to object
Sensed
feature
?Grid Cell
Modules
Lucas et. al., submitted

15.
Our Proposal So Far
1) Grid cells exist throughout the neocortex, in every column.
2) They represent the location of the input to the column relative to
the object being sensed.
3) Individual columns learn complete models of objects.
(by integrating input+location over movement)
4) Each object has its own “location space”.
This creates a framework for reverse engineering the rest of the
neocortex……

16.
Compositional Structure
x
z
y
a
c
b
Cup is a previously learned object.
Logo is a previously learned object.
How can we rapidly and efficiently learn
new object, “cup with logo”, without
relearning cup or logo?
Everything in the world is composed of other things, arranged in a
particular way. How is this accomplished?
Cup and logo have their own location
spaces.
Cup with logo can be represented by a
single transform (blue arrow) that
converts any location in cup space to an
equivalent location in logo space.

17.
Displacement Cells
(hypothesized)
x
z
y
a
c
b
Location “x” on logot=2
module 1 module 2 module n
Location “a” on cupt=1Grid cells
Displacement cells “Logo on cup” at a
particular position
Hawkins et. al., in process

18.
L3
L4
L6a
Object
Location
Sensed
feature
Grid Cell
Modules
Displacement
Cell Modules
L5
Hawkins et. al., in process

19.
Object Behaviors
Object behaviors can be represented and learned as
sequences of displacements.
Displacement N
Displacement A
Hawkins et. al., 2016
(sequence memory)

20.
Thousand Brains Theory of Intelligence
a new understanding of hierarchy
Sense array
Objects
Objects
Objects
Sense array
Every column learns models of objects.
Each model is different depending on its inputs.
If two columns learn models of the same object then connections
between them are useful. (Solves “sensor fusion” problem.)

21.
1) Object-centric location signal in sensory cortex:
Cells fire only if feature is at object-centric location on object, even in V1 and V2.
(Zhou et al., 2000; Willford & von der Heydt, 2015)
2) Grid cells in neocortex:
Human neocortex shows grid cell-like signatures (fMRI and single cell recordings)
Seen while subjects navigate conceptual object spaces and virtual environments.
(Doeller et al., 2010; Jacobs et al. 2013; Constantinescu et al., 2016; )
3) Sensorimotor prediction in sensory regions:
Cells predict their activity before a movement is completed.
(Duhamel et al., 1992; Nakamura and Colby, 2002; Li and DiCarlo, 2008)
4) Neocortex evolved from older brain areas involved in mapping and navigation:
Six-layer neocortex evolved by stacking 3-layer hippocampus and piriform cortex
(Jarvis et al., 2005; Luzatti, 2015)
Empirical Support for Theory
22

22.
Implications of Neocortical Theory
1) Neuroscience
2) Theoretical foundation for
- epistemology
- science of belief and false belief
- limits of human intelligence
- pedagogy
- diseases of the mind
3) Artificial Intelligence and Robotics

23.
True AI requires
- Thousand brains model of intelligence
- Each model built using:
- Object-centric locations and location spaces
- Compositional structure
- Embodiment and learning through movement
True AI does not have to be human like
- Faster, Larger or Smaller
- Different sensors
- Physically distributed
- New embodiments, including virtual

24.
Implications of Neocortical Theory
1) Neuroscience
2) Theoretical foundation for
- epistemology
- science of belief and false belief
- limits of human intelligence
- pedagogy
- diseases of the mind
3) Machine Intelligence and Robotics
- purpose-built brains, e.g. for mathematics or physics
- virtual brains for cyber-security, cyber-warfare
- robotics for industry and military
- robotics for space exploration and colonization

25.
Numenta Team
Subutai Ahmad Marcus Lewis
Thank You
Scott Purdy
Mirko Klukas Luiz Scheinkman