1) The HTM theory proposes a biologically realistic neuron and cellular layer model that recognizes hundreds of patterns and learns via growing new synapses rather than changing weights. 2) The cellular layer model learns predictive models of temporal and sensorimotor sequences. 3) The theory deduces that every cortical column determines the allocentric location of inputs on an object, allowing columns to learn complete models of objects and infer more quickly with multiple columns.

1. HTM Meetup
November 3, 2017
Jeff Hawkins
jhawkins@numenta.com
Numenta Brain Theory Discoveries of 2016/2017

2. 1) Reverse Engineer the Neocortex
- biologically accurate theory
- test empirically and via simulation
2) Enable technology based on cortical theory
- active open source community
- basis for Artificial General Intelligence
- IP licensing
We have made
significant advances
on the cortical theory

4. (Thomson & Bannister, 2003)(Constantinople and Bruno, 2013)
Cortical Column Anatomy

5. March 30, 2016
October 25, 2017
See all papers at Numenta.com/papers
3) How columns in
cortex model objects
through movement
4) Missing Ingredient!
1) Neuron model
2) How layers of
neurons in cortex
model sequences

6. Point Neuron Model
x Real neurons are not
like this!
Integrate and fire neuron: Lapicque, 1907
Perceptron: Rosenblatt 1962;
Deep learning: Rumelhart et al. 1986; LeCun et al., 2015
Artificial Neurons

7. Real and HTM neurons recognize 100’s of unique patterns.
Most recognized patterns act as predictions.
5K to 30K excitatory synapses
- 10% proximal, can cause spike
- 90% distal, cannot cause spike
Dendrites are pattern detectors
- 15 co-active, co-located synapses
has big effect
Real Neuron HTM Neuron Model

8. Learning is by Rewiring, Forming New Synapses
Not by Changing Synaptic Weights
Biology
HTM

9. Modeling a Cellular Layer
HTM Sequence Memory
A
X B
B
C
C
Y
D
Before learning
A
X B’’
B’
C’’
C’
Y’’
D’
After learning
Same columns,
but only one cell active per column.
Sequences A-B-C-D vs. X-B-C-Y

10. March 30, 2016
October 25, 2016
See all papers at Numenta.com/papers
3) How columns in
cortex model objects
through movement
4) Missing Ingredient!
1) Neuron model
2) How layers of
neurons in cortex
model sequences

12. L6b
Output
Location on object
“allocentric”
L4 (input layer)
L2/3 (output layer)
L5
L6a
HTM Sensorimotor Inference Theory (single column)
1) Every column determines allocentric location of input
2) As sensor moves, column is exposed to different
feature/locations on object
3) Output layer “pools” feature/locations. Stable over movement.
4) Columns learn models of complete objects
Object
Input
Sensed Feature
45%Feature@Location

13. Output layer
“Object”
Input layer
“Feature/Location”
Location
on object
Column 1 Column 2 Column 3
Sensory
feature
HTM Sensorimotor Inference Theory (multiple columns)
Each column has partial knowledge of object.
Long range connections in output layer allow columns to vote.
Inference is much faster with multiple columns.

14. FeatureFeatureFeatureLocationLocationLocation
OBJECTS RECOGNIZED BY INTEGRATING INPUTS OVER TIME
Output
Input

15. FeatureLocationFeatureLocationFeatureLocation
RECOGNITION IS FASTER WITH MULTIPLE COLUMNS
Column 1 Column 2 Column 3
Output
Input

16. • Yale-CMU-Berkeley (YCB) Object Benchmark (Calli et al, 2017)
– Diverse set of objects designed for robotics grasping tasks
– 80 common physical objects
– Includes 78 complete high resolution 3D CAD files
Simulations: YCB Object Benchmark

17. • Virtual hand using the Unity game engine
• Inputs
– Curvature based sensor on each fingertip
– Both inputs are highly sparse binary vectors
• Network with 4096 neurons per layer per
column
• Results
• 98.7% recall accuracy (77/78 uniquely classified)
• Convergence time depends on object and sequence of sensations
Simulations: YCB Benchmark

18. Pairwise confusion between objects after 1 touch
Convergence over time

19. Pairwise confusion between objects after 2 touches
Convergence over time

20. Pairwise confusion between objects after 3 touches
Convergence over time

21. Pairwise confusion between objects after 4 touches
Convergence over time

22. Pairwise confusion between objects after 6 touches
Convergence over time

23. Pairwise confusion between objects after 10 touches
Convergence over time

24. Convergence with Multiple Columns

25. • Our model predicts that sensory regions will contain cells tuned to the location
of features in an object's reference frame
• Movement dynamically modulates cell responses even in primary sensory
regions (Trotter and Celebrini, 1999; Werner-Reiss et al., 2003)
• Grid cells solve a similar problem, location of body in environment
“Border ownership cells”
(Willford & von der Heydt, 2015)
Evidence for Allocentric Location in Cortex

26. Summary
1) HTM Neuron Model
- Biologically more realistic
- Functionally more powerful
- Recognizes 100’s of unique patterns
- Most patterns put neuron into “predictive” state
- Learning is via grown of new synapses
2) HTM Cellular Layer Model
- Learns predictive models of sensory input
- Applied to temporal sequences
- Applied to sensorimotor sequences
3) Deduced Allocentric Location is Determined in Every Column
- Every column learns complete models of objects
- Multiple columns infer quickly
4) Allocentric Location Changes “everything”
- Columns and regions are far more powerful than
previously thought
- Changes how we think about hierarchy
- Progress on understanding rest of cortex will accelerate
- Implications for robotics and machine intelligence