A very distilled introduction to the concepts of Hierarchical Temporal Memory (HTM) and Sparse Distributed Representations (SDRs) as implemented by Numenta.

1. NuPIC
Bay Area Meetup
Mar, 25 2015
• HTM and SDRs for Newbies – Jeff Fohl ~20 mins
• Open Source Community Updates – Matt Taylor ~10 mins
• Nupic Demo ~30 mins
– Matt [ Application Demo ]
– Subutai [ Research Demo ]
– Chandan [ Walnut ]

2. HTM and SDRs
for
Newbies

3. Brain

4. Neocortex

5. Neocortical Layers

6. Neocortical Layers

7. Neuron

8. Dendrite

9. Synapses

10. Neocortical Hierarchy

11. Hierarchical Temporal Memory:
HTM

12. Hierarchical Temporal Memory is the model
that describes how information is stored and
processed in the brain.

13. • An HTM is a hierarchy of nearly identical
regions of neurons.
• Regions learn, infer, and recall time-based
sequences.
• Regions are constantly predicting the next
inputs.
• Regions detect anomalous input.

14. Hierarchical

15. Temporal

16. Memory
In order to remember sequences, a region
needs a way to store this information. How
is this done?

17. Sparse Distributed Representation:
SDR

18. SDRs are the fundamental data format of
Hierarchical Temporal Memory.

19. How do we represent an SDR?
It can be written as a vector with the
following properties:
• High dimensionality. It will usually contain at least 2,000 elements.
• Binary. Each element in the vector is either “on” or “off”.
• Sparse. Out of the total number of bits, only a small percentage (usually
2%) will ever be active.
• Distributed. The “on” bits will be spread across the vector.
• Semantic. Each bit in the vector has meaning.

20. 0001000000000010000000000000000000000100000000000000000000000000000000000000000
0000000000000000000000000000000000000000000000000000000000000000000000000000000
0000000000000000000000000000000100000000000000000000000000000000000000000000000
0000000000000000000000000000010000000000000000000000000000000000000000000000000
0000100000000000000000010000000000000000000000000000000000000000000000000000000
0010000000000001000000000000000000000000000000000000000000000000000100000000000
0000000000000000000000000000000000000000000100000000000000000000000000000000000
0000000000000001000000000000000000000000000000000000000000000000000000000000000
0000000000000000000000000000000010000000000000000000000000000000000000000000000
0000010000000000000000000000000000000000000000000000000000000000000000000000000
0000000000000000000000000000000000000000000000000000000000100000000000000000000
0000000000000000000000000000000000000000000000000001000000000000000000000000000
0000000000100000000000000100000000000000000000000000000000001000000000000000000
0000000000000000000000000000000000000000000000000000000000000001000000000000000
0000000000000000000000000000000000000000000000000000000000000000000000000000000
0000000000000000000000000000000000000000000000000000000000000000000000000000000
0000000000000000000000000000000000000000000000000000000000000000000000000000000
0000000000000000000000010000000000000000010000000000000000000000000000000000000
0000000000000000000100000000010000000000000000000000000010000000000000000000000
0000000000000000000000000000000000000000000000000000000000000000000000000000000
0000000000000000000000000000000000000000000000000000000000000000000000000000000
0000000000000000000000000000000000000000000000000000000000000000000000000000000
0000010000000000000000000000000000000000000000000000000000000001000000000000000
0000000001010000000000000000000000000000000000000000000000000000000000000000000
0100000000000000000000000000000000000000000000000000000000000000000010000000000
0000000000000000000000000
This could be written out using 1s and 0s:

21. Or, perhaps using dots, for a visual representation:

22. If you are using so many bits, why not use
them all? By using only 2% at a time, aren’t
you wasting information?
Q:

23. You would be, if you were processing the
information in the way a traditional computer
processes information. But, this is different.
A:

24. Dense Representation

25. m

26. o

27. Sparse Representation

30. uppercase
lowercase
consonant
vowel
soft
hard
open
closed
serif
sans serif
m

31. uppercase
lowercase
consonant
vowel
soft
hard
open
closed
serif
sans serif
m

32. SDRs are ultra high capacity

33. The number of patterns that can be represented by a 2,048 bit SDR with 40 active
bits:
1084
(that’s way more than the number of atoms in the observable universe)

34. Storing SDRs

35. A 2,000 bit SDR, with 40 active bits, means we only need
to store the indices of those 40 – not the entire 2,000 bits.

36. 7,29,47,96,101,134,139,157,180,201,234,245,267,273,291,
299,
304,356,358,363,389,404,435,444,453,467,481,482,495,50
5,603,667,689,703,766,781,834,875,909,945

37. SDR Subsampling

38. indices
60
107
243
575
…
10,000 bits
200 active

40. indices
1318
1686
2975
4441
4461
4517
4860
7146
8179
9158
10,000 bits
10 active

41. SDR Unions

45. Thank you
To get more depth on this topic, visit Numenta’s YouTube channel