Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Sparse Distributed Representations: Our Brain's Data Structure

8,029 views

Published on

Presentation by Subutai Ahmad at Numenta Workshop on October 17, 2014.

Published in: Data & Analytics
  • I'd advise you to use this service: ⇒ www.WritePaper.info ⇐ The price of your order will depend on the deadline and type of paper (e.g. bachelor, undergraduate etc). The more time you have before the deadline - the less price of the order you will have. Thus, this service offers high-quality essays at the optimal price.
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • You can try to use this service ⇒ www.HelpWriting.net ⇐ I have used it several times in college and was absolutely satisfied with the result.
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Hi there! I just wanted to share a list of sites that helped me a lot during my studies: .................................................................................................................................... www.EssayWrite.best - Write an essay .................................................................................................................................... www.LitReview.xyz - Summary of books .................................................................................................................................... www.Coursework.best - Online coursework .................................................................................................................................... www.Dissertations.me - proquest dissertations .................................................................................................................................... www.ReMovie.club - Movies reviews .................................................................................................................................... www.WebSlides.vip - Best powerpoint presentations .................................................................................................................................... www.WritePaper.info - Write a research paper .................................................................................................................................... www.EddyHelp.com - Homework help online .................................................................................................................................... www.MyResumeHelp.net - Professional resume writing service .................................................................................................................................. www.HelpWriting.net - Help with writing any papers ......................................................................................................................................... Save so as not to lose
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • @Damien Stanton Hi Damien! Thank you for your interest and apologies that you are not able to see the symbols in the slides. We will contact you via email.
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Unfortunately the mathematical symbols (lambda, theta, etc.) don't show up in slideshare! Can I please have access to this presentation in PDF?
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

Sparse Distributed Representations: Our Brain's Data Structure

  1. 1. Sparse Distributed Representations: Our Brain’s Data Structure Numenta Workshop October 17, 2014 Subutai Ahmad, VP Research sahmad@numenta.com
  2. 2. Sparse Distributed Representations: Our Brain’s Data Structure Numenta Workshop October 17, 2014 Subutai Ahmad, VP Research sahmad@numenta.com
  3. 3. The Role of Sparse Distributed Representations in Cortex 1) Sensory perception 3) Motor control 4) Prediction 2) Planning 5) Attention Sparse Distribution Representations (SDRs) are the foundation for all these functions, across all sensory modalities Analysis of this common cortical data structure can provide a rigorous foundation for cortical computing
  4. 4. Talk Outline 1) Introduction to Sparse Distributed Representations (SDRs) 2) Fundamental properties of SDRs – Error bounds – Scaling laws
  5. 5. From: Prof. Hasan, Max-Planck- Institut for Research
  6. 6. Basics Attributes of SDRs 1) Only a small number of neurons are firing at any point in time 3) Every cell represents something and has meaning 4) Information is distributed and no single neuron is critical 2) There are a very large number of neurons 5) Every neuron only connects to a subset of other neurons 6) SDRs enable extremely fast computation 7) SDRs are binary x = 0100000000000000000100000000000110000000
  7. 7. Multiple input SDR’s Single bit in an output SDR How Does a Single Neuron Operate on SDRs?
  8. 8. Proximal segments represent dozens of separate patterns in a single segment How Does a Single Neuron Operate on SDRs? Hundreds of distal segments each detect a unique SDR using a threshold Feedback SDR Context SDR Bottom-up input SDR In both cases each synapse corresponds to one bit in the incoming high dimensional SDR
  9. 9. • Extremely high capacity • Recognize patterns in the presence of noise • Robust to random deletions • Represent dynamic set of patterns in a single fixed structure • Extremely efficient Fundamental Properties of SDRs
  10. 10. Notation • We represent a SDR vector as a vector with n binary values where each bit represents the activity of a single neuron: • s = percent of ON bits, w = number of ON bits x =[b0,… ,bn-1] wx = s ´ n = x 1 Example • n = 40, s = 0.1, w = 4 • Typical range of numbers in HTM implementations: n = 2048 to 65,536 s = 0.05% to 2% w = 40 y =1000000000000000000100000000000110000000 x = 0100000000000000000100000000000110000000
  11. 11. SDRs Have Extremely High Capacity • The number of unique patterns that can be represented is: • This is far smaller than 2n, but far larger than any reasonable need • Example: with n = 2048 and w = 40, the number of unique patterns is > 1084 >> # atoms in universe • Chance that two random vectors are identical is essentially zero: n w æ èç ö ø÷ = n! w! n - w( )! 1/ n w æ èç ö ø÷
  12. 12. • Extremely high capacity • Recognize patterns in the presence of noise • Robust to random deletions • Represent multiple patterns in a single fixed structure • Extremely efficient Fundamental Properties of SDRs
  13. 13. Similarity Metric for Recognition of SDR Patterns • We don’t use typical vector similarities – Neurons cannot compute Euclidean or Hamming distance between SDRs – Any p-norm requires full connectivity • Compute similarity using an overlap metric – The overlap is simply the number of bits in common – Requires only minimal connectivity – Mathematically, take the AND of two vectors and compute its length • Detecting a “Match” – Two SDR vectors “match” if their overlap meets a minimum threshold overlap(x,y) º x Ù y match(x,y) º overlap(x,y) ³q q
  14. 14. Overlap example • N=40, s=0.1, w=4 • The two vectors have an overlap of 3, so they “match” if the threshold is 3. y =1000000000000000000100000000000110000000 x = 0100000000000000000100000000000110000000
  15. 15. How Accurate is Matching With Noise? • As you decrease the match threshold , you decrease sensitivity and increase robustness to noise • You also increase the chance of false positives Decrease q q
  16. 16. How Many Vectors Match When You Decrease the Threshold? • Define the “overlap set of x” to be the set of vectors with exactly b bits of overlap with x • The number of such vectors is: Wx (n,w,b) = wx b æ èç ö ø÷ ´ n - wx w - b æ èç ö ø÷ Wx (n,w,b) Number subsets of x with exactly b bits ON Number patterns occupying the rest of the vector with exactly w-b bits ON
  17. 17. Error Bound for Classification with Noise • Give a single stored pattern, probability of false positive is: • Given M patterns, probability of a false positive is: fpw n (q) = Wx (n,w,b) b=q w å n w æ èç ö ø÷ fpX (q) £ fpwxi n (q) i=0 M-1 å
  18. 18. What Does This Mean in Practice? • With SDRs you can classify a huge number of patterns with substantial noise (if n and w are large enough) Examples • n = 2048, w = 40 With up to 14 bits of noise (33%), you can classify a quadrillion patterns with an error rate of less than 10-24 With up to 20 bits of noise (50%), you can classify a quadrillion patterns with an error rate of less than 10-11 • n = 64, w=12 With up to 4 bits of noise (33%), you can classify 10 patterns with an error rate of 0.04%
  19. 19. Neurons Are Highly Robust Pattern Recognizers Hundreds of distal segments each detect a unique SDR using a threshold You can have tens of thousands of neurons examining a single input SDR, and very robustly matching complex patterns
  20. 20. • Extremely high capacity • Recognize patterns in the presence of noise • Robust to random deletions • Represent multiple patterns in a single fixed structure • Extremely efficient Fundamental Properties of SDRs
  21. 21. SDRs are Robust to Random Deletions • In cortex bits in an SDR can randomly disappear – Synapses can be quite unreliable – Individual neurons can die – A patch of cortex can be damaged • The analysis for random deletions is very similar to noise • SDRs can naturally handle fairly significant random failures – Failures are tolerated in any SDR and in any part of the system • This is a great property for those building HTM based hardware – The probability of failures can be exactly characterized
  22. 22. • Extremely high capacity • Recognize patterns in the presence of noise • Robust to random deletions • Represent multiple patterns in a single fixed structure • Extremely efficient Fundamental Properties of SDRs
  23. 23. Representing Multiple Patterns in a Single SDR • There are situations where we want to store multiple patterns within a single SDR and match them • In temporal inference the system might make multiple predictions about the future Example
  24. 24. Unions of SDRs • We can store a set of patterns in a single fixed representation by taking the OR of all the individual patterns • The vector representing the union is also going to match a large number of other patterns that were not one of the original 10 • How many such patterns can we store reliably, without a high chance of false positives? Is this SDR a member? 1) 2) 3) …. 10) 2% < 20%Union
  25. 25. Error Bounds for Unions • Expected number of ON bits: • Give a union of M patterns, the expected probability of a false positive (with noise) is:
  26. 26. What Does This Mean in Practice? • You can form reliable unions of a reasonable number of patterns (assuming large enough n and w) Examples • n = 2048, w = 40 The union of 50 patterns leads to an error rate of 10-9 • n = 512, w=10 The union of 50 patterns leads to an error rate of 0.9%
  27. 27. • Extremely high capacity • Recognize patterns in the presence of noise • Robust to random deletions • Represent multiple patterns in a single fixed structure • Extremely efficient Fundamental Properties of SDRs
  28. 28. SDRs Enable Highly Efficient Operations • In cortex complex operations are carried out rapidly – Visual system can perform object recognition in 100-150 msecs • SDR vectors are large, but all operations are O(w) and independent of vector size – No loops or optimization process required • Matching a pattern against a dynamic list (unions) is O(w) and independent of the number of items in the list • Enables a tiny dendritic segment to perform robust pattern recognition • We can simulate 200,000 neurons in software at about 25-50Hz
  29. 29. Summary • SDR’s are the common data structure in the cortex • SDR’s enable flexible recognition systems that have very high capacity, and are robust to a large amount of noise • The union property allows a fixed representation to encode a dynamically changing set of patterns • The analysis of SDR’s provides a principled foundation for characterizing the behavior of the HTM learning algorithms and all cognitive functions • Sparse memory (Kanerva), Sparse coding (Olshausen), Bloom filters (Broder) Related work
  30. 30. Questions? Math jokes? Follow us on Twitter @numenta Sign up for our newsletter at www.numenta.com Subutai Ahmad sahmad@numenta.com nupic-theory mailing list numenta.org/lists

×