Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Autonomy Incubator Seminar Series: Neuromorphic solutions for autonomous land and aerial vehicles

  • Login to see the comments

  • Be the first to like this

Autonomy Incubator Seminar Series: Neuromorphic solutions for autonomous land and aerial vehicles

  1. 1. Neuromorphic solutions for autonomous land and aerial vehicles Massimiliano Versace CEO, Neurala Inc. Director, Boston University Neuromorphics Lab www.neurala.com Copyright 2013 Neurala, Inc. nl.bu.edu ● Proprietary and Confidential ● versace@neurala.com www.neurala.com ● +1-617-418-6161 1
  2. 2. To produce virtual and robotic agents that autonomously learn to perform useful tasks Copyright 2013 Neurala, Inc. ● Proprietary and Confidential ● www.neurala.com ● +1-617-418-6161 2
  3. 3. Why are not robots everywhere? limited intelligence 1:1 ratio robot/ human operator barriers to adoption Copyright 2013 Neurala, Inc. ● Proprietary and Confidential ● www.neurala.com ● +1-617-418-6161 3
  4. 4. Three ingredients for success Smart mind Powerful brain Copyright 2013 Neurala, Inc. Inexpensive body ● Proprietary and Confidential ● www.neurala.com ● +1-617-418-6161 4
  5. 5. The body Copyright 2013 Neurala, Inc. ● Proprietary and Confidential ● www.neurala.com ● +1-617-418-6161 5
  6. 6. Low cost of sensors, new actuators navigation accelerometer gyroscope magnetometer front and rear cameras NFC barometer speaker microphone proximity light sensor Bluetooth GPS WiFi + cellular humidity temperature Copyright 2013 Neurala, Inc. vision audition ● Proprietary and Confidential ● www.neurala.com ● +1-617-418-6161 6
  7. 7. Three ingredients for success Smart mind Powerful brain Copyright 2013 Neurala, Inc. Inexpensive body ● Proprietary and Confidential ● www.neurala.com ● +1-617-418-6161 7
  8. 8. Many robots, many minds R&D $$$ $$ensors Copyright 2013 Neurala, Inc. R&D $$ $ensors ● Proprietary and Confidential ● R&D $ www.neurala.com ● +1-617-418-6161 8
  9. 9. One mind for autonomous robots? learning parallel processing Copyright 2013 Neurala, Inc. ● Proprietary and Confidential ● www.neurala.com ● +1-617-418-6161 9
  10. 10. Why do we LOVE this mess??!! Neuromorphic = of the form of neurons, brains Even in apparently simple tasks, many brain functions work in synergy to help us solve otherwise VERY HARD PROBLEMS The brain does not work in silos of competences. E.g., as you move around in the world, vision helps navigation (e.g., landmark recognition, obstacle avoidance), navigation helps vision (what to expect in one environment vs. another). Ever thought why you can’t recognize the teller of your grocery at the movie theater? Copyright 2013 Neurala, Inc. ● Proprietary and Confidential ● www.neurala.com ● +1-617-418-6161 10
  11. 11. “Stovepiped” sensemakig Rock detector Sky detector & cloud detector Dust devil detector Onboard Autonomous Science Investigation System for Opportunistic Rover Science, Castano et al., 2007 Copyright 2013 Neurala, Inc. ● Proprietary and Confidential ● www.neurala.com ● +1-617-418-6161 11
  12. 12. Extremely un-stovepiped sensemaking! scene recognition environment map with objects scenes help define objects map helps to find objects object recognition object help define scenes objects help orient (e.g. Citgo) to: obstacle avoidance segmentation optic flow stereo & depth coarse view bias where to look (data acquisition) Copyright 2013 Neurala, Inc. ● Proprietary and Confidential ● www.neurala.com ● +1-617-418-6161 12
  13. 13. Feedforward and feedback processing One key aspect of neuromorphic computing is feedback processing Feedback happens all over the brain Brain is a massively recurrent architecture E.g. in the visual system, for each output fiber from thalamus to the visual cortex, the thalamus receives ~10 feedback fibers 10X What is this feedback for? Copyright 2013 Neurala, Inc. ● Proprietary and Confidential ● www.neurala.com ● +1-617-418-6161 13
  14. 14. Mark coming to the conference room New speakers (Auditory) Desk 0.5 cm lower (Proprioceptive) Leave your desk.. New post-it (Visual) ..walk to the conference room.. Different door knob (Tactile) Chanel #5 smell ..sit in the conference room (Olfactory) Feedback information flow: Feedforward information flow: - leave the desk for theare MATCHING memory-based expectations with sensory At every moment, you talk - walk down the corridor to the conference room (and other cortical) INPUT - find the conference room, find a free spot as close to the exit as humanely EXPECTATION EXPECTATION EXPECTATION EXPECTATION EXPECTATION possible INPUT INPUT Copyright 2013 Neurala, Inc. INPUT ● Proprietary and Confidential INPUT ● INPUT www.neurala.com ● +1-617-418-6161 14
  15. 15. Feedback focuses on anomalous information The president of the United States of America & ? ? Match memory with current input Automatically and effortlessly focus on crucial differences Copyright 2013 Neurala, Inc. ● Proprietary and Confidential ● www.neurala.com ● +1-617-418-6161 15
  16. 16. Predictive feedback is ubiquitous in the brain The sentence “eel is on the ...” is preceded by white noise orange wagon wheel is on the wagon shoe eel is on the ... peel is on the orange heel is on the shoe Percept may be determined by the sound that one expects to hear in auditory context based on previous language experiences Expectations amplify consistent components of the white noise, while suppressing inconsistent components Copyright 2013 Neurala, Inc. ● Proprietary and Confidential ● www.neurala.com ● +1-617-418-6161 16
  17. 17. Adaptive Resonance Theory MATCH ADAPTIVE RESONANCE THEORY (ART) ATTENTIONAL SUBSYSTEM F2 ORIENTING SUBSYSTEM Σ Σ Y RESET y + Math behind neuromorphic systems? + 𝑑𝑦 𝑗 = −𝐴𝑦 𝑗 + (𝐵 − 𝑦 𝑗 ) 𝑑𝑡 + F1 - MISMATCH - X + MATCHING CRITERION Σ w 𝑛 𝑤 𝑖,𝑗 𝑥 𝑖 − (𝑦 𝑗 + 𝐶) 𝑖=0 𝑦𝑘 𝑘≠𝑗 x 𝑑𝑤 𝑖,𝑗 = −𝐷𝑦 𝑗 𝑤 𝑖,𝑗 + 𝐸𝑥 𝑖 𝑦 𝑗 𝑑𝑡 + Copyright 2013 Neurala, Inc. 𝑛 ● Proprietary and Confidential ● www.neurala.com ● +1-617-418-6161 17
  18. 18. With Mark Motter, NASA Langle NASA STTR Phase II Copyright 2013 Neurala, Inc. ● Proprietary and Confidential ● www.neurala.com ● +1-617-418-6161 18
  19. 19. One algorithm for all robots ~32M neurons, ~13B synapses Sensory Classification Prefrontal Cortex Binding What & Where SelfLocalization Unclear Curiosity Drive Unclear Multiple Areas Complex Cells V2 Oriented Edges V1 Luminance LGN PreProcessing Retina Reverse Spread Prefrontal Ctx Forward Spread CA3 Color Opponency LGN? Reward Signal Basal Ganglia Dentate Gyrus Saliency Map Posterior PC Active Goal Location Prefrontal Ctx Forward Spread Trigger IT Reward Location Map Unclear Current Location Enthorinal Ctx PositionInvariant Cells Goal Selection Map Unclear Desired Next Location CA1 Path-Finding Reset Lack of Comfort Drive Unclear Motivation Desired Heading Cingulate Ctx Motor Turn Command Motor Cortex Medial Septum Navigation Motor Run Command Motor Cortex Virtual Environment or Robotic Platform Copyright 2013 Neurala, Inc. ● Proprietary and Confidential ● www.neurala.com ● +1-617-418-6161 19
  20. 20. Example application: MoNETA Morris water maze Goal Selection map MoNETA’s visual input 1st run 4th run 6th run Brain diagram 8th run Copyright 2013 Neurala, Inc. ● Proprietary and Confidential ● Reverse spread www.neurala.com ● +1-617-418-6161 Familiarity map Obstacle map 20
  21. 21. One algorithm for all robots Copyright 2013 Neurala, Inc. ● Proprietary and Confidential ● www.neurala.com ● +1-617-418-6161 21
  22. 22. Simulated Mars Rover Copyright 2013 Neurala, Inc. ● Proprietary and Confidential ● www.neurala.com ● +1-617-418-6161 22
  23. 23. Feedback improves heading estimation fusion (1) (2) Vestibular gyro & accel. Source Vestibular Visual Motor Integrated ring Copyright 2013 Neurala, Inc. Average error (deg) 62.8 35.9 29.9 14.9 ● Proprietary and Confidential ● Visual Motor optic flow wheel velocity Avg. error rate (deg/hr) 1.63 0.916 0.983 0.111 www.neurala.com ● +1-617-418-6161 23
  24. 24. How should robots “see”? Copyright 2013 Neurala, Inc. ● Proprietary and Confidential ● www.neurala.com ● +1-617-418-6161 24
  25. 25. Copyright 2013 Neurala, Inc. ● Proprietary and Confidential ● www.neurala.com ● +1-617-418-6161 25
  26. 26. Biological vision is: Selective (non-uniform sampling) Multimodal (focus vision on speaker) Predictive (look where the ball is going) • computational savings (up to 30x documented vs. uniformly sampled images) • • Traver and Bernardino (2010) Copyright 2013 Neurala, Inc. ● Proprietary and Confidential ● conformal mapping some invariance to changes in scale and rotation www.neurala.com ● +1-617-418-6161 26
  27. 27. Artificial visual system object/scene exogenous spatial attention label cognitive biases WHAT WHERE learning control what surface representations where Visual System Preprocessing eye movements Copyright 2013 Neurala, Inc. ● input image Proprietary and Confidential ● www.neurala.com ● +1-617-418-6161 27
  28. 28. Learning to classify Martian rocks Copyright 2013 Neurala, Inc. ● Proprietary and Confidential ● www.neurala.com ● +1-617-418-6161 28
  29. 29. Expanding the visual system Temporal continuity: “how do you know that the thing you are looking at now is the same you look at 1 second ago?” Upcoming development under the STTR Phase II program: 1) Depth processing, optic-flow aided segmentation, scene “gists” 2) Implementation on robotic platforms Copyright 2013 Neurala, Inc. ● Proprietary and Confidential ● www.neurala.com ● +1-617-418-6161 29
  30. 30. Robotic applications Copyright 2013 Neurala, Inc. ● Proprietary and Confidential ● www.neurala.com ● +1-617-418-6161 30
  31. 31. One brain for all robots: UAVs? Video courtesy of Mark Motter Copyright 2013 Neurala, Inc. ● Proprietary and Confidential ● www.neurala.com ● +1-617-418-6161 31
  32. 32. Optic flow as a collision clue Copyright 2013 Neurala, Inc. ● Proprietary and Confidential ● www.neurala.com ● +1-617-418-6161 32
  33. 33. Sense and Avoid from Flow Expansion SAFE model with Mark Motter, NASA Langley, CIF 2012-13 past environment (s) - action (a) pair reward/punishment learning rate Copyright 2013 Neurala, Inc. ● Proprietary and Confidential ● www.neurala.com ● +1-617-418-6161 value current state/action random action 33
  34. 34. Obstacle avoidance via optic flow learning Collision rate with other UAVs in collision path Cost of crashing ~= cost of deviating course Cost of crashing >> cost of deviating course Each point = MAvg over 200 trials Copyright 2013 Neurala, Inc. ● Proprietary and Confidential ● www.neurala.com ● +1-617-418-6161 34
  35. 35. Obstacle avoidance via optic flow learning Rate at which the UAV turns off-course in the absence of a colliding UAV Cost of crashing ~= cost of deviating course Cost of crashing >> cost of deviating course Each point = MAvg over 200 trials Copyright 2013 Neurala, Inc. ● Proprietary and Confidential ● www.neurala.com ● +1-617-418-6161 35
  36. 36. Obstacle avoidance via optic flow learning Rate at which the UAV turns off-course after a UAV has passed Cost of crashing ~= cost of deviating course Cost of crashing >> cost of deviating course Each point = MAvg over 200 trials Copyright 2013 Neurala, Inc. ● Proprietary and Confidential ● www.neurala.com ● +1-617-418-6161 36
  37. 37. Identifying other UAVs: virtual environment Copyright 2013 Neurala, Inc. ● Proprietary and Confidential ● www.neurala.com ● +1-617-418-6161 37
  38. 38. Identifying other UAVs: real video Copyright 2013 Neurala, Inc. ● Proprietary and Confidential ● www.neurala.com ● +1-617-418-6161 38
  39. 39. NASA STTR Phase II timeline 1a) Develop navigation and space variant vision on land robot; 1b) Develop collision avoidance for UAV; 2) Merge 1 & 2 on UAV; How do pilots look at things? Copyright 2013 Neurala, Inc. ● Proprietary and Confidential ● www.neurala.com ● +1-617-418-6161 39
  40. 40. Three ingredients for success Smart mind Powerful brain Copyright 2013 Neurala, Inc. Inexpensive body ● Proprietary and Confidential ● www.neurala.com ● +1-617-418-6161 40
  41. 41. The brain = + 100 billion neurons 250 trillion synapses Copyright 2013 Neurala, Inc. ● Proprietary and Confidential ● www.neurala.com ● +1-617-418-6161 41
  42. 42. Brain power + Copyright 2013 Neurala, Inc. ● Proprietary and Confidential ● www.neurala.com ● +1-617-418-6161 42
  43. 43. Brains, GPUs, multicore CPUs Copyright 2013 Neurala, Inc. ● Proprietary and Confidential ● www.neurala.com ● +1-617-418-6161 43
  44. 44. Videogames and smartphones + = Graphic Processing Units (GPUs), now in mobile Copyright 2013 Neurala, Inc. ● Proprietary and Confidential ● www.neurala.com ● +1-617-418-6161 44
  45. 45. Parallel vs. serial GPU CPU Copyright 2013 Neurala, Inc. ● Proprietary and Confidential ● www.neurala.com ● +1-617-418-6161 45
  46. 46. digital 48? 25? 12? 2017 2016 4 6 24M 2015 0.65V 2014 12M 2013 0.78V 2012 2018 “fancy” 8M 0.87V 2012 2023 2017 Copyright 2013 Neurala, Inc. ● Proprietary and Confidential # neurons/cm2, power supply GFLOPS per Watt (GPU) & # mobile CPU cores Computing power roadmap ● www.neurala.com ● +1-617-418-6161 46
  47. 47. Mouse brain # neurons/cm2, power supply 2018 3 cm2 2015 5V 6 cm2 28 V 2012 2023 2017 Copyright 2013 Neurala, Inc. ● Proprietary and Confidential ● www.neurala.com ● +1-617-418-6161 47
  48. 48. Human brain # neurons/cm2, power supply 2018 35 m2 2015 2 kV 70 m2 5 kV 2012 2023 2017 Copyright 2013 Neurala, Inc. ● Proprietary and Confidential ● www.neurala.com ● +1-617-418-6161 48
  49. 49. Faster than we think? The Human Brain Project $100M/yr. 2013-23 BRAIN Initiative, projected $300M/yr. 10 yrs. 2012 2023 2017 Copyright 2013 Neurala, Inc. ● Proprietary and Confidential ● www.neurala.com ● +1-617-418-6161 49
  50. 50. Not only academia! The NPU will "not only mimic human-like perception but also have the ability to learn how biological brains do” * See also IBM, Intel Copyright 2013 Neurala, Inc. ● Proprietary and Confidential ● www.neurala.com ● +1-617-418-6161 50
  51. 51. Towards autonomous machines Smart mind Powerful brain Copyright 2013 Neurala, Inc. Inexpensive body ● Proprietary and Confidential ● www.neurala.com ● +1-617-418-6161 51
  52. 52. Outreach: Boston Museum of Science 1.5 million visitors: the most visited cultural attraction in Boston Copyright 2013 Neurala, Inc. ● Proprietary and Confidential ● www.neurala.com ● +1-617-418-6161 52
  53. 53. NASA?...no, MASA! Copyright 2013 Neurala, Inc. ● Proprietary and Confidential ● www.neurala.com ● +1-617-418-6161 53
  54. 54. ~1800 visitors used the app Snow inches by day - 2014 1.2 1 0.8 0.6 0.4 0.2 0 Copyright 2013 Neurala, Inc. ● Proprietary and Confidential ● www.neurala.com ● +1-617-418-6161 54
  55. 55. Neurala in partnership with Massimiliano Versace CEO, Neurala Inc. www.neurala.com versace@neurala.com Copyright 2013 Neurala, Inc. ● Proprietary and Confidential ● www.neurala.com ● +1-617-418-6161 55
  56. 56. Where pathway To What Pathway Reset Shroud Fit Shroud (log-polar) (log-polar) Hot Spot (log-polar) Hot Spot (retinal) Early Vision Processing Working memory (Linear) Working memory Eye mov. Contours Diffusion Attention Shroud Working Memory Eye/Head Movement (head cent.) (Linear) Working memory Eye/head mov. kernel (Linear) Copyright 2013 Neurala, Inc. ● Proprietary and Confidential ● Edges Input (log-polar) www.neurala.com ● +1-617-418-6161 56 56
  57. 57. Where pathway log-polar input representation reset signal (different object) shroud forming (bottom) and fully formed (top) hot-spot working memory Copyright 2013 Neurala, Inc. ● Proprietary and Confidential ● new hot spot 57 (where the eye will look next) 57 www.neurala.com +1-617-418-6161 ●
  58. 58. What pathway Associative learning Core of “what” pathway ART learning Recurrent excitation Spatial shroud exists Where reset Inhibition From external source Top-down priming Teacher Name Category Object Category Boundary/ completed Shroud from Where pathway From “where” pathway View Category Copyright 2013 Neurala, Inc. ● Generates a name for the foveated object (unsupervised). Can also take top down name assigned by external teacher (supervised) Groups multiple views of a single object into an object category. Reset of object category is gated by Where signal Groups similar log-polar views of same object into view categories Proprietary and Confidential ● www.neurala.com ● +1-617-418-6161 58
  59. 59. What pathway top-down teaching signal bottom-up activated name category view-specific neuron weights learning multiple views of the same digit current input viewspecific neurons Copyright 2013 Neurala, Inc. ● Proprietary and Confidential ● Confidential www.neurala.com +1-617-418-6161 Neurala – AFRL site visit, ● 59 59 59
  60. 60. Semantic influence on data collection ? initial hypothesis new instance (ambiguous view) 5 invariance 0 1 “heat maps” 0 5 0 1 2 … 5 2 3 4 learning 5 Where to look next to maximize discrimination 6 “heat maps” 7 8 9 Few presentations Copyright 2013 Neurala, Inc. ● Proprietary and Confidential ● www.neurala.com ● +1-617-418-6161 60
  61. 61. Obstacle avoidance via optic flow learning Collision rate with otherwise “missed” UAVs Cost of crashing ~= cost of deviating course Cost of crashing >> cost of deviating course Each point = MAvg over 200 trials Copyright 2013 Neurala, Inc. ● Proprietary and Confidential ● www.neurala.com ● +1-617-418-6161 61

×