Neuromorphic Digial Chip

1. Motivation Spiking Neural Network (SNN) SNN Learning Learning Engine Architecture and Design Implementation Results
Loihi: A Neuromorphic Manycore Processor with On-Chip
Learning
Neuromorphic Computing
Mehmood Saleem
Chair of Highly-Parallel VLSI Systems and Neuro-Microelectronics
July 12, 2019
Mehmood Saleem TU - Dresden Loihi: A Neuromorphic Manycore Processor July 12, 2019 1 / 29

2. Motivation Spiking Neural Network (SNN) SNN Learning Learning Engine Architecture and Design Implementation Results
Outline
1 Motivation
2 Spiking Neural Network (SNN)
3 SNN Learning
4 Learning Engine
5 Architecture and Design Implementation
6 Results
Mehmood Saleem TU - Dresden Loihi: A Neuromorphic Manycore Processor July 12, 2019 2 / 29

3. Motivation Spiking Neural Network (SNN) SNN Learning Learning Engine Architecture and Design Implementation Results
Motivation
Mehmood Saleem TU - Dresden Loihi: A Neuromorphic Manycore Processor July 12, 2019 3 / 29

4. Motivation Spiking Neural Network (SNN) SNN Learning Learning Engine Architecture and Design Implementation Results
Motivation
Loihi (Lo-ee-hee), a 60 mm2 chip with 14 nm
technology.
Implementation of Spiking Neural Networks
(SNN) on digital circuit.
Novel Features
Programmable Synaptic Learning rules.
Hierarchical Connectivity.
Dendritic Compartments and Synaptic Delays.
[DSL+18] Loihi Chip
Mehmood Saleem TU - Dresden Loihi: A Neuromorphic Manycore Processor July 12, 2019 4 / 29

5. Motivation Spiking Neural Network (SNN) SNN Learning Learning Engine Architecture and Design Implementation Results
Spiking Neural Network (SNN)
Mehmood Saleem TU - Dresden Loihi: A Neuromorphic Manycore Processor July 12, 2019 5 / 29

6. Motivation Spiking Neural Network (SNN) SNN Learning Learning Engine Architecture and Design Implementation Results
SNN as Main Processing Element
Dynamical System
Interaction of Neurons through spikes.
Communication of Neuron through directed links,
Synapses.
Triggers an impulse with the exceeding threshold of
state variables.
CUBA Leaky-Integration And Fire (LIAF) Model
Two state variables synapse current isyn,i(t) and
membrane potential umem defines the local state of
Neuron.
isyn,i (t) =
j=i
wij (σj (t) ·
1
τu
e
−t
τu )H(t) + bsyn
where, σ for kth impulse defined as the sum of the dirac
functions
σ(t) =
k
δ(t − tk )
Figure: Ion Channels [hCMP19]
Figure: Neuron Block [hCMP19]
Mehmood Saleem TU - Dresden Loihi: A Neuromorphic Manycore Processor July 12, 2019 6 / 29

7. Motivation Spiking Neural Network (SNN) SNN Learning Learning Engine Architecture and Design Implementation Results
Spike Firing
Membrane Potential
Integration of synapse current, to get membrane potential (umem).
dumem,i
dt
=
1
τu
umem,i (t) + isyn,i (t) − θi · σi (t)
Firing of the spike, once membrane potential exceed the threshold θi (t), and reset
to zero the after the occurrence of event.
Digital Implementation
Implement the continuous behaviour of LIAF on digital using discrete time-step
model.
All neurons will maintain a consistent understanding of time.
WK
σK αk
ik
bk
U(0)mem
ik
Discrete Time
Integrator
σK.ϴ
Uk,mem
Mehmood Saleem TU - Dresden Loihi: A Neuromorphic Manycore Processor July 12, 2019 7 / 29

8. Motivation Spiking Neural Network (SNN) SNN Learning Learning Engine Architecture and Design Implementation Results
SNN Feature
Solve l1-minimizing sparse coding problem
(LASSO) using Spiking Locally Competitive
Algorithm (S-LCA) and Fast Iterative
Shrinkage-Threshold Algorithm (FISTA).
Determination of spare set of coefficients which
represents the input as linear combination of
features from a feature dictionary.
Sparse Approximation
S-LCA FISTA
One-many spike com-
munication
Matrix arithmetic-based
solutions
Fast, good approxima-
tion at start-up and Flat-
tened in transient
Slow approximation at
start-up, better approxi-
mation in transient.
S-LCA: Firing rate converges to one fixed
point, good for approximate results.
FISTA: Precise Solution.
Figure: Evolution of network dynamics
FigureMehmood Saleem TU - Dresden Loihi: A Neuromorphic Manycore Processor July 12, 2019 8 / 29

9. Motivation Spiking Neural Network (SNN) SNN Learning Learning Engine Architecture and Design Implementation Results
SNN Feature
Novel Features
Parallelism compared to sequential in conventional CPU.
Neuron development withing time-step.
Concurrent evolution of a neuron states.
Challenge
Support of parallelism in underlying architecture.
Mehmood Saleem TU - Dresden Loihi: A Neuromorphic Manycore Processor July 12, 2019 9 / 29

10. Motivation Spiking Neural Network (SNN) SNN Learning Learning Engine Architecture and Design Implementation Results
SNN Learning
Mehmood Saleem TU - Dresden Loihi: A Neuromorphic Manycore Processor July 12, 2019 10 / 29

11. Motivation Spiking Neural Network (SNN) SNN Learning Learning Engine Architecture and Design Implementation Results
SNN Learning
SNN Learning
Adaption of synaptic weights with time.
Reduce the loss of function over training samples.
Locality Constraint
Synapse weight access and modified by destination neuron.
Requirement of decentralized algorithm to update the weights.
Rule use only locally information e.g, spike from pre-synapse.
Mehmood Saleem TU - Dresden Loihi: A Neuromorphic Manycore Processor July 12, 2019 11 / 29

12. Motivation Spiking Neural Network (SNN) SNN Learning Learning Engine Architecture and Design Implementation Results
SNN Learning
Loihi Features
Locality information for programmable synaptic learning process.
Configurable time-constants to filter pre- and post-synapse spike trains.
Short time constants => learning rule by uti-
lize precise-timing.
Long time-constants => Captures spike
rates.
Value Added Features
Two additional variables/synapse and can be used for the reinforcement of
learning.
To enforce the learning, for the purpose of punishment or reward signals. Special
reward traces are used that are signed impulse signals.
Mehmood Saleem TU - Dresden Loihi: A Neuromorphic Manycore Processor July 12, 2019 12 / 29

13. Motivation Spiking Neural Network (SNN) SNN Learning Learning Engine Architecture and Design Implementation Results
Additional Features
Addition of Random Number
It supports the learning by estimating the probability of the event or probabilistic
inference.
Random numbers are added. intentionally to the synapse current, membrane
voltage and refractory delays.
It also supports the sampling algorithms for neural networks.
Dendritic Tree Processing
SNN neurons decomposed into tree of compartment units.
Input synapse shared within all the compartment units with single spike
generating unit as root.
The state variables of neuron belongs to each compartment unit.
All the state variables are re-configurable to promote the functions between the
compartments.
Mehmood Saleem TU - Dresden Loihi: A Neuromorphic Manycore Processor July 12, 2019 13 / 29

14. Motivation Spiking Neural Network (SNN) SNN Learning Learning Engine Architecture and Design Implementation Results
Learning Engine
Mehmood Saleem TU - Dresden Loihi: A Neuromorphic Manycore Processor July 12, 2019 14 / 29

15. Motivation Spiking Neural Network (SNN) SNN Learning Learning Engine Architecture and Design Implementation Results
Pairwise Spike-timing dependent plasticity (STDP)
STDP is simple, event-driven and can be implemented to
hardware.
For Pre-Post synapse in the figure, spike timings of two neurons
must be maintained.
The synapses weight update based on the following expression:
∆ωij =
A− · F(t − ti
post), pre-synaptic
A+ · F(t − tj
pre), post-synaptic
Weight Update
The first condition is easily get, as the weight was pre-computed.
For firing of postsynaptic neuron, backward routing look-up
needed to fulfill the second condition.
Constraint
Algorithm for reverse lookup is a topic of active research and
limit network topologies.
Figure: Pre-Synape
and Post-
Synapse
[SG10]
Mehmood Saleem TU - Dresden Loihi: A Neuromorphic Manycore Processor July 12, 2019 15 / 29

16. Motivation Spiking Neural Network (SNN) SNN Learning Learning Engine Architecture and Design Implementation Results
Pairwise Spike-timing dependent plasticity (STDP)
Introduce epoch time
Delay all synapse state to the end of periodic learning Tepoch.
Must to iterated over each core’s input axons.
This chip perform this task sequentially. Therefore, triggered at
the edges of epoch time.
Issue
Direct implementation of reverse look-up table must be avoided.
As the architecture has pipeline stages which must need to
iterate over input axons to maintain the spike timings.
Mehmood Saleem TU - Dresden Loihi: A Neuromorphic Manycore Processor July 12, 2019 16 / 29

17. Motivation Spiking Neural Network (SNN) SNN Learning Learning Engine Architecture and Design Implementation Results
Pairwise Spike-timing dependent plasticity (STDP)
Synapse update at every epoch time.
At the fulfillment of pre-/post-synapse conditions.
Microcode operations for the determination of
one-more transformation of synapse state’s
variables.
Rule Set
Encoding Term (Ti,j ) Description
0 xo + C Pre-synaptic spike count.
1 x1 + C 1st Pre-synaptic trace.
2 x2 + C 2nd Pre-synaptic trace.
3 yo + C Post-synaptic spike count
4 y1 + C 1st Post-synaptic trace.
5 y2 + C 2nd Post-synaptic trace.
.. ..
9 wgt + C Synaptic weight.
10 dly + C Synaptic delay.
11 tag + C Synaptic tag.
.. ..
Transformed Synaptic Variable
Z := Z +
i
Si
ni
j
(Vij + Cij )
Where, Vij + Cij represents the product
term and denoted as (Ti,j ), Z
represents the transform synaptic
variable (wgt , dly , tag) and Cij , Si are
the unicode specied constants
Mehmood Saleem TU - Dresden Loihi: A Neuromorphic Manycore Processor July 12, 2019 17 / 29

18. Motivation Spiking Neural Network (SNN) SNN Learning Learning Engine Architecture and Design Implementation Results
Architecture and Design Implementation
Mehmood Saleem TU - Dresden Loihi: A Neuromorphic Manycore Processor July 12, 2019 18 / 29

19. Motivation Spiking Neural Network (SNN) SNN Learning Learning Engine Architecture and Design Implementation Results
Architecture
The chip offers off-chip interfaces for the extension
of mesh in four planar direction with other chips.
Network-on-chip (NOC) supports write, read
request and response messages for
core-management and spike messages for SNN
computation.
Time synchronisation achieved with the aid of
barrier messages between cores.
Neural Network Units
Each neuromorphic core implements 1024 SNN
units which are grouped into a set of trees as
constituting neurons.
3 x x86
processors
Off-Chip Interface
Mesh of 128
Neuromorphic
Cores
Communication
between cores
in packetized
msgs
Figure: Architecture of Loihi
Mehmood Saleem TU - Dresden Loihi: A Neuromorphic Manycore Processor July 12, 2019 19 / 29

20. Motivation Spiking Neural Network (SNN) SNN Learning Learning Engine Architecture and Design Implementation Results
Architecture Features
Sparse Network Compression
The chip supports 3 sparse matrix compression
models. Fan-out neuron indices computed based
on the index-state stored with each synapse.
Core-Core multicast communication.
Variable synaptic formats, weigth precision from 1 -
9 signed/un-signed.
Population based hierarchy connectivity
For the support of convolutional neural networks it
supports weight sharing mechanisms.
Population based instances mapped to specific
connectivity templates to reduced the overall
network connectivity.
Neuron
Core
Core
Core
Figure: Multicasting
Mehmood Saleem TU - Dresden Loihi: A Neuromorphic Manycore Processor July 12, 2019 20 / 29

21. Motivation Spiking Neural Network (SNN) SNN Learning Learning Engine Architecture and Design Implementation Results
Design Implementation
To increase the throughput parallel flow attain using pipe-lining, by breaking large
event break into small.
Pre-synaptic trace is stored in SYN_MEM with SYN_MAP which increases the
multiple serial access per entry of spike. Hence, balances the pipeline throughput.
Asynchronous design pattern implemented by distributing the Read-Memory-Write
state of over single-ported SRAM banks. Increased performance.
Figure: Top-Level Micro-architecture of Core [DSL+
18]
Mehmood Saleem TU - Dresden Loihi: A Neuromorphic Manycore Processor July 12, 2019 21 / 29

22. Motivation Spiking Neural Network (SNN) SNN Learning Learning Engine Architecture and Design Implementation Results
Results
Mehmood Saleem TU - Dresden Loihi: A Neuromorphic Manycore Processor July 12, 2019 22 / 29

23. Motivation Spiking Neural Network (SNN) SNN Learning Learning Engine Architecture and Design Implementation Results
Realization
Illustration of energy and performance of
pre-silicon Loihi.
Performance Evaluation
Parameter Value at
0.75 V
Cross-Sectional spike band-
width/tile
3.44 Gb/s
Energy/synaptic-spike Op
(min)
23.6 pJ
Time/synaptic-spike Op
(max)
3.5 ns
Energy/neuron Update (ac-
tive/inactive)
81 pJ / 52 pJ
Energy/neuron Update (ac-
tive/inactive)
8.4 ns / 5.3 ns
33 MB SRAM
14 nm FinFET
Proccess
2.1 billion FET
128 Neuro
Cores
Figure: 60 mm2
LoihiChip
Mehmood Saleem TU - Dresden Loihi: A Neuromorphic Manycore Processor July 12, 2019 23 / 29

24. Motivation Spiking Neural Network (SNN) SNN Learning Learning Engine Architecture and Design Implementation Results
State-of-the-art
Comparison of Intel Loihi with state-of-the-art IBM
True North.
Performance Evaluation
Chip Synaptic
Variables
Neuron
Density
Number
of Neuron
Cores
million/mm2 /mm2
TrueNorth 0.7 1,024 256
Loihi 2.1 2,184 128
Figure: 60 mm2
LoihiChip
Figure: TrueNorth Chip [ASC+
15]
Mehmood Saleem TU - Dresden Loihi: A Neuromorphic Manycore Processor July 12, 2019 24 / 29

25. Motivation Spiking Neural Network (SNN) SNN Learning Learning Engine Architecture and Design Implementation Results
Benchmark of Chip
Sparse coding problem evaluated on 52x52
image, with 224-atom dictionary and 8x8 patch
size with 4 pixels of patch stradel.
Loihi provides a factor of 18 compression in
synaptic resources.
Solve problem within 1% of optimal solution. Figure: Image reconstruction from the sparse
coefficients computed using the Loihi
predecessor Chip [DSL+
18]
Mehmood Saleem TU - Dresden Loihi: A Neuromorphic Manycore Processor July 12, 2019 25 / 29

26. Motivation Spiking Neural Network (SNN) SNN Learning Learning Engine Architecture and Design Implementation Results
Application Comparison
Comparison of solving l1 minimization on Loihi and Atom
Two well-known algorithms (LARS, FISTA) execute on Atom CPU at 1.67 GHz.
The following table shows the ratio Atom/Loihi, from Atom the best number
between two algorithms has been used.
Computational efficiency of Loihi and Atom (Atom/Loihi)
Number of Unknowns 400 1,700 32,256
Number of non-zeros in solu-
tions
≈ 10 ≈30 ≈420
Energy 2.58x 8.08x 48.74x
Delay 0.27x 2.76x 118.18x
EDP 0.7x 22.33x 5760x
Mehmood Saleem TU - Dresden Loihi: A Neuromorphic Manycore Processor July 12, 2019 26 / 29

27. Motivation Spiking Neural Network (SNN) SNN Learning Learning Engine Architecture and Design Implementation Results
Conclusion and Further Scope
To support further applications
Intel fifth and most complex chip in the domain of neuromorphic system.
Remarkable features for neuromorphic applications.
Proof-of-concept for variety of machine learning applications.
Further evaluation of different network applications and scaling of the architecture.
Mehmood Saleem TU - Dresden Loihi: A Neuromorphic Manycore Processor July 12, 2019 27 / 29

28. Motivation Spiking Neural Network (SNN) SNN Learning Learning Engine Architecture and Design Implementation Results
Take Home Message
Loihi, A digital chip for neuromorphic systems
The abundant parallelism of Neuromporphic systems is the key for
Neuromporphic applications and poorly served by CPU.
Spiking Neural networks are the best suited for machine learning applications and
energy efficient.
Digital area seems to be the promising for Neuromporphic system, by leveraging
scaled CMOS technologies.
Adapted the synapse weights is a challenge for plasticity.
Mehmood Saleem TU - Dresden Loihi: A Neuromorphic Manycore Processor July 12, 2019 28 / 29

29. Motivation Spiking Neural Network (SNN) SNN Learning Learning Engine Architecture and Design Implementation Results
Bibliography
F. Akopyan, J. Sawada, A. Cassidy, R. Alvarez-Icaza, J. Arthur, P. Merolla,
N. Imam, Y. Nakamura, P. Datta, G. Nam, B. Taba, M. Beakes, B. Brezzo, J. B.
Kuang, R. Manohar, W. P. Risk, B. Jackson, and D. S. Modha.
TrueNorth: Design and Tool Flow of a 65 mW 1 Million Neuron Programmable
Neurosynaptic Chip.
IEEE Transactions on Computer-Aided Design of Integrated Circuits and
Systems, 34(10):1537–1557, Oct 2015.
M. Davies, N. Srinivasa, T. Lin, G. Chinya, Y. Cao, S. H. Choday, G. Dimou,
P. Joshi, N. Imam, S. Jain, Y. Liao, C. Lin, A. Lines, R. Liu, D. Mathaikutty,
S. McCoy, A. Paul, J. Tse, G. Venkataramanan, Y. Weng, A. Wild, Y. Yang, and
H. Wang.
Loihi: A Neuromorphic Manycore Processor with On-Chip Learning.
IEEE Micro, 38(1):82–99, January 2018.
Prof. Dr.-Ing. habil. Christian Mayr and Dr.-Ing. Johannes Partzsch.
Neuromorphic VLSI Systems Lecture.
Technische Universität Dresden, pages 23, 87, 2019.
J. Sjöström and W. Gerstner.
Spike-timing dependent plasticity.
Scholarpedia, 5(2):1362, 2010.
revision #184913.
Mehmood Saleem TU - Dresden Loihi: A Neuromorphic Manycore Processor July 12, 2019 29 / 29