chapter 5.pptx: drainage and irrigation engineering
Eyeriss Introduction
1. Introduction to Eyeriss1
Michael (Tao-Yi) Lee
tylee@mlpanda.rocks
NTU IoX Center
October 24, 2017
1Y. H. Chen et al. “Eyeriss: An Energy-Efficient Reconfigurable Accelerator for Deep
Convolutional Neural Networks”. In: IEEE Journal of Solid-State Circuits 52.1 (Jan. 2017),
pp. 127–138.
2. Outline
1 Introduction
2 Eyeriss Highlights
Memory Hierarchy
Row Stationary Data Flow
Network-on-a-chip (NoC)
Compression and Data Gating
3 Summary
4 Appendix
Michael Lee (NTU) Introduction to Eyeriss October 24, 2017 2 / 37
3. Introduction
Contributions of Eyeriss
A novel energy-efficient CNN dataflow that has been verified in
a fabricated chip
A taxonomy of CNN dataflows that classifies previous work into
three categories (WS, OS, NLR)
Figure: Eyeriss Die Photo (35 fps @ 278 mW running AlexNet[10])
4000µm
4000µm
168 PE
GlobalBuffer
Michael Lee (NTU) Introduction to Eyeriss October 24, 2017 3 / 37
4. Introduction
Features of Eyeriss
Use row stationary (RS) on spatial architecture with 168
processing elements to reduce energy cost of data flow
4 level memory hierachy: Maximally local data reuse
Network-on-a-chip (NoC)
Multicast
P2P single cycle delievery
Compression and data gating
Run-length compression (RLC)
PE data gating
Michael Lee (NTU) Introduction to Eyeriss October 24, 2017 4 / 37
6. Introduction
Recap on CNN
Forward computation in CONV layers
Given ofmap O, ifmap I, bias B, weight W, stride size U
O[z][u][x][y] =ReLU B[u] +
C−1
k=0
R−1
i=0
S−1
j=0
I[z][k][Ux + i][Uy + j] × W[u][k][i][j]
partial sum
(1)
where 0 ≤ z < N, 0 ≤ u < M, 0 ≤ y < E, 0 ≤ x < F
E = (H − R + U)/U (2)
F = (W − S + U)/U (3)
Michael Lee (NTU) Introduction to Eyeriss October 24, 2017 6 / 37
7. Eyeriss Highlights Memory Hierarchy
PE Matrix and Memory Hierarchy
1. Spatial Architecture: Allows data to flow in four directions
2. PE operates independently with one CLKcore (i.e. not systolic)
MAC
pixel
W
psumi
psumo
MAC
pixel
W
psumi
psumo
MAC
pixel
W
psumi
psumo
MAC
pixel
W
psumi
psumo
MAC
pixel
W
psumi
psumo
MAC
pixel
W
psumi
psumo
MAC
pixel
W
psumi
psumo
MAC
pixel
W
psumi
psumo
MAC
pixel
W
psumi
psumo
DRAM
Challenge
How to optimize data flow in order to minimize energy consumption?
Michael Lee (NTU) Introduction to Eyeriss October 24, 2017 7 / 37
8. Eyeriss Highlights Memory Hierarchy
PE Matrix and Memory Hierarchy (Eyeriss)
4 level memory hierachy
DRAM → Global Buffer (GLB) → Network-on-a-Chip (NoC) →
Register File (RF)
on-chip
RF(1kB)
(EC=1X)
RF(1kB)
(EC=1X)
RF(1kB)
(EC=1X)
RF(1kB)
(EC=1X)
RF(1kB)
(EC=1X)
RF(1kB)
(EC=1X)
RF(1kB)
(EC=1X)
RF(1kB)
(EC=1X)
RF(1kB)
(EC=1X)
NoC (EC=2X)
DRAM GLB
(EC=6X)
FIFO (EC2
=500X)
2Relative energy cost
Michael Lee (NTU) Introduction to Eyeriss October 24, 2017 8 / 37
9. Eyeriss Highlights Row Stationary Data Flow
CNN dataflows
Row Stationary
Weight Stationary (WS)
Output Stationary (OS)
No Local Reuse (NLR)
Michael Lee (NTU) Introduction to Eyeriss October 24, 2017 9 / 37
10. Eyeriss Highlights Row Stationary Data Flow
Comparison of Dataflows (I)
Focus on flows of psum, weight and pixels in next slides
RS uses 1.4X – 2.5X lower energy than other dataflows
Michael Lee (NTU) Introduction to Eyeriss October 24, 2017 10 / 37
11. Eyeriss Highlights Row Stationary Data Flow
Row Stationary in 1D Convolution
a b c
Kernel
∗
a b c d e
Image
=
a b c
PSum
Michael Lee (NTU) Introduction to Eyeriss October 24, 2017 11 / 37
12. Eyeriss Highlights Row Stationary Data Flow
Row Stationary in 1D Convolution
a b c
Kernel
∗
a b c d e
Image
=
a b c
PSum
Michael Lee (NTU) Introduction to Eyeriss October 24, 2017 11 / 37
13. Eyeriss Highlights Row Stationary Data Flow
Row Stationary in 1D Convolution
a b c
Kernel
∗
a b c d e
Image
=
a b c
PSum
Michael Lee (NTU) Introduction to Eyeriss October 24, 2017 11 / 37
14. Eyeriss Highlights Row Stationary Data Flow
Row Stationary in 1D Convolution (PE)
a b c
Kernel
∗ a b c d e
Image
= a b c
PSum
PEReg File
c b a
c b a
a
de
Michael Lee (NTU) Introduction to Eyeriss October 24, 2017 12 / 37
15. Eyeriss Highlights Row Stationary Data Flow
Row Stationary in 1D Convolution (PE)
a b c
Kernel
∗ a b c d e
Image
= a b c
PSum
PEReg File
d c b
c b a
b
e
a
Michael Lee (NTU) Introduction to Eyeriss October 24, 2017 12 / 37
16. Eyeriss Highlights Row Stationary Data Flow
Row Stationary in 1D Convolution (PE)
a b c
Kernel
∗ a b c d e
Image
= a b c
PSum
PEReg File
e d c
c b a
c b a
Michael Lee (NTU) Introduction to Eyeriss October 24, 2017 12 / 37
17. Eyeriss Highlights Row Stationary Data Flow
Row Stationary in 2D Convolution
1a 1b 1c
2a 2b 2c
3a 3b 3c
Kernel
∗
1a 1b 1c 1d 1e
2a 2b 2c 2d 2e
3a 3b 3c 3d 3e
4a 4b 4c 4d 4e
5a 5b 5c 5d 5e
Image
=
1a 1b 1c
2a 2b 2c
3a 3b 3c
PSum
Michael Lee (NTU) Introduction to Eyeriss October 24, 2017 13 / 37
18. Eyeriss Highlights Row Stationary Data Flow
Row Stationary in 2D Convolution
1a 1b 1c
2a 2b 2c
3a 3b 3c
Kernel
∗
1a 1b 1c 1d 1e
2a 2b 2c 2d 2e
3a 3b 3c 3d 3e
4a 4b 4c 4d 4e
5a 5b 5c 5d 5e
Image
=
1a 1b 1c
2a 2b 2c
3a 3b 3c
PSum
Michael Lee (NTU) Introduction to Eyeriss October 24, 2017 13 / 37
19. Eyeriss Highlights Row Stationary Data Flow
Row Stationary in 2D Convolution
1a 1b 1c
2a 2b 2c
3a 3b 3c
Kernel
∗
1a 1b 1c 1d 1e
2a 2b 2c 2d 2e
3a 3b 3c 3d 3e
4a 4b 4c 4d 4e
5a 5b 5c 5d 5e
Image
=
1a 1b 1c
2a 2b 2c
3a 3b 3c
PSum
Michael Lee (NTU) Introduction to Eyeriss October 24, 2017 13 / 37
21. Eyeriss Highlights Row Stationary Data Flow
Row Stationary in 2D Convolution (PE)
1a 2a 3a1b 2b 3b
PEReg File
3d 3c 3b
3c 3b 3a
1b
3e
1b
PEReg File
4d 4c 4b
3c 3b 3a
2b
4e
2b
PEReg File
5d 5c 5b
3c 3b 3a
3b
5e
3b
PEReg File
2d 2c 2b
2c 2b 2a
1b
2e
1b
PEReg File
3d 3c 3b
2c 2b 2a
2b
3e
2b
PEReg File
4d 4c 4b
2c 2b 2a
3b
4e
3b
PEReg File
1d 1c 1b
1c 1b 1a
1b
1e
1b
PEReg File
2d 2c 2b
1c 1b 1a
2b
2e
2b
PEReg File
3d 3c 3b
1c 1b 1a
3b
3e
3b
Michael Lee (NTU) Introduction to Eyeriss October 24, 2017 14 / 37
22. Eyeriss Highlights Row Stationary Data Flow
Row Stationary in 2D Convolution (PE)
1a 2a 3a1b 2b 3b1c 2c 3c
PEReg File
3e 3d 3c
3c 3b 3a
1c 1b 1a
PEReg File
4e 4d 4c
3c 3b 3a
2c 2b 2a
PEReg File
5e 5d 5c
3c 3b 3a
3c 3b 3a
PEReg File
2e 2d 2c
2c 2b 2a
1c 1b 1a
PEReg File
3e 3d 3c
2c 2b 2a
2c 2b 2a
PEReg File
4e 4d 4c
2c 2b 2a
3c 3b 3a
PEReg File
1e 1d 1c
1c 1b 1a
1c 1b 1a
PEReg File
2e 2d 2c
1c 1b 1a
2c 2b 2a
PEReg File
3e 3d 3c
1c 1b 1a
3c 3b 3a
Michael Lee (NTU) Introduction to Eyeriss October 24, 2017 14 / 37
23. Eyeriss Highlights Row Stationary Data Flow
Row Stationary in 2D Convolution (PE)
1a 2a 3a1b 2b 3b1c 2c 3c
PEReg File
3e 3d 3c
3c 3b 3a
1c 1b 1a
PEReg File
4e 4d 4c
3c 3b 3a
2c 2b 2a
PEReg File
5e 5d 5c
3c 3b 3a
3c 3b 3a
PEReg File
2e 2d 2c
2c 2b 2a
1c 1b 1a
PEReg File
3e 3d 3c
2c 2b 2a
2c 2b 2a
PEReg File
4e 4d 4c
2c 2b 2a
3c 3b 3a
PEReg File
1e 1d 1c
1c 1b 1a
1c 1b 1a
PEReg File
2e 2d 2c
1c 1b 1a
2c 2b 2a
PEReg File
3e 3d 3c
1c 1b 1a
3c 3b 3a
Psum propagate vertically Psum propagate vertically Psum propagate vertically
Psum propagate diagnally Pixel propagate diagnally Pixel propagate diagnally
Pixel propagate diagnally
Pixel propagate diagnally
Weight propagate horizontally
Weight propagate horizontally
Weight propagate horizontally
Michael Lee (NTU) Introduction to Eyeriss October 24, 2017 14 / 37
24. Eyeriss Highlights Row Stationary Data Flow
Weight Stationary (WS)3
Minimize weight read energy consumption
maximize convolutional and filter reuse of weights
Examples:
Chakradhar et al. 2010
Gokhale et al. 2014
Park et al. 2015
Cavigelli et al. 2015
3Image adopted from Yu-Hsin Chen, Joel Emer, and Vivienne Sze. ISCA 2016 Slides of
Eyeriss: A Spatial Architecture for Energy-Efficient Dataflow for Convolutional Neural Networks.
Michael Lee (NTU) Introduction to Eyeriss October 24, 2017 15 / 37
25. Eyeriss Highlights Row Stationary Data Flow
Output Stationary (OS)4
Minimize partial sum R/W energy consumption
maximize local accumulation
Examples:
Gupta et al. 2015
Du et al. 2015
Peemen et al. 2013
4Image adopted from Yu-Hsin Chen, Joel Emer, and Vivienne Sze. ISCA 2016 Slides of
Eyeriss: A Spatial Architecture for Energy-Efficient Dataflow for Convolutional Neural Networks.
Michael Lee (NTU) Introduction to Eyeriss October 24, 2017 16 / 37
26. Eyeriss Highlights Row Stationary Data Flow
No Local Reuse (NLR)5
Use a large global buffer as shared storage
Reduce DRAM access energy consumption
Examples:
Chen et al. 2014
Chen et al. 2014
Zhang et al. 2015
5Image adopted from Yu-Hsin Chen, Joel Emer, and Vivienne Sze. ISCA 2016 Slides of
Eyeriss: A Spatial Architecture for Energy-Efficient Dataflow for Convolutional Neural Networks.
Michael Lee (NTU) Introduction to Eyeriss October 24, 2017 17 / 37
27. Eyeriss Highlights Row Stationary Data Flow
Comparison of Dataflows (II)
RS reuses data in local register files (RF), a lot! ⇒ Saves energy of
moving data
Michael Lee (NTU) Introduction to Eyeriss October 24, 2017 18 / 37
28. Eyeriss Highlights Row Stationary Data Flow
Beyond 2D Convolution - Multiple Images
Processing in PE
Concatenate image rows
Michael Lee (NTU) Introduction to Eyeriss October 24, 2017 19 / 37
29. Eyeriss Highlights Row Stationary Data Flow
Beyond 2D Convolution - Multiple Filters
Processing in PE
Interleave filter rows
Michael Lee (NTU) Introduction to Eyeriss October 24, 2017 20 / 37
30. Eyeriss Highlights Row Stationary Data Flow
Beyond 2D Convolution - Multiple Channels
Processing in PE
Interleave channels
Michael Lee (NTU) Introduction to Eyeriss October 24, 2017 21 / 37
32. Eyeriss Highlights Row Stationary Data Flow
AlexNet PE Mapping
Michael Lee (NTU) Introduction to Eyeriss October 24, 2017 23 / 37
33. Eyeriss Highlights Row Stationary Data Flow
AlexNet Inter-Pass Data Caching
Michael Lee (NTU) Introduction to Eyeriss October 24, 2017 24 / 37
34. Eyeriss Highlights Row Stationary Data Flow
AlexNet Shape Parameters
How do we map AlexNet onto Eyeriss?
L H6
R E C M U
1 227 11 55 3 96 4
2 31 5 27 48 256 1
3 15 3 13 256 384 1
4 15 3 13 192 384 1
5 15 3 13 192 256 1
m7
n e p q r t
96 1 7 16 1 1 2
64 1 27 16 2 1 1
64 4 13 16 4 1 4
64 4 13 16 3 2 2
64 4 13 16 3 2 2
6H: ifmap width, R: kernel width, E: ofmap width, C: Channels, M: # kernels, U: Stride
7m: # ofmap chan stored in GLB, n: # ifmap, e: width of PE set, p: # filters proc., q: #
chan proc., r: # pe proc. diff. chan., t: # pe proc. diff. filter.
Michael Lee (NTU) Introduction to Eyeriss October 24, 2017 25 / 37
35. Eyeriss Highlights Row Stationary Data Flow
AlexNet Shape Mapping Illustrated
Michael Lee (NTU) Introduction to Eyeriss October 24, 2017 26 / 37
36. Eyeriss Highlights Network-on-a-chip (NoC)
NoC Optimized for RS
Global input/output network: use Multicast Controller (MC) to
broadcaset GLB data into assigned PE. Data is augmented with
(row, col) in GLB
filter GI/ON
ifmap GI/ON
psum GI/ON
Local network: dedicated 64b data bus is implemented to pass
the psums from the bottom PE to the top PE directly
Michael Lee (NTU) Introduction to Eyeriss October 24, 2017 27 / 37
38. Eyeriss Highlights Network-on-a-chip (NoC)
Populate Data with Global Input / Output Network
Michael Lee (NTU) Introduction to Eyeriss October 24, 2017 29 / 37
39. Eyeriss Highlights Compression and Data Gating
Run-Length Compression (RLC)
ReLU produces many zeros in activated ofmap, use RLC to save
power in DRAM R/W
Michael Lee (NTU) Introduction to Eyeriss October 24, 2017 30 / 37
40. Eyeriss Highlights Compression and Data Gating
Data Gating / Zero Skipping
Simply skip tasks when either pixel or weight is zero
Michael Lee (NTU) Introduction to Eyeriss October 24, 2017 31 / 37
42. Summary
Performance Summary and Comparison
Eyeriss[5] NVIDIA TK1
Technology 65nm 1P9M 28nm
Chip Size 4.0×4.0 N/A
Core Area 3.5×3.5 N/A
Gate Count 1176k N/A
Word Bit-Width 16b Fixed 32b Float
Core Clock(MHz) 200 852
On-Chip Buffer Size (kB) 108 64
Total Register Size (kB) 75.3 256
#MAC 168 192
Throughput(fps) 34.7 68
Measured Power Idle (mW) 3700
Measured Power Active (mW) 278 10002
Michael Lee (NTU) Introduction to Eyeriss October 24, 2017 33 / 37
43. Summary
Summary
RS optimizes for best overall energy efficiency while existing
CNN dataflows only focus on certain data types.
RS has higher energy efficiency than existing dataflows
1.4X ∼ 2.5X higher in CONV layers
at least 1.3X higher in FC layers. (batch size ≥ 16)
Michael Lee (NTU) Introduction to Eyeriss October 24, 2017 34 / 37
44. Appendix
Bibliography I
Lukas Cavigelli et al. “Origami: A Convolutional Network Accelerator”. In: Proceedings of the 25th Edition on Great
Lakes Symposium on VLSI. GLSVLSI ’15. Pittsburgh, Pennsylvania, USA: ACM, 2015, pp. 199–204. isbn:
978-1-4503-3474-7. doi: 10.1145/2742060.2743766. url: http://doi.acm.org/10.1145/2742060.2743766.
Srimat Chakradhar et al. “A Dynamically Configurable Coprocessor for Convolutional Neural Networks”. In:
Proceedings of the 37th Annual International Symposium on Computer Architecture. ISCA ’10. Saint-Malo, France:
ACM, 2010, pp. 247–257. isbn: 978-1-4503-0053-7.
Yu-Hsin Chen, Joel Emer, and Vivienne Sze. ISCA 2016 Slides of Eyeriss: A Spatial Architecture for Energy-Efficient
Dataflow for Convolutional Neural Networks.
Tianshi Chen et al. “DianNao: A Small-footprint High-throughput Accelerator for Ubiquitous Machine-learning”. In:
Proceedings of the 19th International Conference on Architectural Support for Programming Languages and
Operating Systems. ASPLOS ’14. Salt Lake City, Utah, USA: ACM, 2014, pp. 269–284. isbn: 978-1-4503-2305-5.
doi: 10.1145/2541940.2541967. url: http://doi.acm.org/10.1145/2541940.2541967.
Y. H. Chen et al. “Eyeriss: An Energy-Efficient Reconfigurable Accelerator for Deep Convolutional Neural
Networks”. In: IEEE Journal of Solid-State Circuits 52.1 (Jan. 2017), pp. 127–138.
Y. Chen et al. “DaDianNao: A Machine-Learning Supercomputer”. In: 2014 47th Annual IEEE/ACM International
Symposium on Microarchitecture. Dec. 2014, pp. 609–622. doi: 10.1109/MICRO.2014.58.
Michael Lee (NTU) Introduction to Eyeriss October 24, 2017 35 / 37
45. Appendix
Bibliography II
Z. Du et al. “ShiDianNao: Shifting vision processing closer to the sensor”. In: 2015 ACM/IEEE 42nd Annual
International Symposium on Computer Architecture (ISCA). June 2015, pp. 92–104. doi:
10.1145/2749469.2750389.
V. Gokhale et al. “A 240 G-ops/s Mobile Coprocessor for Deep Neural Networks”. In: 2014 IEEE Conference on
Computer Vision and Pattern Recognition Workshops. June 2014, pp. 696–701. doi: 10.1109/CVPRW.2014.106.
Suyog Gupta et al. “Deep Learning with Limited Numerical Precision”. In: Proceedings of the 32Nd International
Conference on International Conference on Machine Learning - Volume 37. ICML’15. Lille, France: JMLR.org, 2015,
pp. 1737–1746. url: http://dl.acm.org/citation.cfm?id=3045118.3045303.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. “ImageNet Classification with Deep Convolutional Neural
Networks”. In: Advances in Neural Information Processing Systems 25. Ed. by F. Pereira et al. 2012, pp. 1097–1105.
S. Park et al. “4.6 A1.93TOPS/W scalable deep learning/inference processor with tetra-parallel MIMD architecture
for big-data applications”. In: 2015 IEEE International Solid-State Circuits Conference - (ISSCC) Digest of Technical
Papers. Feb. 2015, pp. 1–3. doi: 10.1109/ISSCC.2015.7062935.
M. Peemen et al. “Memory-centric accelerator design for Convolutional Neural Networks”. In: 2013 IEEE 31st
International Conference on Computer Design (ICCD). Oct. 2013, pp. 13–19. doi: 10.1109/ICCD.2013.6657019.
Michael Lee (NTU) Introduction to Eyeriss October 24, 2017 36 / 37
46. Appendix
Bibliography III
Chen Zhang et al. “Optimizing FPGA-based Accelerator Design for Deep Convolutional Neural Networks”. In:
Proceedings of the 2015 ACM/SIGDA International Symposium on Field-Programmable Gate Arrays. FPGA ’15.
Monterey, California, USA: ACM, 2015, pp. 161–170. isbn: 978-1-4503-3315-3. doi: 10.1145/2684746.2689060.
url: http://doi.acm.org/10.1145/2684746.2689060.
Michael Lee (NTU) Introduction to Eyeriss October 24, 2017 37 / 37