Session ID: HKG18-304 Session Name: HKG18-304 - Scalable AI server Speaker: Shuichi Yamane Track: LEG AI;Ecosystem Day ★ Session Summary ★ Current hardware set on AI application have been dominated by GPGPU. We see this trend very natural since ""LEARNING"" which requires processing power was the start point. Now we are in the phase to utilize the ""DATA"" for inference. Attaching CNN dedicated processing accelerator into Socionext SynQuacer server becomes AI server with superior power efficiency to existing hardware set. The scalability of this AI server will make AI inference process similar to Index searching. --------------------------------------------------- ★ Resources ★ Event Page: http://connect.linaro.org/resource/hkg18/hkg18-304/ Presentation: http://connect.linaro.org.s3.amazonaws.com/hkg18/presentations/hkg18-304.pdf Video: http://connect.linaro.org.s3.amazonaws.com/hkg18/videos/hkg18-304.mp4 --------------------------------------------------- ★ Event Details ★ Linaro Connect Hong Kong 2018 (HKG18) 19-23 March 2018 Regal Airport Hotel Hong Kong --------------------------------------------------- Keyword: LEG AI;Ecosystem Day 'http://www.linaro.org' 'http://connect.linaro.org' --------------------------------------------------- Follow us on Social Media https://www.facebook.com/LinaroOrg https://www.youtube.com/user/linaroorg?sub_confirmation=1 https://www.linkedin.com/company/1026961

1. Scalable AI server
Mar. 2018
S3 project

2. 1
Where is the data for AI inference?
Object
❔
CNN
Feature
weight
❔ Matching Result
Where do we
place data?
Copyright 2018

3. 2
What we do on demo
Original
Data set
VGG16 model
with leaned weight
Object
❔ CNN
Feature
weight
❔
Copyright 2018

4. 3
In real world
Learned
data
Inference
Learning
Sensors
Copyright 2018

5. 4
Multi tasking
Sensors
Object & Weight Search matching data
For large size model data
Copyright 2018

6. 5
Technical requirement
Sensors
Object & Weight Search matching data
Copyright 2018
CNN part Search part
CNN computing performance
# of channel for sensor connection
Scalability
Low latency
High throughput

7. 6
AI specific hardware
Copyright 2018
SPR280S AI module：22TOPS
x 16 CPU card
= 352 TOPS
2x performance
1/5x power

8. 7
Configuration
Copyright 2018
x 16
x 16
All CPU cards are connected via Socionext DDT

9. 8
Socionext DDT?
Copyright 2018
Item Performance on Socionext DDT(Direct Data Transaction) Technology
Latency 1.5us/path Same as PCIe Transaction latency
Fixed Datapath and Direct Data transfer from Node to
Node
Bandwidth 20-40G Ethernet
class
Performs Maximum TCP Window Speed from Beginning
Logical Ethernet Communication on PCI Express

10. 9
Extended version
2x chassis
connected
by
PCI express optical cable
Copyright 2018

11. 10
PEC: Processor Element Card
Server node = PEC
PEC: Processor Element Card
Copyright 2018

12. 11
CPU on PEC
CPU: Cortex A53
PEC: Processor Element Card
24 cores/1GHz
No speculation algorithm
No OOO execution
Safe
Copyright 2018

13. 12
CPU-CPU connection
PEC: Processor Element Card
Socionext DDT
Low latency
High throughput
μSEC latency
Copyright 2018

14. 13
Server-server connection
PEC: Processor Element Card
DDT-R
Low latency
High throughput
112x servers
DDT-R
DDT
DDT
DDT
DDT
Copyright 2018

15. 14
AI accelerator chip
Copyright 2018
● Matrix Neural Processor
High performance
@ low clock frequency
● Outstanding power efficiency
➢ 7000 Gops/W
• Google TPU: 613
• Tesla P100: 60
Parameter Performance
Clock (MHz) 100
Throughput(Top/s) 5.6
Power (W) 0.8
100x efficiency

16. 15
AI accelerator chip on 96 board
Copyright 2018
Movie