This presentation covers a talk on the topic of "AI on the edge". The talk was delivered in the Conference on Artificial Intelligence and Robotics Technology held on Jan 28, 2021 by National Center of Artificial Intelligence Pakistan & working group by Ministry of Science and Technology on AI & Robotics.

1. 1
AI on the Edge
Lifecycle of AI solution to edge computing
Usman Qayyum
Additional Director (AI)
PhD (AI & Robotics)
Post-doc (Self-driving Cars)
mrusmanqayyum@hotmail.com

2. 2
Artificial Intelligence
Human Intelligence exhibited by machines

3. 3
AI Life Cycle

4. 4
IOT on Edge-based AI is the most
disruptive trend in modern application
development

5. 5
Edge Computing

6. 6

7. 7
Edge-AI is Data-First Application
Development

8. 8
Edge Computing: Drivers

9. 9
EDGE is Local

10. 10
Similar Concept

11. 11
Selecting AI on Edge

12. 12
Edge Computing Applications: Humanity

13. 13
Edge Computing Applications: Environment

14. 14
What are we going to discuss now ?
● AI algorithms optimize for hardware
○ Quantization
○ Pruning
● Edge Hardware
○ CPU
○ GPU
○ FPGA
○ ASIC
○ Software Toolkit for Edge

15. 15

16. 16
● We can use CPU Cache
● No need to read from RAM
● I can run it in a device with smaller memory

17. 17
What can I do to optimize?
COMPUTE LESS
COMPUTE FASTER
MOVE LESS DATA

18. 18
What can I do to optimize?
COMPUTE FASTER
● Increase the speed you compute
○ Increase the clock speed.
● Compute several things at the same time
○ SIMD instructions
○ Multiprocessing

19. 19
What can I do to optimize?
COMPUTE LESS & MOVE LESS DATA
● Decrease the number of parameters
● Decrease the size of the parameters

20. 20
What can I do to optimize?
PRUNING
Removing branches that are not important for our goal
L1 layer regularization
Structured pruning by varying regularizer parameters
Deep compression

21. 21
What can I do to optimize?
QUANTIZE
Reduce the representation of the data values we use
Deep compression
Super Nice explanation of 8bit quantization in TF

22. 22
LET’S SEE SOME HARDWARE

23. 23
CPU to Neuromorphic Computing

24. 24
Intel AI on Edge modules

25. 25
Qualcomm Snapdragon 845 / Apple
A12X

26. 26
HiKey 970
● This is the first HiSilicon SoC with a
NPU
● Price 300$

27. 27
Sophon Edge
Sophon Edge was released in 2018 and it
features the SoC Sophon BM1880 which
includes a TPU. This SoC can theoretically
deliver up to 2TOPS/s
Price 129$

28. 28
FPGA
FPGA is in the middle point between CPU and ASIC
The good thing of hardware world:
● Low latency
● Low power consumption
The good things from software:
● You can update your hardware
● Cheaper than building in silicon
Quite complex to program

29. 29
FPGA
They are complex to program but...
● Libraries to the rescue!
CHaiDNN Library for Xilinx
OpenVINO for Intel devices

30. 30
Chamaleon96 Cyclone V FPGA
● This board is going to be different
compared to the rest because this
one has a FPGA instead of a
dedicated hardware accelerator
● Price 130$

31. 31
FPGA
INTEL STRATIX/ARRIA/CYCLONE Series
XILINX ZYNQ Series

32. 32
Let’s talk about some hardware

33. 33
TPU
CPU GPU TPU
● It’s an ASIC
● TPUv1 has only 8bit fixed-point support (ONLY USEFUL FOR TRAINING)
● TPUv2 includes 16bit float (NOW WE CAN TRAIN :D )

34. 34
Google Coral Dev Board
● A Raspberry Pi look like
board but featuring some
sort of TPU that allows
running NNs in the edge
really fast.
● Google Promises real
time inference for NNs
● Price $129

35. 35

36. 36
Mini TPU
Google says the next:
“Edge TPU enables users to concurrently execute
multiple state-of-the-art AI models per frame, on a
high-resolution video, at 30 frames per second, in a
power-efficient manner.”
IT’S in EARLY ACCESS STAGE :(
Edge TPU

37. 37
Intel Compute Stick 2
USB 3.0 Connection
Able to run object detection at ~10fps
Compatible with Tensorflow, Caffe2 or ONNX
(as long you have the supported layers)
V1 ~60€
V2 ~ 100€ but 8x speed up compared to v1.

38. 38
1.3 TFLOPS FP32 from the GPU - GPU FP16 is 10.4 TFLOPS.
NVIDIA Jetson Xavier
GTX1080ti has ~10TFLOPS at FP32

39. 39
OpenVino Toolkit by Intel

40. 40
OPENVINO FOR EDGE

41. 41
OpenVino Performance

42. 42
AI on Edge (Complete Roadmap)

43. 43
Conclusion: Why Edge Computing

44. 44
Thanks for your attention.
AI on the Edge
Usman Qayyum
Additional Director (AI)
PhD (AI & Robotics)
Post-doc (Self-driving Cars)
mrusmanqayyum@hotmail.com