This document discusses neural network file formats and inference frameworks. It describes common file formats like ONNX and NNEF that can represent neural networks. Inference frameworks like TensorFlow Lite, OpenVINO, and ONNX Runtime provide APIs to run these models. They operate at two levels: constructing a compute graph from the model and executing with optimized kernel functions for different hardware like CPUs, GPUs, and NPUs. The document also introduces the Kneron NPU and its compiler flow for optimized inference.

1. Neural Network File Format for Inference Framework
Kobe Yu

2. About Me
● Kobe Yu
● Experience
○ PhD student(Advisor JK Lee) @PLLab NTHU
○ Compiler engineer @Kneron
● Open source project
○ Agrino, Field sensor for agricultue
○ Plant detection by deep learning
● Community
○ Open hack farm
○ FarmBot Taiwan user group

3. About The Licenses
Attribution-ShareAlike 3.0 Unported (CC BY-SA 3.0)
You are free to:
Share — copy and redistribute the material in any medium or format
Adapt — remix, transform, and build upon the material
for any purpose, even commercially.
This license is acceptable for Free Cultural Works.
The licensor cannot revoke these freedoms as long as you follow the license terms.

4. Outline
● Neural Network Workflow
● Neural Network File Format
● Inference Framework
● Kneron NPU

5. AI Architecture Edge v.s Centralized
source: https://www.linkedin.com/pulse/ai-5g-wireless-network-edge-berge-ayvazian/

6. Neural Network Workflow - From Training to Inference
source: https://developer.qualcomm.com/docs/snpe/overview.html

7. Neural Network File format
● Graph and and nodes
○ Network struct(DAG)
○ Node with attribute and pretrained weights
● Training Framework
○ Tensorflow: pb
○ Caffe: .prototext .caffemodel
○ Pytorch: .h5
● Inference Framework
○ Qualcomm SNPE: .dlc
○ Intel OpenVINO : xml + raw binary(pretrain weights)
○ Tensorflow: .tflite
● Exchangeable File format
○ ONNX (ONNX community)
○ NNEF (Khronos)

8. Neural Network File Format - ONNX
● Content
○ Graph / Node with attributes
○ Pretrain weights
■ Float
■ Fixed point (Quantization)

9. Flatbuffer
source: https://google.github.io/flatbuffers/flatbuffers_benchmarks.html

10. Neural Network File format - NNEF
source: https://www.khronos.org/nnef

11. Neural Network File format
source: https://www.khronos.org/nnef

12. ONNX and ONNX Runtime
source: https://mc.ai/accelerate-and-productionize-ml-model-inferencing-using-open-source-tools/

13. Inference Framework Framework
● Forware only
● Target independent
○ Abstruction layer for hareware target
■ OpenVINO / DLDT - inference engine
■ ONNX Runtime - execution provider
■ Android / NNAPI
■ ARM / NN SDK
■ ...
● Target dependent
○ Highly optimized kernel funciton
○ Intel - MKLDNN(CPU), clDNN(GPU), FPGA plugin, VPU(Movidius)
○ ARM / Compute library - CPU / GPU
○ Nvidia / TensorRT

14. Inference Framework - ONNX Runtime
source: https://github.com/microsoft/onnxruntime/blob/master/docs/InferenceHighLevelDesign.md

15. Inference Framework - Intel OpenVINO
source: https://docs.openvinotoolkit.org/2018_R5/_docs_MO_DG_Deep_Learning_Model_Optimizer_DevGuide.html

16. Inference Framework - ARM NN SDK
SOURCE: https://developer.arm.com/ip-products/processors/machine-learning/arm-nn

17. Inference Framework - Android NNAPI
Source: https://developer.android.com/ndk/guides/neuralnetworks

18. Performance
source: https://medium.com/@aallan/benchmarking-edge-computing-ce3f13942245

19. NNEF to clDNN

21. Kneron NPU - KL520

22. Kneron Compiler Flow

23. Kneron KL520 Firmware Architecture
source: http://doc.kneron.com:8888/

24. Summary
● Neural network file format(model file) v.s Inference framework
○ a.out / .exe v.s Unix like OS / Windoes
● Inference framework
○ Two level software stack
■ Level one: Construct compute graph
■ Level two: Highly optimized kernel function(implementation of nn node)
■ Dedicated hardware
● NPU
○ Provide height level ISA( convolution, pooling, activation function...etc)

25. Q&A