High level introduction of OpenVINO toolkit features that make it best tool for CV/DL deployment @ Intel platforms

1. Yury Gorbachev
1

2. Intel ConfidentialIntel Confidential
Brief OpenVINO™ Introduction
• OpenVINO ™ is
• set of tools and libraries for CV/DL application developers
• high performance, low footprint solution for deployment
• API for unified access to CV/DL capabilities of Intel platforms
• OpenVINO ™ is not
• tool for data scientists
• solution for training of deep learning models

3. Intel ConfidentialIntel Confidential 3
OpenVINO™ Benefits
• Powerful combination of highly optimized Classical CV and DL primitives
• Allows to run inference on Intel CPU, GPU, VPU and FPGA
• Best performing solution on all Intel architectures
• ~2x faster than fastest TensorFlow and MxNet on CPU
• Smallest execution footprint (lowest memory consumption)
• 2x smaller than MXNet, 4x smaller than TensorFlow
• Minimum # of dependencies (no dependencies on training frameworks)
• Multiple OS support (Linux and Windows)

4. Intel ConfidentialIntel Confidential 4
OpenVINO™ Specifics
• Highly optimized implementations of DL primitives
• Most efficient on each Intel platform
• Focus on inference only
• Aggressive layer fusion at the inference step
• Including HW accelerated steps tuned for inference
• Efficient activation memory reuse
• Often close to bare minimum

5. Intel ConfidentialIntel Confidential 5
DL Workflow
Caffe
MXNet
TensorFlow
Caffe2
PyTorch
Serialized
trained
DL model
ONNX
MKLDNN
Plugin
clDNN
Plugin
FPGA
Plugin
Myriad
Plugin
Inference
Engine
Deploy
Application
Model
Optimizer
IR
.xml
.bin
Step 1: Import model from Framework format
to Framework independent representation
Step 2: Update application
to use Inference Engine API
Eliminate unnecessary layers,
lossless fusion where possible
Remove framework dependency
Accuracy against original model ensured

6. Intel ConfidentialIntel Confidential 6
Customization Capabilities
• OpenVINO™ provides good coverage of DL primitives out of the box
• Constantly growing list of primitives to support new DL topologies
• Frequent releases, substantial additions
• Not a problem if something is missing!
• Good extension mechanism for adding new primitives
• Possible to add proprietary layers, more optimized layers, etc.
• Both in Model Optimizer and Inference Engine (import and run)

7. Intel ConfidentialIntel Confidential 7
Application Design Workflow
Desig
n
Verify
logic
Debug
Fix
Best done on CPU:
- Easier to verify
- Simpler debugging procedures
CHANGE
TARGET
Check
scalability
Fix
pipeline
System
testing
Best done on Actual target (e.g. VPU):
- Exact performance
- Correct timings
Accuracy and functionality
across targets

8. Intel ConfidentialIntel Confidential 8
Heterogeneous Execution
• When a certain primitive is not supported on a target
• Custom proprietary primitive or inefficient HW for a task
• Heterogeneous execution ensures full topology execution
• Automatic data transfer between targets whenever needed
• Work splitting and scheduling
• No need to do any manual network manipulations!
FPGA CPU

9. Intel ConfidentialIntel Confidential 9
Power of Parallel Execution
Asynchronous API provides capabilities for:
• Running main thread in parallel with ongoing inference (CPU/GPU)
• Hiding data transfer latency for accelerators (VPU, FPGA)
• Filling accelerators with work
Transfer 1
Inference 1
Transfer 2
Transfer 1 Transfer 2
Inference 2
Inference 1 Inference 2
Result 1
Result 1
Result 2
Result 2
Sync API
Async API

10. Intel ConfidentialIntel Confidential 10
Efficient Frame Preprocessing
DECODE
Layout
Transform
Resize to
network
Detection
network
Resize to
network
Object Analysis
Network
Crop
Preprocessing
DL Inference
• Preprocessing is typically done via manual coding or using libraries
• OpenCV is most popular
• Fastest OpenCV build is available in OpenVINO™
• Deep Learning Inference Engine encapsulates basic preprocessing capability
• Automatic frame resize based on input frame and network size
• Automatic layout conversion and cropping
• In a nutshell –> Just provide frame and it will be suitable for inference automatically

11. Intel ConfidentialIntel Confidential 11
INT8 Quantization for CPU
• INT8 provides additional acceleration using AVX-512
• Not all CPU targets support it
• Very minor quality/accuracy loss
• No retraining is required
• No code update is needed
IR
Dataset
Analysis
tool Updated IR

12. Intel ConfidentialIntel Confidential 12
OpenVINO™ Model Zoo
• OpenVINO provides pre-trained DL models for deployment
• Lightweight, low compute, real time on Intel platforms
• Cover popular CV use cases
• Face analysis, Security use-cases (person, vehicle, bicycle detection)
• Transportation analytics (road segmentation, vehicle/pedestrian detection)

13. Intel ConfidentialIntel Confidential 13
Extensive Set of Samples
• Actual examples of applications, not just API demonstration
• Switching between targets, work distribution
• Multiple models in pipeline with preprocessing
• Open Models Zoo demonstration

14. 14