For the full video of this presentation, please visit: https://www.edge-ai-vision.com/2023/06/bring-your-ml-models-to-the-edge-with-the-degirum-delight-cloud-platform-a-presentation-from-degirum/ Shashi Chilappagari, Co-Founder and Chief Architect of DeGirum, presents the “Bring Your ML Models to the Edge with the DeGirum DeLight Cloud Platform,” tutorial at the May 2023 Embedded Vision Summit. In this talk, Chilappagari presents an overview of the DeGirum DeLight Cloud Platform, which accelerates the edge AI application development process by bringing together application software, machine learning models and AI accelerator hardware. The platform provides cloud access to multiple AI hardware accelerators, including DeGirum’s ORCA, allowing developers to get started without investing time and money on bringing up hardware. It provides intuitive, hardware-agnostic APIs that enable developing a single software stack for multiple hardware options. Along with a rich model zoo covering a wide variety of applications, the platform provides tools to enable developers to port their own models. Chilappagari highlights the challenges in model porting, including proper specification of pre-processing and post-processing logic; targeting the model for multiple hardware; and pipelining various stages in the application to optimize performance. He illustrates, with the help of examples, how the DeLight platform overcomes these challenges.

1. Bring Your ML Models to
the Edge with the DeGirum
DeLight Cloud Platform
Shashi Chilappagari
Co-Founder and Chief Architect
DeGirum Corp.

2. • Not all ML models are compatible with all
hardware
• Runtime software can be hardware
specific
• Different types of ML models need
different software
Developing AI Applications for the Edge
© 2023 DeGirum Corp. 2
Software
Hardware
Models

3. Evaluating AI Hardware Accelerators is Hard
Buy
hardware
Install
software
toolchain
Port/design
hardware
compatible ML
model
Develop
application
software
Benchmark
performance
Repeat for all hardware options
Result: Separate application software & models for
each hardware option
Wasted time, money, and effort
© 2023 DeGirum Corp. 3

4. Problems with the Hardware Evaluation Process
Impossible to judge a hardware option based on datasheet
No unified application software that makes evaluation easy
© 2023 DeGirum Corp. 4

5. 1. Cloud access to AI hardware
2. Software that supports
multiple hardware options
3. Tools to choose/port/design
ML models
DeGirum Cloud
Platform & Model Zoo
DeGirum Device Farm
Access through
DeGirum Cloud Server
Direct Access through
Peer-To-Peer VPN
DeGirum Solution: DeLight Cloud Platform
Highlights
© 2023 DeGirum Corp. 5
ORCA EdgeTPU
EdgeTPU
EdgeTPU
GPU
GPU
GPU
CPU
CPU
CPU
Other
Other
Other
ORCA
ORCA

6. • ORCA-NNX (DeGirum): Available now
• CPU (Intel, AMD, ARM): Available now
• Edge TPU (Google): Coming soon
• GPU (Nvidia): Coming soon (Jetson and Orin series)
• Others can be added on demand
Supported Hardware Options
6
© 2023 DeGirum Corp.

7. DeGirum PySDK: Sophisticated Applications
with Simple APIs
7
© 2023 DeGirum Corp.
• DeGirum cloud farm
• AI server IP address
• Local hardware
import degirum as dg
zoo=dg.connect(ai_server, model_zoo, token)
model=zoo.load_model(model_name)
res=model(image)
res.image_overlay
• Cloud model zoo
• Local model zoo
• Single model file
Each line in the above code solves a different problem faced by developers

8. What Do Our Customers Want?
8
© 2023 DeGirum Corp.
No model zoo is large enough: Developers want to
bring their own models
Need tools that allow frictionless integration of
trained models to application software

9. Challenges in Integrating ML Models
into Applications
© 2023 DeGirum Corp. 9

10. ML Application Software Pipeline
© 2023 DeGirum Corp. 10
Application software is much more than just ML inference
Input
Capture
Input
Decoding
Input
Resizing
Input Pre-
Processing
ML
Inference
Output
Post-
Processing
Output
Resizing
Output
Rendering
Application
Logic

11. Model Integration Challenge 1
11
© 2023 DeGirum Corp.
Challenge: Efficient, accurate, and comprehensive
implementation of resizing, pre-processing, and post-
processing stages
Importance: Model accuracy depends on getting all
implementations and parameters right

12. • Using the same pre/post-processing parameters during training
and inference is crucial to maintaining model accuracy when
porting to new hardware
• Software needs to support various options for these parameters
• Image backend, resize method, interpolation option, input
normalization, input quantization, nms threshold, confidence
threshold
• Comprehensive and accurate implementation is a lot of work!!!
Why is it Hard to Implement these Stages?
12
© 2023 DeGirum Corp.

13. Model inference API includes decoding, resizing, pre-processing, and
post-processing
import degirum as dg
zoo=dg.connect()
model=zoo.load_model(‘efficientnet’)
res=model(‘cat.jpg’)
res.image_overlay
How Does DeGirum PySDK Solve the Problem?
13
© 2023 DeGirum Corp.

14. Model Configuration JSON File
14
© 2023 DeGirum Corp.
{
"PRE_PROCESS": [
{
"InputType": "Image",
"InputN": 1,
"InputH": 224,
"InputW": 224,
"InputC": 3,
"InputQuantEn": true,
"InputQuantOffset": 114,
"InputQuantScale": 0.018658,
"InputPadMethod": "crop-last",
"InputImgNormEn": true,
"InputImgMean": [ 0.485, 0.456, 0.406 ],
"InputImgStd": [ 0.229, 0.224, 0.225 ],
"InputResizeMethod": "bicubic",
"InputCropPercentage": 0.875
}
],
"MODEL_PARAMETERS": [
{
"ModelPath": "efficientnet_es_orca_2.n2x"
}
],
"POST_PROCESS": [
{
"OutputPostprocessType": "Classification",
"OutputTopK": 5,
"OutputConfThreshold": 0.1,
"LabelsPath": "labels_ILSVRC2012_1000.json"
}
],
"DEVICE": [
{
"DeviceType": "ORCA",
"RuntimeAgent": "N2X"
}
],
}
Allows specifying different parameters

15. Model Integration Challenge 2
15
© 2023 DeGirum Corp.
Challenge: Visualizing output of ML model
Importance: Developers need to visualize ML model output
to get a qualitative sense of model performance

16. • Different types of models provide different types of information
• Image classification: Top labels and their probabilities
• Object detection: Object categories and their locations
(bounding boxes)
• Semantic segmentation: Category label for every pixel
• Pose detection: Key point locations
• Visualizing output for different types of models requires developing
custom output rendering logic for each type
Why is Model Output Visualization Challenging?
16
© 2023 DeGirum Corp.

17. How Does DeGirum PySDK Solve the Problem?
17
© 2023 DeGirum Corp.
import degirum as dg
zoo=dg.connect()
model=zoo.load_model(‘yolov5s’)
res=model(image)
res.image_overlay Model inference results object
helps visualize the output

18. Output Visualization Using PySDK
18
© 2023 DeGirum Corp.
import degirum as dg
zoo=dg.connect()
model=zoo.load_model(‘efficientnet’)
res=model(image)
res.image_overlay

19. Output Visualization Using PySDK
19
© 2023 DeGirum Corp.
import degirum as dg
zoo=dg.connect()
model=zoo.load_model(‘yolov5s’)
res=model(image)
res.image_overlay

20. Output Visualization Using PySDK
20
© 2023 DeGirum Corp.
import degirum as dg
zoo=dg.connect()
model=zoo.load_model(‘deeplab_seg’)
res=model(image)
res.image_overlay

21. • Image classification
• Object detection
• Semantic segmentation
• Pose detection
• Hand landmark detection
• License plate OCR
• Custom logic via python
Postprocessors Supported by PySDK
21
© 2023 DeGirum Corp.

22. Model Integration Challenge 3
22
© 2023 DeGirum Corp.
Challenge: Developing software that works for multiple
hardware options
Importance:
(1) Enables easy and fair comparison of hardware options
(2) Enables multiple product lines with different hardware
but single software

23. • Different hardware use different runtimes
• N2X: DeGirum
• OpenVINO: Intel
• TensorRT: Nvidia
• TFLite: Google
• ONNXRT: Microsoft
• Unifying multiple hardware options into one software requires significant
amount of work
Challenge in Developing Unified Software
23
© 2023 DeGirum Corp.

24. How Does DeGirum PySDK Solve the Problem?
24
© 2023 DeGirum Corp.
DeGirum
Core
Runtime
Agent
Interface
TRT Runtime
Plugin
TFLite
Runtime
Plugin
ONNX
Runtime
Plugin
DeGirum
N2X Runtime
Plugin
GPU
DLA
CPU
EdgeTPU
CPU
GPU
Other
CPU
Orca
PySDK
Server
DeGirum
PySDK
Client
Common runtime agents are integrated as plugins

25. import degirum as dg
zoo=dg.connect()
model=zoo.load_model(‘yolov5_orca’)
res=model(image)
res.image_overlay
Software Supporting Multiple Hardware
25
© 2023 DeGirum Corp.

26. import degirum as dg
zoo=dg.connect()
model=zoo.load_model(‘yolov5_cpu’)
res=model(image)
res.image_overlay
Software Supporting Multiple Hardware
26
© 2023 DeGirum Corp.

27. import degirum as dg
zoo=dg.connect()
model=zoo.load_model(‘yolov5_gpu’)
res=model(image)
res.image_overlay
Software Supporting Multiple Hardware
27
© 2023 DeGirum Corp.

28. Model Integration Challenge 4
28
© 2023 DeGirum Corp.
Challenge: Efficient pipelining of all stages in the ML
application
Importance:
(1) Needed for optimizing application performance
(2) Needed to ensure hardware is utilized efficiently

29. • Software contains multiple stages with each stage running
(possibly) on a different hardware resource
• ML inference on AI hardware accelerator, application logic on
host CPU, input decoding on special decoders
• Scheduling workloads for different hardware components
efficiently is a difficult problem
• Balancing trade-offs between frames per second (FPS) and latency
adds extra layer of complexity
• Managing multiple applications requires lot of system level testing
Challenges in Pipelining
29
© 2023 DeGirum Corp.

30. How Does DeGirum PySDK Solve the Problem?
30
© 2023 DeGirum Corp.
import degirum as dg
import mytools
camera_id = 0
zoo=dg.connect()
model=zoo.load_model(‘yolo_v5s_face’)
with mytools.Display("AI Camera") as display,
mytools.open_video_stream(camera_id) as stream:
for res in model.predict_batch(mytools.video_source(stream)):
display.show(res.image_overlay)

31. • DeGirum DeLight platform speeds your work in selecting
processors and models, and building portable edge ML
applications by providing
• Cloud access to edge hardware
• Software that supports multiple hardware options
• Simple to use inference APIs
• Tools that enable visualizing model outputs
• Methods to optimize application performance
Summary
31
© 2023 DeGirum Corp.
Save time,
money,
and effort

32. •Request access to DeGirum DeLight Cloud Platform at
https://cs.degirum.com/
•See code examples of PySDK at
https://github.com/DeGirum/PySDKExamples
•Download compiler, AI server, and AI client dockers at
https://hub.docker.com/u/degirum
•Watch demos at https://www.youtube.com/@degirumcorp.5330
Additional Information
32
© 2023 DeGirum Corp.

33. Resources
2023 Embedded Vision
Summit
Visit our booth!
DeGirum Website
https://www.degirum.com/
DeGirum Cloud Platform
https://cs.degirum.com/
DeGirum GitHub Repo
https://github.com/DeGirum/Py
SDKExamples
© 2023 DeGirum Corp. 33