presentation of Nvidia deepstream toolkit

1. Nvidia Toolkits -
DeepStream
IoT
Team

2. NVIDIA DEEPSTREAM SDK
Build and deploy AI-powered Intelligent Video Analytics apps and services. DeepStream
offers a multi-platform scalable framework with TLS security to deploy on the edge and
connect to any cloud.

3. NVIDIA DEEPSTREAM SDK
• With DeepStream SDK you can apply AI to streaming video and can
simultaneously optimize video decode/encode, image scaling and
conversion and edge-to-cloud connectivity for complete end-to-end
performance optimization
• Developers can build seamless streaming pipelines for AI-based video,
audio and image analytics using DeepStream.
• DeepStream offers an IoT integration interface with Kafka, MQTT and AMQP
and turnkey integration with AWS IoT and Microsoft Azure IoT.
• By using DeepStream, you can deploy at scale and manage containerized
apps with Kubernetes and Helm Charts.

4. DEEPSTREAM SDK PLUG-INS
•H.264 and H.265 video decoding
•Stream aggregation and batching
•TensorRT-based inferencing for detection,
classification and segmentation
•Object tracking reference implementation
•On-screen display API for highlighting objects
and text overlay
•Frame rendering from multi-source into a 2D grid
array
•Accelerated X11/EGL-based rendering
•Filtering based on Region of Interest (ROI)
•JPEG decoding
•Scaling, format conversion, and rotation
•Dewarping for 360-degree camera input
•Metadata generation and encoding
•Messaging to cloud
•Audio/Video Template Plug-In

5. NVIDIA DEEPSTREAM SDK
• Python bindings
• The DeepStream Python application uses the Gst-Python API action to construct the pipeline
and use probe functions to access data at various points in the pipeline.
• Prerequisites
• Ubuntu 18.04
• DeepStream SDK 5.1 or later
• Python 3.6
• Gst Python v1.14.5
1.Clone the deepstream_python_apps repo under <DeepStream 5.1 ROOT>/sources:
git clone https://github.com/NVIDIA-AI-IOT/deepstream_python_apps
2.This will create the following directory:
<DeepStream 5.1 ROOT>/sources/deepstream_python_apps
Sample Application Source Details

6. APPLICATION ARCHITECTURE
• The image below shows the architecture of the NVIDIA® DeepStream
reference application.

7. APPLICATION ARCHITECTURE
The list of GStreamer plugins implemented by NVIDIA and provided as a part of DeepStream SDK include:
• The Stream Muxer plugin (Gst-nvstreammux) to form a batch of buffers from multiple input sources.
• The NVIDIA TensorRT™ based plugin (Gst-nvinfer) for primary and secondary (attribute classification of primary objects) detection
and classification respectively.
• The OpenCV based tracker plugin (Gst-nvtracker) for object tracking with unique ID.
• The Multi Stream Tiler plugin (Gst-nvmultistreamtiler) for forming 2D array of frames.
• The Onscreen Display (OSD) plugin (Gst-nvdsosd) to draw shaded boxes, rectangles and text on the composited frame using the
generated metadata.
• The Message Converter (Gst-nvmsgconv) and Message Broker (Gst-nvmsgbroker) plugins in combination to send analytics data to
a server in the Cloud.
Configuration Groups
• The application configuration is divided into groups of configurations for each component and application-specific component.

8. REMOTELY MANAGING AND CONTROLLING THE
APPLICATION
• supports bi-directional communication to send and receive cloud-to-
device messages
• This is particularly important for various use cases, such as triggering
the application to record an important event, changing operating
parameters and app configurations, over-the-air (OTA) updates, or
requesting system logs and other vital information.

9. REMOTELY MANAGING AND CONTROLLING THE
APPLICATION
• For Device-to-cloud messaging: The Gstnvmsgbroker plugin by default calls
the lower-level adapter library with the appropriate protocol.
• You can choose between Kafka, AMQP, MQTT, or Azure IoT, or you can
even create a custom adapter.
• For cloud-to-edge messaging: The supported protocol in DeepStream 5.0 is
Kafka using the new low-level msgbroker library, interacting directly with the
DeepStream application.
• DeepStream also supports OTA updates of AI models while the application is
running.

10. SMART VIDEO RECORDING
• There is often a need to have event-based video recording. Instead of
continuously recording the content, smart recording can save valuable
disk space.
• Smart record only records the event when specific rules or conditions
are met. The trigger to signal the record can come locally from the
app, from some service running at the edge or from the cloud. provide
faster searchability.
• When an event must be recorded, it’s useful to start saving the clip
before the trigger.
• The smart recording module keeps a video cache so that the
recorded video not only has frames after the event is generated.

11. SMART VIDEO RECORDING
• Recording can be triggered by JSON messages received from the
cloud. The message format is as follows:
The source code for deepstream-test5 can be found in the following directory:
$DEEPSTREAM_DIR/sources/apps/sample_apps/deepstream-test5/
The implementation of the smart record event manager can be found in the following file:
$DEEPSTREAM_DIR/sources/apps/apps-common/src/deepstream_source_bin.c

12. OTA MODEL UPDATE
• With DeepStream 5.0, you can now update a model while the
application is running.
• This means that models can be updated with zero-downtime. This is
important for mission-critical applications that can’t accept any latency.
• This feature can also be useful when models need to be swapped
continuously.

13. SECURE COMMUNICATION
• With DeepStream 5.0, the Kafka adapter supports secure
communication using TLS-based encryption, thereby ensuring the
confidentiality of data.
• TLS (Transport Layer Security) is a successor to SSL.
• DeepStream 5.0 supports two forms of client authentication: two-way
TLS authentication based on SSL certificates and SASL/Plain
authentication based on a username/password mechanism.

14. THANKS FOR
YOUR
ATTENTION