ICSE 2022 SEIP presentation material of paper, "Toward Among-Device AI from On-Device AI with Stream Pipelines".

1. Toward Among-Device AI
from On-Device AI
with Stream Pipelines
https://nnstreamer.ai
MyungJoo Ham, Ph.D.
May, 2022
NNStreamer is a project hosted by Linux Foundation AI & Data
MyungJoo Ham, Sangjung Woo, Jaeyun Jung, Wook Song, Gichan Jang, Yongjoo Ahn, Hyoungjoo Ahn

2. https://nnstreamer.ai https://github.com/nnstreamer
Background: Applications
Augmented Worker (Factory)
Identify Task
Monitor Activities
(hand movement,
vibration, sound)
Check if the
activity is valid
(e.g., complete
assembly?)
Notify if the recent
activity requires further
actions / corrections

3. https://nnstreamer.ai https://github.com/nnstreamer
Background: Applications
Robotic Vacuum (= Mobile Camera) + Pet Care Services

4. https://nnstreamer.ai https://github.com/nnstreamer
Background: Applications
Consistent AI Services for Home Appliances
Consistent Service Quality & Performance
+ Consistent Context!

5. https://nnstreamer.ai https://github.com/nnstreamer
Objectives
Devices from different manufacturers!!
Consistent AI Services for All Devices

6. https://nnstreamer.ai https://github.com/nnstreamer
Recap: On-Device AI Pipeline Framework
ICSE 2021 SEIP, “NNStreamer: Efficient and Agile Development of On-Device AI Systems”
Neural Networks == Filters of Stream Pipelines
On-Device AI Services as Pipelines
(Gstreamer) Pipeline description
Implementation
• Lower devel cost
• Lower CPU/mem overhead
• Higher throughput
• Higher porttability

7. Requirements

8. https://nnstreamer.ai https://github.com/nnstreamer
Requirements of Among-Device AI Pipelines
Each AI or input/output service exists atomically and is deployed independently. Such a
service includes an AI inference processor, a shared input stream, or a user interface.
R1, deployability
CAMERA SRC / by Team A
AI Service 001 / by Team B
AI Service X / by Team B
AI Service Y / by Team C
Display SINK / by Team A
CAMERA SRC / by Team A
App 003 / by Team D
AI Service 002 / by Team A

9. https://nnstreamer.ai https://github.com/nnstreamer
Requirements of Among-Device AI Pipelines
Transmission between such entities may be schemaless, or updating schema should not
require recompiling or redeploying software. It should be able to compress and
synchronize transmitted data frames from different devices.
R2, portability
CAMERA SRC / by Team A
AI Service 001 / by Team B
AI Service X / by Team B
AI Service Y / by Team C
Display SINK / by Team A
CAMERA SRC / by Team A
App 003 / by Team D
AI Service 002 / by Team A

10. https://nnstreamer.ai https://github.com/nnstreamer
Requirements of Among-Device AI Pipelines
Discovering such entities and connecting to them should not require the knowledge of
specific addresses (e.g., IP address). Besides, there may be multiple entities discovered for a
given connection request, where one of them will be connected.
R3, discoverability
AI Service X / by Team B
AI Service Y / by Team C
AI Service X / by Team D
Need Service “Y”
(e.g., “face id”)
Need Service “X”
(e.g., “speech recog”)
Interchangeable

11. https://nnstreamer.ai https://github.com/nnstreamer
Requirements of Among-Device AI Pipelines
If a connected entity becomes unavailable, it can be automatically connected to an
alternative entity
R4, fault-tolerance
AI Service X / by Team B
AI Service X / by Team D
Need Service “X”
(e.g., “speech recog”)
Serves as
“fault tolerance fallback” for 1
1
2
Serves as
“fault tolerance fallback” for 2

12. https://nnstreamer.ai https://github.com/nnstreamer
Requirements of Among-Device AI Pipelines
Any vendors and application developers may freely use, alter, or redistribute the given
software framework for any purpose without charges. Besides, it is desirable if anyone may
participate in its development to reduce fragmentation and a bias for an affiliation.
R5, openness
CAMERA SRC / by Team A
AI Service 001 / by Team B
AI Service X / by Team B
AI Service Y / by Team C
Display SINK / by Team A
CAMERA SRC / by Team A
App 003 / by Team D
AI Service 002 / by Team A

13. https://nnstreamer.ai https://github.com/nnstreamer
Requirements of Among-Device AI Pipelines
It should be extensible for different platforms. In other words, it should be able to be
connected with different AI pipeline frameworks (e.g., MediaPipe [12] and DeepStream [15])
or services with different operating systems, especially those for microcontrollers (a.k.a.
RTOS).
R6, extensibility
Ultra-lightweight device (RTOS)
DeepStream Pipeline in
Jetson or AWS/Azure
AI Service 001 / by Team B
AI Service X / by Team B
AI Service Y / by Team C
Display SINK / by Team A
CAMERA SRC / by Team A
AI Service 002 / by Team A
“extremely portable”
communication
base class

14. https://nnstreamer.ai https://github.com/nnstreamer
Requirements of Among-Device AI Pipelines
Each AI service is an on-device AI application that has requirements mentioned in [8].
 Don’t deteriorate the previous work. (backward compatibility?)
R7

15. Design

16. https://nnstreamer.ai https://github.com/nnstreamer
Design & Features
Protocol: Pub/Sub based Stream Sink/Src
Publisher Subscriber
Broker
(MQTT)
Pipeline Pipeline
mqttsink mqttsrc
…… ……
Data*
Connection &
Control
I publish
“camera/left/1920x1080”
I need
“camera/left/*”
* In 2021, data-control
separation was WIP.
(applied in query only)
Since NNStreamer 2.1, this is
applied to pub/sub, too.

17. https://nnstreamer.ai https://github.com/nnstreamer
Design & Features
Protocol: Query, “Inference Off-loading”
+ Data-Control Separation
Pipeline (“client”)
Pipeline (“server”)
Query Client
……
Query Server Src Query Server Sink
……
……
Publisher
Subscriber
“obj-detect/homeenv/yolov3/mod”
“obj-detect/homeenv/*”
Pipeline (“server”)
Pipeline (“server”)
“obj-detect/homeenv/ssd/mod”
“obj-detect/homeenv/yolov4/org”
Pipeline (“client”)
Pipeline (“client”)
Pipeline (“client”)
Pipeline (“client”)
Pipeline (“client”)
Pipeline (“client”)
Pipeline (“client”)
Pipeline (“client”)
Broker (MQTT)
(control/connection only)

18. https://nnstreamer.ai https://github.com/nnstreamer
Design & Features
• Synchronization & Timestamp Management
• “NNStreamer-Edge”
Lightweight Library for Pub/Sub & Query
• For non-GStreamer / non-pipeline processes
• Note: this is merged with another open source project, “AITT”
(MQTT for Distributed AI services)
Others..

19. Evaluations & Applications

20. https://nnstreamer.ai https://github.com/nnstreamer
Evaluation Result: Application Examples
Augmented Worker (Factory)
Identify Task
Monitor Activities
(hand movement,
vibration, sound)
Check if the
activity is valid
(e.g., complete
assembly?)
Notify if the recent
activity requires further
actions / corrections

21. https://nnstreamer.ai https://github.com/nnstreamer
Evaluation Result: Application Examples
Augmented Worker (Factory)
System expressed in < 20 lines, with no noticeable overheads

22. https://nnstreamer.ai https://github.com/nnstreamer
Evaluation Result: Performance
MQTT vs ZeroMQ: MQTT slows down to 47% with x3.18 CPU overheads!
Performance / MQTT – ZeroMQ comparison

23. https://nnstreamer.ai https://github.com/nnstreamer
Evaluation Result: Performance
MQTT-Hybrid (separatered) vs TCP-raw: No more overheads!
Performance / Query + Data-Control Separation

24. Conclusion

25. https://nnstreamer.ai https://github.com/nnstreamer
Future Direction / New Requirements
Separate AI service (pipeline writers) and AI application (pipeline users) developers
AI Service Deployment
옷 옷
AI Service Developer
(pipeline & model writer)
AI App Developer
(pipeline user)
I want to keep updating
pipelines & models
I want to deploy services
directly.
I do not want to rebuild or update my
code for their changes.
I do not want to “know” the location of
the pipeline or model.
 (WIP) NNStreamer 2.2+
“ml-service”

26. https://nnstreamer.ai https://github.com/nnstreamer
Future Direction / New Requirements
Among-Device Training & AI Personalization
Lightweight Device
(Cortex M4)
High-performance Device
(Cortex A7x + GPU + NPU)
Inference Pipeline
Training Pipeline
(AI Personalization)
Data Source Model
Data to train
Data Source’
Model
updates

27. https://nnstreamer.ai https://github.com/nnstreamer
Future Direction / New Requirements
Matter & SmartThings: Home IoT Standardization
 Inter-device AI Data Stream Standard?
(Video data stream standard: discussion started)
More features & adaptation for “among-device AI”
- More protocols (webrtc, rtsp, ......)
- More OS/device supports (Tizen-RT, robotics, ......)
- Merging w/ another open source project, AITT
AI Data Stream Standard?

28. Open Source Community
• Join us @ github.com/nnstreamer/nnstreamer
We already have contributors and users from
- NXP, Collabora, Fainders.AI, Klleon.io, ......
- Samsung Products include: mobile, wearable, TV, robotic
vacuum, robotics, oven, fridgers, and many more!