The document discusses NNStreamer, an open source project for developing on-device and among-device AI systems using stream pipelines. It proposes expanding NNStreamer to support registration and deployment of AI services, as well as among-device training to enable AI personalization across different devices. Several application examples are provided, including augmented reality for factory workers that can monitor activities and notify when corrections are needed.

1. Toward Among-Device AI
from On-Device AI
with Stream Pipelines
https://nnstreamer.ai
MyungJoo Ham, Ph.D.
May, 2022
NNStreamer is a project hosted by Linux Foundation AI & Data
MyungJoo Ham, Sangjung Woo, Jaeyun Jung, Wook Song, Gichan Jang, Yongjoo Ahn, Hyoungjoo Ahn

2. https://nnstreamer.ai https://github.com/nnstreamer
Why On/Among-Device AI?
Privacy & Data Protection
Latency
Cost
Privacy, Latency, Cost!
vs
vs
vs

3. https://nnstreamer.ai https://github.com/nnstreamer
Recap: On-Device AI Pipeline Framework
ICSE 2021 SEIP, “NNStreamer: Efficient and Agile Development of On-Device AI Systems”
Neural Networks == Filters of Stream Pipelines
On-Device AI Services as Pipelines
(Gstreamer) Pipeline description
Implementation
• Lower devel cost
• Lower CPU/mem overhead
• Higher throughput
• Higher porttability
Deployed for wearable, TV, mobile,
obotic vacuum, oven, … in Samsung
Other users: NXP, FAINDERS.AI, KLLEON, ……

4. https://nnstreamer.ai https://github.com/nnstreamer
Background: Applications
Augmented Worker (Factory)
Identify Task
Monitor Activities
(hand movement,
vibration, sound)
Check if the
activity is valid
(e.g., complete
assembly?)
Notify if the recent
activity requires further
actions / corrections
Order to send data

5. https://nnstreamer.ai https://github.com/nnstreamer
Background: Applications
Augmented Worker (Factory)
Monitor Activities
(hand movement,
vibration, sound)
Notify if the recent
activity requires further
actions / corrections
Identify Task
Pre-
process
decision
Data storing
decision
send data
Pre-
process
Pre-
process
Check if the
activity is valid
(e.g., complete
assembly?)
Notify
REASSEMBLE!

6. https://nnstreamer.ai https://github.com/nnstreamer
Evaluation Result: Application Examples
Augmented Worker (Factory)
System expressed in < 20 lines, with no noticeable overheads

7. https://nnstreamer.ai https://github.com/nnstreamer
Background: Applications
Consistent AI Services: Quality & Context
+ 3rd parties!

8. https://nnstreamer.ai https://github.com/nnstreamer
Future Direction / New Requirements
Separate AI service (pipeline writers) and AI application (pipeline users) developers
AI Service Deployment
옷 옷
AI Service Developer
(pipeline & model writer)
AI App Developer
(pipeline user)
NNStreamer 2.2+
“ml-service”
Register/update/launch/deploy
pipelines & models
Call Pipelines &
models
remote or not!

9. https://nnstreamer.ai https://github.com/nnstreamer
Future Direction / New Requirements
Among-Device Training & AI Personalization
Lightweight Device
(Cortex M4)
High-performance Device
(Cortex A7x + GPU + NPU)
Inference Pipeline
Training Pipeline
(AI Personalization)
Data Source Model
Data to train
Data Source’
Model
updates

10. Open Source Community
• Join us @ github.com/nnstreamer/nnstreamer
We already have contributors and users from
- NXP, Collabora, Fainders.AI, Klleon.io, ......
- Samsung Products include: mobile, wearable, TV, robotic
vacuum, robotics, oven, fridgers, and many more!