For the full video of this presentation, please visit: https://www.edge-ai-vision.com/2023/07/modernizing-the-development-of-ai-based-iot-devices-with-wedge-a-presentation-from-midokura-a-sony-group-company/ Dan Mihai Dumitriu, Chief Technology Officer of Midokura, a Sony Group Company, presents the “Modernizing the Development of AI-based IoT Devices with Wedge” tutorial at the May 2023 Embedded Vision Summit. IoT device development has traditionally relied on a monolithic approach, with all firmware developed by a single vendor using a rigid waterfall model, typically in C, and infrequently updated. This paradigm is insufficient. For AI-enabled IoT devices to reach their potential, developers must be able to easily program and update low-cost, low-power AI-capable sensors, such as Sony’s IMX500. In this talk, Dumitriu presents Wedge—a combination of runtime, device agent and cloud service. Wedge automates software life-cycle management for devices, provides isolation and enables agile development. Wedge is based on WebAssembly, a binary instruction format for a stack-based virtual machine. Wedge’s runtime and memory overhead are within 2x of native C/C++ code, and it supports languages such as Python and AssemblyScript. Dumitriu also presents the vision sensing pipeline, a sensor data processing layer developed on top of Wedge, which is accessible via REST API and a visual interface.

1. © 2023 Midokura (Sony Group)
Modernizing the
Development of AI-
Based IoT Devices with
Wedge
Dan Mihai Dumitriu
CTO
Midokura (Sony Group)

2. © 2023 Midokura (Sony Group)
• Sony Semiconductor Solutions is the largest vendor of CMOS image sensors
• AITRIOS is a new business to develop sensing solutions based on the sensor
portfolio
• Embedded Edge AI and IoT are key ingredients
• Midokura is a subsidiary of Sony Semicon focused on software infrastructure
Introduction
2

3. © 2023 Midokura (Sony Group)
Sony AITRIOS
3
Address privacy concerns
by elaborating metadata-only
Low power and
cost friendly
Faster and optimized
on-sensor AI processing
Less bandwidth needed
and very low latency
AI-enabled
image sensor
(IMX500)
AI/device
management
cloud service
New type of
AI/computer vision
hardware platform
IMX500

4. © 2023 Midokura (Sony Group)
An Accessible Platform that Enables Intelligent
Solutions Based on Distributed Visual Sensing
4
Advanced vehicle
assistance
Smart
Manufacturing
Retail stores
Retail stores
Smart Home
Bring to market a
plethora of low cost,
easy-to-use sensing
devices
Low barrier of entry for
solution developers
Agile development of
sensing applications
Low operational cost of
vision sensing apps
Polyglot Development Marketplace to connect
AI Developers & Solution
Developers
Targeting solution developers for various vertical applications.

5. © 2023 Midokura (Sony Group) 5
Problems of Development on Tiny
IoT Devices

6. © 2023 Midokura (Sony Group)
• Operating system (RTOS) is effectively like a library
• Application and OS must be tested together
• Waterfall development
• Heavy testing process late in development cycle
• Cannot continuously deploy applications
Development is Not Agile
6

7. © 2023 Midokura (Sony Group)
• Tiny IoT devices are based on MCUs (microcontroller units)
• MCU typically cannot do virtual memory
• Without memory isolation, cannot do agile development
• Dynamic deployment of applications is unsafe
No Safety and Isolation
7

8. © 2023 Midokura (Sony Group)
• High barrier of entry for application developers
• Apps are typically written in C
• Poor code reuse across device types
• HW specific interfaces and drivers
• AI developers typically don't know C well
• Develop mostly in Python
Application Development Is Difficult
8

9. © 2023 Midokura (Sony Group) 9
Introducing WEdge

10. © 2023 Midokura (Sony Group)
• Like Kubernetes, but for tiny IoT devices
• Automated lifecycle management of workloads
• Strong isolation of modules
• Leverages WebAssembly Micro Runtime (WAMR)
• WEdge cloud does automatic optimization for target device
• Polyglot SDK enables developer productivity
Intro to WEdge
10

11. © 2023 Midokura (Sony Group)
• WebAssembly (Wasm) is a low-level bytecode format that runs in a sandboxed
environment
• Designed as a portable target for compiling high-level languages like C, C++, and Rust
• Runs on various platforms, including browsers, servers, and now even on tiny IoT devices
• Compact size, fast execution, and high security compared to other runtime environments
• AoT (Ahead of Time) compilation supports multiple target ISAs via LLVM backends
• High runtime performance - within 2x of native
Intro to WebAssembly
11

12. © 2023 Midokura (Sony Group)
• Combination of language-level and runtime-level protections
• Linear memory model
• All memory accesses are bounds-checked to prevent buffer overflows or
underflows
• Enforces type safety at both compile-time and runtime
• Control Flow Integrity (CFI) prevents control flow hijacking attacks
• WebAssembly System Interface (WASI) for standardized and secure way to
access system resources
• Fine grained control
WebAssembly for Sandboxing
12

13. © 2023 Midokura (Sony Group)
WEdge Device Stack
13
WEdge Services API
OS
Abstraction
Layer
WASI
Web Assembly Micro Runtime
Native Libraries & Device Drivers
OS + BSP
HW
Module
1
. . .
Module
2
Module
3
Module
4
Module
N

14. © 2023 Midokura (Sony Group)
• Sensors
• Read image
• Configure (e.g. frame rate)
• Communication
• Send telemetry to cloud
• HTTP PUT/GET/POST
• Other protocols (e.g. CAN)
WEdge Services API
14
• AI/ML
• Load model
• Run inference
•
Data storage
• Local DB
• Distributed DB cache

15. © 2023 Midokura (Sony Group)
SDKs in various languages
Polyglot Development
15

16. © 2023 Midokura (Sony Group)
Decouple from Target Architecture & OS
16

17. © 2023 Midokura (Sony Group)
• Enables more flexible and dynamic development cycles
• Devs can break down complex software into smaller, more
manageable modules that can be developed, tested, and
deployed independently
• Polyglot development helps devs code in their language of choice
• For AI developers, that is Python
Agile Development Enabler
17

18. © 2023 Midokura (Sony Group) 18
Vision Sensing Applications

19. © 2023 Midokura (Sony Group)
• Programming sensing applications is complex
• Code reuse is typically poor and applications are monolithic
• VSA models a complex application as a series of simple nodes
• Promotes code reuse
• Enables low-code development via API or GUI
• VSA nodes are deployed as Wasm modules
Vision Sensing Applications (VSA)
19

20. © 2023 Midokura (Sony Group)
Goal: Make an intelligent sensor perform a complex task
How: Model this complex task as a sequence of individual small tasks
Complex task => VSA
Small tasks (partial) => Nodes of the VSA
Why?
● Reusable nodes - can have different creators
● Allow to customize an existing application to a new use case by modifying some nodes
● Security: Use Webassembly technology with its sandboxing approach to allow custom
code without impact on the device security
Vision Sensing Application (VSA)
20

21. © 2023 Midokura (Sony Group) 21
Example Application: People Counting

22. © 2023 Midokura (Sony Group)
People Counting VSA
22

23. © 2023 Midokura (Sony Group)
VSA of People Counting Application
23

24. © 2023 Midokura (Sony Group)
• WebAssembly is a great choice for tiny IoT devices
• High performance and low overhead
• WEdge enables high development productivity and agility
• Vision Sensing Applications (VSAs) are a convenient paradigm for
programming sensing applications
• Promoting component reuse
• Enabling low-/no-code development
Conclusion
24

25. © 2023 Midokura (Sony Group)
Further Information
25
Midokura
https://www.midokura.com
AITRIOS
https://www.aitrios.sony-semicon.com/en/
IMX500
https://developer.sony.com/develop/imx500/
Demo Video
https://bit.ly/mido-vsa-demo
Please visit our booth!

26. © 2023 Midokura (Sony Group) 26
Thank you