For the full video of this presentation, please visit: https://www.edge-ai-vision.com/2021/07/software-defined-cameras-for-edge-computing-of-the-future-a-presentation-from-arm/ Parag Beeraka, Head of the Smart Camera and Vision Business at Arm, presents the “Software-Defined Cameras for Edge Computing of the Future,” tutorial at the May 2021 Embedded Vision Summit. Computer-vision-enabled cameras have demonstrated the potential to bring compelling functionality to numerous applications. But to realize the full potential and assist with the growth of AI-enabled cameras, it’s necessary to drastically simplify the work of developing, deploying and maintaining these cameras. Hardware and firmware standards are key to accomplishing this. In this talk, Beeraka introduces Arm’s vision for a set of hardware and software standards addressing four key elements of software-defined cameras: security, machine learning, cloud enablement and software portability. For example, smart camera machine learning workloads can run on a variety of processing engines. Common frameworks are needed so that these workloads can be seamlessly and efficiently mapped onto the available processing engines without requiring that the camera developer delve into processing engine details. Similarly, most smart cameras are starting to rely on cloud services for storage as well as model and software updates. Standardizing interfaces to these key elements will give smart camera developers the ability to quickly integrate the cloud services best matched to their needs, without having to master the details of those elements.

1. © 2021 Arm
Software Defined Cameras for
Edge Computing of the future
Parag Beeraka
Head of Smart Camera and Vision Business
Arm

2. © 2021 Arm
“Software Defined” – The Trend
Today
Future
Technology
Enablers
Software Defined Embedded Edge
Orchestrator
Containers
Virtualization
Enterprise
Operating System
Software
Defined
Vehicle
Software
Defined
Industrial
Systems
Software
Defined
Camera
Software Defined Data Center
Software
Defined
Compute
Software
Defined
Networking
Software
Defined
Storage

3. © 2021 Arm
Transformation of Computer Vision and Camera Products
Underpinned by 3 Key Trends
3
Run cloud-native applications
and deploy services over time
Cloud Native
Increased
Security Focus
Resilient and secure
deployments
Shift to
Edge Compute
AI-enabled
intelligent cameras

4. © 2021 Arm
Shift to Edge Compute
4
Person 1 99%
Access Granted
Person 2 99%
Access Granted
Location and Tracking
• Identify people and other
objects
• Track direction and motion
to predict location
• Enable hotspot detection
Automatic Boundaries
• Create boundary conditions
quickly with Image
Segmentation
Increasing Intelligence
• Estimate actions with pose
estimation and understanding
• Ability to interact verbally with
security system
Smart and Secure
• Grant access onto property
• Privacy preserving with secure
local inference
• Third party model/IP
protection

5. © 2021 Arm
The Arm AI Platform and Code Generation
5
LLVM
Applications
Tools
CPU
Cortex-A
Neoverse
GPU
Mali
NPU/microNPU
Ethos-N
CPU
v8 MVE
Cortex-M
OpenCL Ethos-N/U Driver
LLVM
CMSIS-NN
Ethos-U
TVM
Relay
NPU
Support Lib
microNPU
Optimizer
TFL TF keras
PyTorch ONNX
MXNet
Auto
TVM
Code Generation Technology for the
Arm AI Platform

6. © 2021 Arm
Increased Security Focus
6
Root of Trust
Data Security
End-End
Security
(Edge to Cloud)
Data Privacy
Secure Boot
Trusted
Firmware
Secure Key
Management
Any Platform, Any Architecture,
Any Hardware
Any Workload, Any Programming
Language, Any Container Runtime, Any
Packaging
Discrete
Trusted
Platform
Module
Firmware
Trusted
Platform
Module
Local
Hardware
Security
Module
Remote
Hardware
Security
Module
Trusted
Apps
Requirements
Solution
+

7. © 2021 Arm
Cloud Native Provides Camera​ and Vision Products with
Machine Learning Capabilities
7
OS
CPU NPU GPU ISP
Software
Defined
ML
Software Defined Camera platform
Container Container
Vision
Algorithms
/ Camera
Services
Cloud Service Provider
Training
Data Store
Train Model
Model
Deployment
Orchestrator
Customer
App Store
Download Model
Upload Field Data
Service Oriented Architecture
Concurrent Execution Of Inferencing
On-Demand Service Deployment Enabling Post
Sale “App Store”
Continuous Online ML Model Updates
Use Case Scenarios

8. © 2021 Arm
Embrace a Standards-based Ecosystem
Arm is working with many ecosystem partners (OEMs, ODMs, silicon vendors) to drive a standards
approach in order to scale deployments by replacing custom solutions with standards-based solutions and
become the platform of choice for future camera deployments
8
SiP A
FW/OS
SDK
Apps…
SiP B
FW/OS
SDK
Apps…
SiP C
FW/OS
SDK
Apps…
Common OSes
Apps… Apps… Apps…
Ref
Board
Custom
solutions per
Silicon Provider
(SiP)
SiP A SiP B SiP C
FW FW FW
Arm
SystemReady
Standards-based Solution
Custom Solutions

9. © 2021 Arm
Cameras and Vision Products can be “Software Defined”
9
FUNCTION A FUNCTION B FUNCTION C
Hardware abstraction
Camera Hardware
❑ Functions enabled by software are abstracted
from hardware
❑ Functions enabled using cloud-native Service-
Oriented Architecture (SOA) software
development model:
✓ Functions delivered as services are self-
contained units of software
✓ System for publishing available services to the
camera
✓ Centralized management of these services
❑ Ease of integration into Cloud Service Providers
infrastructure
❑ Seamless software and machine learning
development on the cloud and deployment at
the edge

10. © 2021 Arm
Software Defined Camera Architecture Proposal
10
Camera and Vision Software Applications
Containerized Multi ML
Model / Inference Engines
Containerized
Analytics/Storage
Firmware (UEFI, U-Boot,TF-A)
OS (Linux)
Container Run times – K3S
OpenCV
Security – PSA, PARSEC
Communication – RSTP, REST, SOAP
Platforms
OpenCV
Arm NN, TVM
Vulkan MLA Runtime
V4L2 BL
AS
Cloud Service Providers

11. © 2021 Arm
Arm’s Product Portfolio for Camera and Vision Market
11
Balanced performance
and efficiency
Cortex-A53/A55
Cortex-A35
Highest efficiency,
smallest area
Cortex-A76/A78
Highest performance
Balanced performance
and efficiency
Mali-G57
Mali-G78
Highest performance
Configurable to support
0 - 10 TOP/s
Ethos-N78
Ethos-N78
Configurable to support
0 - 10 TOP/s
Ethos-N78
Configurable to support
0 - 10 TOP/s
Cortex-M
Highest efficiency,
smallest area
Ethos-U55/U65
Configurable to support
0 – 1 TOP/s
Linux
RTOS
Arm Mali Image
Signal Processors
Arm CryptoCell
Battery
Powered
Low
Mid
High
+

12. © 2021 Arm
Arm is Bringing Compute, Imaging and Artificial
Intelligence Capabilities to all Devices
12
Arm scalable ML
computing solutions enable
devices to
Think
Cortex-A and Cortex-M
activate devices to
Sense
Advanced Mali-Image Signal
Processors empower devices
to
Mali GPU
Cortex-A
Arm Heterogenous Computing solutions
Cortex-M
+
+
Arm Compute Library/ CMSIS-NN / Arm NN
software libraries and open source stack
Arm ML processors
Arm Mali-Image
Signal Processors
Secure Subsystem
See

13. © 2021 Arm
Summary
• The transformation of computer vision and camera products are underpinned by 3 key trends: The
shift to edge compute, the focus on security and cloud native enabling machine learning capabilities
• It is essential to embrace a standards-based approach to speed up the transition to make cameras
“Software Defined” thus expediting the adoption of vision products to drive the next wave of
computing
• Arm is working with many ecosystem partners (OEMs, ODMs, silicon vendors) to drive a standards
approach in order to scale deployments by replacing custom solutions with standards-based solution
13

14. © 2021 Arm
Resources
14
2021 Embedded Vision Summit
Arm demonstrations:
• An Open Source Approach to Cloud Native
Vision Workload Deployment on Arm
• Moving the Gym to Your Living Room with
Body Pose Tracking on Your mart TV
• Real-time Object Tracking with OpenMV
Arm Products
Arm CPUs and NPUs
https://www.arm.com/products/silicon-ip-cpu
Arm Multimedia
https://www.arm.com/products/silicon-ip-
multimedia
Arm Solutions
https://www.arm.com/solutions/smart-cities
https://www.arm.com/solutions/industrial
https://www.arm.com/solutions/iot

15. Thank You