For the full video of this presentation, please visit: https://www.edge-ai-vision.com/2023/07/using-a-neural-processor-for-always-sensing-cameras-a-presentation-from-expedera/ Sharad Chole, Chief Scientist and Co-founder of Expedera, presents the “Using a Neural Processor for Always-sensing Cameras” tutorial at the May 2023 Embedded Vision Summit. Always-sensing cameras are becoming a common AI-enabled feature of consumer devices, much like the always-listening Siri or Google assistants. They can enable a more natural and seamless user experience, such as automatically locking and unlocking the device based on whether the owner is looking at the screen or within view of the camera. But the complexities of cameras, and the quantity and richness of the data they produce, mean that much more processing is required for an always-sensing camera compared with listening for a wake word. Without careful attention to neural processing unit (NPU) design, an always-sensing camera can wind up consuming excessive power or performing poorly, which can lead to an unsatisfactory user experience. In this talk, Chole explores the architecture of a neural processor in the image signal path, discusses use cases, and provides tips for how OEMs, chipmakers, and system architects can successfully evaluate, specify, and deploy an NPU in an always-on camera.

1. Using a Neural Processor
for Always-Sensing
Cameras
Sharad Chole
Chief Technologist, Co-founder
Expedera

2. • Like always-listening Siri, always-sensing enables a more natural and seamless
user experience
• Quality and richness of camera data require much more processing than audio
implementations
• Like always-listening, always-sensing must be processed locally
• Technical requirements:
• ~500 GOPS to 1 TOPS NPU
• Ultra-low power
• Ultra-small area
• Multiple models
Always-sensing Cameras: What and Why
2
© 2023 Expedera Inc.

3. • Optimized edge AI inference IP solutions based on
revolutionary packet architecture, application-
configured for our customers
• Silicon Valley startup founded in 2018
• 3 R&D centers, numerous patents
• Broad, worldwide deployments
• 10M+ devices in-field
• Over 200 ExaOps (200,000,000,000,000,000,000
operations/second) deployed by our customers
• Soft IP: designs in multiple leading-edge nodes
About Expedera
3
© 2023 Expedera Inc.

4. • Like always-listening Siri, always-sensing enables a more natural and seamless
user experience
• Quality and richness of camera data require much more processing than audio
implementations
• Like always-listening, always-sensing must be processed locally
• Technical requirements:
• ~500 GOPS to 1 TOPS NPU
• Ultra-low power
• Ultra-small area
• Multiple models
Always-sensing Cameras: What and Why
4
© 2023 Expedera Inc.

5. • NPUs are used in wake-word (audio)
applications - why can’t the same approach
apply to video?
• Workloads – Edge-friendly vision NNs
• ISPs/DSPs aren’t designed for this sort of
specialized processing
• We asked ourselves – rather, customers asked
us – can we apply the same low power, small,
targeted workload approach to video?
• We have; otherwise, I wouldn't be giving this talk
NPUs and Always-Sensing Are an Ideal Match
© 2023 Expedera Inc.
0
2
4
Yolo V5n
MobileNet V3s
NanoDet+
0
2
4
Yolo V5n
MobileNet V3s
NanoDet+
0
15
30
Yolo V5n
MobileNet V3s
NanoDet+
Weights
(Million)
Operations
(Billion)
Activations
(Million)
5

6. • Secure access: facial recognition
• Power management: “find a face” detection to turn on/down/off display
• Gesture recognition: Innovative UX control and operability
• Motion detection: bandwidth-friendly security
• Object detection: capturing events or triggers
• Privacy: “shoulder surfing” alerts
Always-sensing Use Cases
6
© 2023 Expedera Inc.

7. Existing Solutions are Limited
7
Applications Processor
D
D
R
CPU
GPU/DSP
ISP
• NPUs, GPU/DSPs, and ISPs are standard
in Application Processors
• The system NPU – the “Big” NPU – is
not a good match for always-sensing
• Excessive power consumption
• Privacy, memory & data security
concerns
• Contention by multiple applications
• “Hitting a nail with a piledriver”
NPU

8. • Purpose-designed for smallest area, lowest
power consumption, and target networks
• Area: reduces cost
• Latency: programmable, guaranteed FPS
• Power consumption: conserves battery life
• No requirement for external memory
• Keep processing and data storage local
• Increased power efficiency
• Better security and privacy
The Case for a Little NPU: Ideal Always-sensing
8
© 2023 Expedera Inc.
Specification Big NPU Little NPU
Area ~5-8X 1X
Subsystem
Size
>10X 1X
Latency
Contention w/
activity
Deterministic
Data Exposure System DDR
Within always-
sensing
subsystem

9. 3: Fully-integrated AP
2: Co-processor
1: Near-sensor processing
Big/Little NPU Architecture Options
9
© 2023 Expedera Inc.
ISP Applications Processor
D
D
R
Little
NPU
ISP
Little
NPU
Frame
Buffer
Weight
Storage
Little
NPU
Applications Processor
D
D
R
CPU
GPU/DSP
ISP
Big NPU
Applications Processor
D
D
R
CPU
GPU/DSP
Big NPU
CPU
GPU/DSP
Big NPU

10. Expedera’s NPU Architecture
10
© 2023 Expedera Inc.
Minimal memory
requirements via
Packet Set
Architecture
Ideal power and
processing
efficiencies
Optimized for best
performance per
area

11. • Packet - aggregate of work with a notion of
dependencies and deterministic execution
• A contiguous fragment of a NN layer with the entire
context of execution: layer type, attributes, priority
• Packets manage activations better/more
intelligently
• Results: minimum number of moves without
hurting accuracy
• Greatly increases performance while lowering power
and area requirements
Packets: A Radical Approach to AI Inference
© 2023 Expedera Inc. 11
Neural Network layers
Layer broken down into packets
Packet stream natively executable on NPU
Can execute in parallel

12. Expedera OriginTM NPU Engine
12
© 2023 Expedera Inc.
• Revolutionary packet-based architecture reduces
design and implementation complexity while
improving real-world performance
• Hardware and software are designed together
• Solves complex software in hardware – compiler co-
designed and optimized for unique architecture
• Just-in-time memory management implements a
unified compute pipeline
• Expedera-optimized for customer use case(s)
18 TOPS/W
ResNet50 INT8 in TSMC 7nm @ 1GHz
No sparsity, compression, or pruning applied,
though supported
70-90%
Average sustained NPU utilization
across common networks
0.003 ~ 128 TOPS
Single core performance,
PetaOps with multi-core
SSP
VSP Logic
VSP Memory MMP
PSM
MMP
PSM
MMP MMP
VSP Logic
VSP Memory
Origin Architecture: 5 fundamental building blocks

13. Architectural Building Blocks
13
© 2023 Expedera Inc.
• Scalable matrix math processors (MMP)
perform weight multiplication.
• Vector scalar processing (VSP) logic
handles memory access, data reshaping, and
some vector operations.
• VSP memory provides storage for input,
output, and intermediate activations.
• Accumulation buffering and quantization logic
happen in partial summation (PSUM) blocks.
• Orchestrating operations is a single
sequence/scheduler processor (SSP)
working with compiler software for a unique
packet-based sequencing approach.
SSP
VSP Logic
VSP Memory MMP
PSM
MMP
PSM
MMP MMP
VSP Logic
VSP Memory
Origin Architecture: 5 fundamental building blocks
Decoupled building blocks,
optimized for workload needs

14. Packets: Superior Memory Optimization
14
© 2023 Expedera Inc.
• Expedera’s packet-based architecture
requires minimal external memory
(weights only) for the networks shown
• Higher throughput
• Lower system power
• Better privacy and security
• Uniformly spread-out bandwidth
• Sustained utilization
• Tolerance towards latency variations
0
5
10
15
20
25
30
35
40
Million
Bytes
Bandwidth Requirements @ 0.5MB of NPU Memory
Yolo v5N Layer-Based Yolo v5N Packet-based
MobileNet V3s Layer-Based MobileNet V3s Packet-based
NanoDet+ Layer-Based NanoDet+ Packet-based

15. Customer-provided, Field-proven Results
© 2023 Expedera Inc.
Competition
FPS Comparison
Competition
With Expedera
Power Consumption (mW)
15
Customer-provided data, 4K video rate low light denoising:
20X faster throughput using less than half the power
“Expedera Redefines AI Acceleration for the Edge”
- TechInsights (Linley) Microprocessor Report, April 2021

16. Best Practices for Always-sensing Designs
16
© 2023 Expedera Inc.
Little NPUs achieve necessary
performance within strict power
and area budgets;
don’t settle for the big NPU
Memory management
Keep all data within the always-
sensing subsystem, extending
battery life while enhancing
security & privacy

17. Resources
Summit & Alliance Resources
• Visit us at booth #319
• Alliance website
• https://www.edge-ai-
vision.com/companies/expedera/
Expedera Resources
• Company Website
• http://www.expedera.com/
• White papers, technical briefs, webinars, other
• Pre-silicon PPA Estimations
• Want cycle-accurate PPA numbers for your use
case(s) well before silicon?
• info@expedera.com
• Contact us directly
• info@expedera.com
17
© 2023 Expedera Inc.