Combining Flexibility and Low-power in Embedded Vision Systems

Copyright © 2014 Synopsys Inc. 1
Pierre Paulin, Director R&D
Santa Clara, 29 May 2014
Combining Flexibility and Low-power in
Embedded Vision Subsystems:
An Application to Pedestrian Detection
Bruno Lavigueur, Senior R&D Engineer

• Pedestrian Detection algorithm overview
• Computation and bandwidth requirements
• Embedded Vision Reference Platform
• Programming Tools and Architecture
• Application Mapping to a Heterogeneous Multi-Core
Platform
• From Functional implementation in OpenCV
to a fully optimized mapping to GPP and ASIP cores
• Final optimized mapping
• Power — Performance — Area analysis
• FPGA-based prototype
• Lessons learned, outlook
Outline

• EDA tool and IP provider
• $1.96B in revenue (FY 2013)
• ~8700 employees ( > 5600 R&D engineers)
• ~81 offices worldwide
• Products for Designing Embedded Vision Systems
• Embedded Cores (ARC HS, EM, 600, 700)
• Application Specific Processor (ASIP) design tools
• Semiconductor IP (DDR, DMA, AXI, HDMI, USB, A/D, …)
• Synthesis and verification for SoCs and FPGAs
• FPGA-based rapid prototyping system
Synopsys — EDA Industry Leadership

• Pedestrian detection
• One of the most popular EV applications
• Standard feature in luxury vehicles
• Moving to mid-size and compact vehicles
in the next 5-10 years, also due to
legislation efforts
• Implementation requirements
• Low cost
• Low power (small form factor, and/or battery powered)
• Programmable (to allow for in-field SW upgrades)
• Most popular algorithm for pedestrian detection is
Histogram of Oriented Gradients (HOG)
Pedestrian Detection and HOG

Histogram Of Oriented Gradients
Gradient Computation
Apply Sobel operators:
+1 +2 +1
0 0 0
−1 −2 −1
and
+1 0 −1
+2 0 −2
+1 0 −1
Grey scale
conversion
Scale to multiple
resolutions
Gradient computation
Histogram
computation per block
Normalization of the
histograms
SVM per window
position
Non-max suppression
Scale to Multiple Resolutions
Use a fixed 64x128-pixel detection window.
Apply this detection window to scaled frames.

Histogram Of Oriented Gradients
The image is divided in 8x8-pixel cells. For very block of 2x2 cells, apply
Gaussian weights and compute 4 histograms of orientation of gradients.
Histogram Computation
Normalization of the Histograms
(1) L2 Normalization (2) clipping (saturation) (3) L2 Normalization
Support Vector Machine
Linear classification of histograms
for every 64x128 windows position.
Non-Max Suppression
Cluster multi-scale dense scan of
detection windows and select unique
Grey scale
conversion
Scale to multiple
resolutions
Histogram
histograms
SVM per window
position
Non-max suppression

Embedded Vision
Reference Platform Overview

Embedded Vision Reference Platform
Embedded Vision
Reference Platform
Ported OpenCV library
Pedestrian Detection, etc.
C API to ASIP-based
vision accelerators
Configurable ARC HS
RISC processor
ASIP-based accelerators
HAPS® FPGA-based
prototyping system
Pre-verifiedflow
andexamples

Time-to-market and Flexibility vs.
Power-Performance-Area Trade-offs
Subsystem
Controller
HS
Emb. Vision
Accelerators
ASIP
ASIP
ASIP
1X 100X
P
A
R
A
L
L
E
L
I
S
M
Pre-processing:
- Filtering
- Color conversion
- Image scaling
- Feature extraction
and matching
- Segmentation
Power-Performance-Area Efficiency
Time-to-market, Flexibility
10X
MQX
Lightweight O/S
High-level processing:
- Control
- Multi-object tracking
- Post-processing
- High-level command
interpretation
Data
Level
Parallelism
Task
Level
Parallelism
Sequential
Tasks

• ARC HS family of high-performance cores
• ARC HS 36 Performance, power at 28 nm HPM process (worst case):
• Scalable to 1.6 GHz
• 1.9 DMIPS/MHz
• 37 uW/MHZ
• Application-Specific Instruction-set Processors (ASIP)
• User-driven design of processors tailored to a specific application
• Ability to guide performance-power-area and flexibility trade-offs
• Automatic generation of implementation, C compiler and
programming tools from instruction-set specification
• Connectivity components
• DMA, AXI, DDR, etc.
Main architectural components
53 Dhrystone GIPS/W

Embedded Vision Flow and Architecture
HOG Embedded App.
Base driversMQX runtime
AXI-4 local interconnect
DMA,
Sync
& I/OHS DCCM
Dedicated Streaming Interconnect (FIFOs)
D D D
ASIP1 ASIPn
C/C++ C API to Accelerators
HAPS-70 S12
12M ASIC
Gate equiv.
L2
SRAM
ASIP2

HOG Mapping and Refinement Flow
Camera
HOG
Detection
DVI
Output

• Refinement from an OpenCV high-level functional
description, to a fully optimized multi-processor SoC
combining a GP RISC with multiple ASIPs
• Main steps
• OpenCV functional reference
• Optimization and Porting onto MQX RTOS
• Profiling of all major functions
• Identification of high compute kernels
• Development of ASIPs using Synopsys ASIP design and
exploration tools
• Stepwise refinement
• From GPP only  to GPP + multiple ASIPs
HOG Mapping and Refinement Flow

0.00
5.00
10.00
15.00
20.00
25.00
30.00
35.00
40.00
45.00
50.00
Rescale Grad Hist Norm SVM Other
% of processing
% of processing
ARC and ASIP Exploration Tool Flow
Optimizing
Compiler
Assembler,
Linker
Instrn.-Set
Simulator
Debugger,
Profiler
C code
ARC HS S/W
Optimization
Processor
Description
Language
Optimizing
Compiler
Assembler,
Linker
Instrn.-Set
Simulator
Debugger,
Profiler
RTL
Gen.
Sim, FPGA,
RTL Synthesis
C code
ASIP HW/SW
Optimization
ARC-ASIP Trade-off Exploration
MQX RTOS

Grey scale
conversion
HOG Functional Validation on ARC HS
(640 × 480 pixels)
AXI local interconnect
DMA,
Sync
& I/O
D D D
ASIP1 ASIP2
Rescaling Gradient Histogram SVM
Normali-
zation
Non-max
suppression
ASIP4
L3 Ext. DRAM
DCCMHS
Subs. ctrl
1
• C fixed point profiling results: 2.25 G cycles per frame

ARC HS
G cycles
0.1
1.4
17.3
31.9
1.2
15.7
0.004
Histogram Of Oriented Gradients Profiling
(640 × 480 pixels, at 25 FPS)
Grey scale
conversion
Scale to multiple
resolutions
Histogram
histograms
SVM per window
position
Non-max suppression

(640 × 480 pixels, at 25 FPS)
Grey scale
conversion
Scale to multiple
resolutions
Histogram
histograms
SVM per window
position
Non-max suppression
0.00
5.00
10.00
15.00
20.00
25.00
30.00
35.00
40.00
45.00
50.00
% of processing
% of processing

0.00
2.00
4.00
6.00
8.00
10.00
12.00
14.00
16.00
18.00
# ARC HS
# ARC HS
(640 × 480 pixels, at 25 FPS)
Grey scale
conversion
Scale to multiple
resolutions
Histogram
histograms
SVM per window
position
Non-max suppression
Single Core
Multicore?
Accelerate!

Grey scale
conversion
Task Assignment #2
DMA,
Sync
& I/O
D D D
ASIP1 ASIP2
Normali-
zation
Non-max
suppression
ASIP4
2
L3 Ext. DRAM
DCCMHS
Subs. ctrl1.6 GHz
1.6 GHz
400 MHz

Task Assignment #3
DMA,
Sync
& I/OHS DCCM
Subs. ctrl
D D DD
ASIP1 ASIP2 ASIP3 ASIP4
3
L3 Ext. DRAM
Grey scale
conversion
Normali-
zation
Non-max
suppression
1.6 GHz
400 MHz

Task Assignment #4
DMA,
Sync
& I/OHS DCCM
Subs. ctrl
D D DD
ASIP1’ ASIP2 ASIP3 ASIP4
4
L3 Ext. DRAM
Grey scale
conversion
Normali-
zation
Non-max
suppression
400 MHz
400 MHz

Task Assignment #4 With On-Chip L2
DMA,
Sync
& I/O
D D DD
ASIP1’ ASIP2 ASIP3 ASIP4
4
HS DCCM
L2
SRAM
L3 Ext. DRAM
Grey scale
conversion
Normali-
zation
Non-max
suppression
Storage of
scaled images
200 MB/s 80 MB/s
400 MHz
400 MHz

Power, Gate Count Comparisons (28 nm)
640 × 480 pixels, at 25 FPS
0
200
400
600
800
1000
1200
1400
Config #2 Config #3 Config #4
ASIP gates (K)
ARC gates (K)
Gates (K)
2 3 4
0.0
20.0
40.0
60.0
80.0
100.0
120.0
140.0
ASIP power (mW)
ARC power (mW)
Power (mW)
2 3 4
0
1
2
3
4
5
6
ASIP design and S/W
ARC S/W
2 3 4
Effort (person-months)
HAPS FPGA-based
demo platform
Note: Gates and power for processors
and local memory

• 1 ARC HS, 4 ASIPs, AXI interconnect, private SRAM, L2 SRAM
• Fixed point version of HOG derived from OpenCV
• 25 frames/second at 400 MHz (ARC and ASIPs)
• TSMC HPM process, 28nm
• Gate count (at 400 MHz): 471K gates
• 303K gates for ASIPs, 168K gates for ARC HS 36
• Power consumption: 60 mW
• Prototype running on HAPS board (ASIPs)
• 4 frames/second at 70 MHz
26
Final Results for Demonstrator Platform
Demonstration available at our booth
4

Lessons Learned
Subsystem
Controller
HS
Emb. Vision
Accelerators
ASIP2
ASIP1’
ASIP3
1X 100X
P
A
R
A
L
L
E
L
I
S
M
1’) Rescaling + Gradient
2) Histogram
3) Normalization
4) SVM
10X
Data
Level
Parallelism
Talk
Level
Parallelism
1) Greyscale
2) Non-max suppr.
3) Display
4) Control, O/S ASIP4
Sequential
Tasks
4

Lessons Learned
Subsystem
Controller
HS
Emb. Vision
Accelerators
ASIP2
ASIP1’
ASIP3
1X 60X~80X
P
A
R
A
L
L
E
L
I
S
M
Area Efficiency
Data
Level
Parallelism
Talk
Level
Parallelism
Combined:
471K gates,
60 mW @ 28nm
ASIP4
Sequential
Tasks
400 MHz
303K gates, 58 mW
(Logic = 12 mW
SRAM = 46 mW)
4
20% utilization
1.6 GHz: 473K gates, 37 uW/MHZ
400 MHz: 168K gates, 18 uW/MHz

Accelerator
C API
Data
Level
Parallelism
Talk
Level
Parallelism
Sequential
Tasks
Embedded Vision Platform Directions
& Wish List
Subsystem
Controller
HS
1X 100X
Pre-processing:
- Filtering
- Color conversion
- Image scaling
- Feature extraction
and matching
- Segmentation
High-level processing:
- Body part detection
- Multi-object tracking
- Post-processing
- Command
interpretation
Close
coupling
Vision
Extn.
SIMD
(64 bit)
OpenCV
MQX O/S
10X
Emb. Vision
Accelerators
ASIP
ASIP
ASIP
P
A
R
A
L
L
E
L
I
S
M

• Embedded vision applications combine complex algorithms
and high data rates with a need for low power
• Need to trade-off Flexibility vs. Power-Perf-Area
• Flexibility via High-performance ARC HS core
• Ability to trade-off power vs. performance
• Scaling to multi-core, specialization and SIMD usage
• Highest PPA via ASIPs
• Performance gains and power efficiency due to tailored
instruction sets and dedicated memory architecture
• While fully programmable, gains are application specific
Conclusions

Backup slides

Design flow for the Vision Sub System
ARC
HS
DW
AXI interco
DesignWare
DMA
DesignWare
DDR
ARChitect
ASIP
Processor
Designer
Core
Assembler
ASIP
descriptionASIP ISA
description
Ref Sub
System
ASIP
Synthesis +
P&R tools
Core
Consultant
SubSys
settings
ARC
settings
coreKitTool
Core
Builder
Core
Builder
User
config
VCS
DVE MDB PDBG
Legend :
HAPS

Synopsys’ ASIP Design Tool Flow
Processor
Description
Language
Optimizing
Compiler
Assembler Linker
Instruction-Set
Simulator
Debugger Profiler
RTL Generator
RTL Sim &
FPGA
RTL
Synthesis
Full-featured SDK with graphical debugger
Compiler supports processor specific data-
types and operators
Advanced optimizations allow C
programmers to easily tap into architectural
efficiencies
Fast retargeting to evaluate
incremental processor
architecture changes quickly.
High level language to quickly capture ISA
Tight control of architecture (RTL-level)
Fast simulation technology
Easy integration into System C virtual platforms
Multicore and on-chip debugging
Smooth integration with
RTL implementation and
verification flows

Architectural Optimization Space
ASIP architectural optimization space
Parallelism Specialization
Instruction-
level
parallelism
Data-
level
parallelism
Task-
level
parallelism
Orthogonal
instruction
set (VLIW)
Encoded
instruction
set
Vector
processing
(SIMD)
Multi-
core
App.-specific
data types
App.-specific
instructions
Connectivity & storage
matching application’s
data-flow
App.-spec.
data
processing
App.-spec.
memory
addressing
App.-spec.
control
processing
Distributed regs,
sub-ranges
Multiple mem’s,
sub-ranges
Jumps, subroutines,
interrupts, HW do-loops,
residual control, predication…
Direct, indirect, post-
modification, indexed,
stack indirect…
Any exotic
operator
Integer, fractional,
floating-point, bits,
complex, vector…
Single or
multi-cycle
Relative or absolute, address
range, delay slots…
Pipeline
Synopsys ASIP tools …
• Support a wide range of ASIP
architectures
• Support RTL accelerator tricks for
highest PPA efficiency
• Enable ASIP optimization through
architectural exploration
Multi-
threading

Combining Flexibility and Low-power in Embedded Vision Systems

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (13)

Similar to Combining Flexibility and Low-power in Embedded Vision Systems

Similar to Combining Flexibility and Low-power in Embedded Vision Systems (20)

More from Edge AI and Vision Alliance

More from Edge AI and Vision Alliance (20)

Recently uploaded

Recently uploaded (20)

Combining Flexibility and Low-power in Embedded Vision Systems