For the full video of this presentation, please visit: http://www.embedded-vision.com/platinum-members/nxp/embedded-vision-training/videos/pages/may-2016-embedded-vision-summit For more information about embedded vision, please visit: http://www.embedded-vision.com Tom Wilson, ADAS Product Line Manager at NXP Semiconductors, presents the "Sensing Technologies for the Autonomous Vehicle" tutorial at the May 2016 Embedded Vision Summit. Autonomous vehicles will necessarily utilize a range of sensing technologies to see and react to their surroundings. We are witnessing dramatic advances not just for embedded vision, but also in complementary technologies like radar and LiDAR. Each of these sensing technologies provides unique capabilities for giving a vehicle a complete view of its surroundings. This presentation compares vision-based sensing with complementary sensing technologies, explores key trends in sensors for autonomous vehicles, and analyses challenges and opportunities in fusing the output of multiple sensor technologies to enable robust perception and mapping for autonomous vehicles.

1. Copyright © 2016 2016 NXP Semiconductor 1
Tom Wilson
May 3, 2016
Sensing Technologies for the
Autonomous Vehicle

2. Copyright © 2016 2016 NXP Semiconductor 2
Driver Assistance
Partial Automation
Semi autonomous
Fully autonomous
Eyes off
ADAS: From Safety to Autonomous
ADAS: >$11B in 2016 growing to
>$130B in 2026 CAGR ~29%
ABI Research, 2016
How?
“All our knowledge begins with the senses” Immanuel Kant

3. Copyright © 2016 2016 NXP Semiconductor 3
Autonomous
Requirement
Sensing Technology Comparison
Camera Radar LiDAR
Object Detection M H H H
Classification H M - H
Density of Raw Data H M L H
Velocity Measurement - H - H
Lane Detection H - - H
Traffic Sign Recognition H - - H
Range of Sensor M (150m) H (250m) M (100m) Full range
Rain, Fog, Snow L H L H
Night - H H H
Sensor size Small to Med Small Med Mix
Cost H (ADAS) L H Mix
Rating: H = High, M=Medium, L = Low

4. Copyright © 2016 2016 NXP Semiconductor 4
Car’s Eye View: Vision
t=0.1st=0s
Optical flow for Motion Estimation Histogram of Oriented
Gradients: Pedestrians

5. Copyright © 2016 2016 NXP Semiconductor 5
Car’s Eye View: Radar
Pedestrian moving radially
(towards or away)
Pedestrian moving laterally
Velocity is a Radar “Feature” for motion estimation
Doppler also used for classification

6. Copyright © 2016 2016 NXP Semiconductor 6
Car’s Eye View: LiDAR
360° Scanning LiDAR
Image courtesy Velodyne
Fixed-Beam LiDAR
Image courtesy Leddartech
Scan compared to map to subtract
Stationary objects
Simple detection and ranging
No classification

7. Copyright © 2016 2016 NXP Semiconductor 7
Proliferation of Sensors
Assist
Co-Pilot
Automated
Vision Radar LiDAR

8. Copyright © 2016 2016 NXP Semiconductor 8
Car’s Eye View: Vision
Fwd Facing Multi-Function ADAS Camera
LDW, TSR, Pedestrian Detection, FCW, IHC
Surround-View Object Detection, Classification
180º FOV
Rear-View Camera
Scene-view,
Object Detection

9. Copyright © 2016 2016 NXP Semiconductor 9
Car’s Eye View: Radar
Long Range/Mid-Range
Forward Facing
AEB, ACC, FCW
Mid-Range/Short-Range
Multi-mode “Corner Radar”
Long Range/Mid-Range
Rear Collision Avoidance
Complementing AEB)

10. Copyright © 2016 2016 NXP Semiconductor 10
Car’s Eye View: LiDAR (Fixed Beam)
Mid-Range
AEB, FCW
Cross traffic
Blind spot
Rear collision
avoidance

11. Copyright © 2016 2016 NXP Semiconductor 11
The Full Sensor Suite for Autonomous

12. Copyright © 2016 2016 NXP Semiconductor 12
Game of “King of the Hill”
Market Acceptance
Detection Capability
Cost
Low Cost
Low Det’n
High Det’n
High Cost
Low Cost
Med. Det’n
Med Cost
Med Det’n
Med Cost
Med Det’n
Med Cost
High Det’n
Low Cost
High Det’n

13. Copyright © 2016 2016 NXP Semiconductor 13
Game of “King of the Hill”
Detection Capability
Cost
Market Acceptance
High Detection
Decreasing Cost
Low Cost
Increasing Detection
Low Cost
Low Det’n
High Det’n
High Cost
Low Cost
High Det’n
Market winners move
“up the hill” Market losers move
“down the hill”

14. Copyright © 2016 2016 NXP Semiconductor 14
Climbing the Autonomous Vehicle Hill
Detection Capability
Cost
Market Acceptance
Vision: FF-DAS
Multi-Function
Scanning LiDAR
Radar
Vision: Park-Assist
Fixed-Beam LiDAR

15. Copyright © 2016 2016 NXP Semiconductor 15
Sensor Network for Fusion
Fusion
ECU
FF ADAS
Camera
LiDAR
Surround-View Cameras
Side-facing LiDAR
Design Challenge: Partitioning of
processing and interconnect selection

16. Copyright © 2016 2016 NXP Semiconductor 16
• Each level assesses
associations from prior
level
• Bandwidth from each level
to the next depends on
sensor type
• Partitioning of processing
will vary by Sensor
Fusion Processing and Partitioning
Level 0: Feature Assessment
Level 1: Object Assessment
Level 2: Situation Assessment
Level 4: Process Refinement
Level 3: Impact Assessment
Resources
Signals/Features
Measurements
Objects
Situations
Situations/Plans
Situations/PlansPlans
Plans
Situations
Objects
Signals/Features

17. Copyright © 2016 2016 NXP Semiconductor 17
• ADAS cameras may process
objects or even situations
• Surround-view cameras
typically send raw data
• Radar and LiDar typically
processing to features (L0),
• Radar extends to objects (L1)
• Ideally process all levels in
ECU, not always possible
System Partitioning Example
ADAS
Camera
Radar LiDAR
Levels 3 & 4
Level 2
Level 1
Level 0
Fusion ECU
Surround
View
Camera

18. Copyright © 2016 2016 NXP Semiconductor 18
• No single sensing technology will provide complete information coverage
• Cameras, Radar, LiDAR (and ultrasound!) will all be utilized and fusion
processing elaborates knowledge from the full range of sensing data
• Key challenge: how to partition fusion processing within the constraints of
• Bandwidth / processing capability
• Getting the most reliable impact assessments
• NXP enables autonomous driving with front-end sensors, radar, vision and LiDAR
processing, interconnect technology, fusion processing and vehicle (V2X) comms
Summary