For the full video of this presentation, please visit: https://www.edge-ai-vision.com/2021/02/using-an-isp-for-real-time-data-augmentation-a-presentation-from-pony-ai/ Timofey Uvarov, Camera System Lead at Pony.AI, presents the “Using an ISP for Real-time Data Augmentation” tutorial at the September 2020 Embedded Vision Summit. Image signal processors (ISPs) are tasked with processing raw pixels delivered by image sensors in order to optimize the quality of images. In computer vision applications, much attention is focused on tuning the ISP, both to obtain good training data and to optimize image quality in deployed systems when faced with widely varying, dynamic imaging conditions (e.g., changes in lighting). In this presentation, Uvarov describes a data-driven solution to these challenges developed in a collaboration between Pony.AI and On Semiconductor. His company’s approach manipulates the ISP configuration during training data collection in order to perform real-time data augmentation. By training deep neural networks (DNNs) with the resulting augmented data, Pony.AI is able to create DNNs that are robust to variations in imaging conditions and ISP settings.

1. © 2020 Pony.AI
Using an ISP for Real-time Data
Augmentation
Timofey Uvarov
Pony.ai
Sept 2020

2. © 2020 Pony.AI
Contents
2
Introduction to Pony.ai
Exposure in photography and computer vision
Experiments with exposure control
Conclusions and problem formulation
Proposed solution with automotive ISP
Rererences

3. © 2020 Pony.AI
Pony.ai
3

4. © 2020 Pony.AI 4
Pony.ai – global autonomous technology leader
Existing sites
New launches in 2020
F R E M O N T
coverage ~50 km2
coverage ~100 km2
coverage ~200 km2
B E I J I N G
G U A N G Z H O U
Actively testing in two urban
locations under T3 permit
Robotaxi pilot serving the
general public with Hyundai
I R V I N E
coverage ~100 km2
R&D hub with commuter pilot
with City of Fremont
Robotaxi pilot launched Q4
2018
To launch large-scale Robotaxi
fleet operations
H I G H L I G H T S
100+ vehicles
500km2+ operational coverage
3.0+ million autonomous kilometers driven
S H A N G H A I
coverage ~30 km2

5. © 2020 Pony.AI 5
Core technology: vehicle-agnostic “virtual driver”
In-depth partnerships with
camera industry leaders to
build next generation camera
for machine vision
Vehicle
AD Software Stack
Planning
Perception
Prediction
Control
Sensor
fusion
Sensors & Hardware
HD Map Integration &
Localization
LiDAR
Radar
Data &
Infrastructure
Vehicle
Interface
Partner
Product layer
Compute
End Customer Interface
GNSS/ IMU
Select Vehicle Platforms
Developed in-house by Pony.ai
Developed by partner selected by Pony.ai
Camera

6. © 2020 Pony.AI
Collaboration between Pony.ai and ON Semiconductor
About ON Semiconductor
• Global semiconductor company with ~35,000 employees
• Broad automotive portfolio, segment contributing 33% of total revenue
• #1 automotive image sensor provider globally
Scope of Collaboration
• Collaborate on developing next-generation image sensing and processing technologies for machine vision and AI
• Today’s presentation covers one of the collaboration topics “Using an ISP for real-time data augmentation”
Active Safety
Solutions
In-Vehicle
Networking
ASIL Compliant
Power Management
ONSemi Products Enabling AD

7. © 2020 Pony.AI
Exposure
7

8. © 2020 Pony.AI
Perceptual CV pipeline architecture
8
Scen
e
Len
s
ISP CNN/DN
N
Light Light RAW
Ctrl
RGB
YUV
Prediction
Sensor
(CIS)

9. © 2020 Pony.AI
Exposure in photography
9
Exposure is the amount of light per unit area reaching a frame
of photographic film or the surface of an electronic image
sensor
Exposure is determined by shutter speed, lens aperture, and
scene luminance.
Exposure control is fundamental in photography.
"Correct" exposure (in photography) may be defined as an
exposure that achieves the effect the photographer intended.
R2
R1

10. © 2020 Pony.AI
Exposure in CV and automotive
10
Lens aperture is fixed (no moving parts)
ISP (image signal processor) controls:
- Exposure time ET (integration time/shutter speed)
- Gain (ISO / film sensitivity)
based on scene light metering and history (statistics)
Exposure time is first priority. Gain compensates for dark image with short ET
R1

11. © 2020 Pony.AI
Exposure in automotive and CV
11
Exp time 1 (min)Exp time 2 Exp time .. Exp time N (max)
Gain 1 Gain .. Gain Max
ISP setting 1
(min exp)
Exposure
ISP setting 2 ISP setting .. ISP setting .. ISP setting N’
(max exp)
Exposure time is first priority. Gain compensates for dark image with short ET
ISP usually has different setings for highest and lowest exposure points on
the spectrum above (sometimes in between)

12. © 2020 Pony.AI
Exposure
12
Quantization
controls image sensor
Noise/SNR
Affect Dynamic Range (DR), tone reproduction
Affects sharpness, motion blur,
Responsible for under/over saturation
Drives ISP setting
Color accuracy and reproduction
Exposure control is crucial for object detection and safety of AV

13. © 2020 Pony.AI
Lab experiment 1:
Automatic Exposure
13

14. © 2020 Pony.AI
Setup
14
Camera:
- Sensor in single
exposure mode
- ISP: OnSemi AP200
- Lens: 180D ImmerVision
Light:
- Cool-lux LED
(1/25/50/75 strength)
- Imatest Light-box
Scene:
- Imatest Contrast
resolution chart
- Small car models (4x)
- Human Vision test chart
- Moving object

15. © 2020 Pony.AI
Test 1: AE + changed light
15
Within saturation Within saturation Within saturation Over - saturation
Strong light (~500lux)
Weak light (<100lux)

16. © 2020 Pony.AI
Test 1: Automatic Exposure / Changed light
16
Lost visibility
Lost visibility

17. © 2020 Pony.AI
Test 1: AE / Changed light
17
Same
Image
Can’t
Be
Properly
exposed
Too dark

18. © 2020 Pony.AI
Lab Experiment 2:
Manual Exposure
18

19. © 2020 Pony.AI
2111
Manual Exposure : Full Scene
19

20. © 2020 Pony.AI
Manual Exposure: Moving object
20

21. © 2020 Pony.AI
Manual Exposure: Human Vision Chart
21

22. © 2020 Pony.AI
Imatest Contrast Resolution
22
Measures the visibility of low contrast objects in
larger fields over a wide range of brightness
The range of tones a camera responds to with good
contrast and Signal-to-Noise Ratio (SNR))
is a function of the sensor and lens (and to some
extent, the signal processing)
R3

23. © 2020 Pony.AI
Manual Exposure: Imatest Contrast Resolution
23

24. © 2020 Pony.AI
Manual Exposure: Shadows
24

25. © 2020 Pony.AI
Manual Exposure: Highlights
25

26. © 2020 Pony.AI
Field experiment, conclusions
26

27. © 2020 Pony.AI
Exposure time difference
27
B: Short exposure time: sharp image,
high noise (high gain)
A: Long exposure time: motion blur, less
noise (low gain)
mAP depends:
- ISP settings
- Training / Test set / Object sizes
- Model architecture
Chaning one of the the above can change
detection accuracy?
Common question for ISP tuning: A or B?
Answer: A and B.

28. © 2020 Pony.AI
Conclusions
Its is impossible to choose the exposure parameters which
will be optimal for all objects in the scene.
Same object appears differently depending on light,
exposure control and ISP parameters.
Training set never has all possible variations of lighting
conditions for any object.
.

29. © 2020 Pony.AI
Augmentation
29

30. © 2020 Pony.AI
Data improvement
R7
4

31. © 2020 Pony.AI
Augmentation (Auto ML)
R7
Data is the most critical piece of AI applications.
Not having enough labeled data often leads to
overfitting, which means the model will not be
able to generalize to unseen examples.
This can be mitigated by data augmentation,
which effectively increases the amount and
diversity of data seen by the network. It is done by
artificially producing new data by applying
transformations on an original dataset such as
rotation, cropping, occlusion, etc.

32. © 2020 Pony.AI
Learning Data Augmentation Strategies for Object Detection
Resnet-101/COCO (Small,Med, Large)
R8
Learned augmentation policy
Systematically improves object
detection
Learned data augmentation
improves model regularization

33. © 2020 Pony.AI
Augmented train
33
- Original image sequence E
- Augmented sequence: E – anchor frames,
- k augmented frames between each two anchors

34. © 2020 Pony.AI
Augmented train for training networks
34
- (En)’ – exposure time of En
- (En.i)’ = (En)’ + A*sin(i*2pi/(k+1)),
- A – bracketing / augmentation strength
If (k == 1), (En.i)’ = rand(A)*(-1)^n

35. © 2020 Pony.AI
Example of augmented exposure trains
35

36. © 2020 Pony.AI
White Balance (WB) augmentation
White balance (WB) is the process of
removing unrealistic color casts, so that
objects which appear white in person are
rendered white in your photo
Proper camera white balance has to take
into account the "color temperature" of a
light source, which refers to the relative
warmth or coolness of white light
Raw
Image
STAT
S
Color TEMP.
Guess
R-gain
B-gain
R5
R6

37. © 2020 Pony.AI
WB augmentation (Auto-Exposure)
37

38. © 2020 Pony.AI
WB + Exposure augmentation
38

39. © 2020 Pony.AI
Augmented demo
39
Prototype:
- Single exposure mode CIS
- ISP: OnSemi AP0200
Target Platform:
OnSemi AP0300
CEVA XM-4 dsp

40. © 2020 Pony.AI
ISP augmentation vs. Post processing
40
ISP augmentation Post ISP Pre-processing
Baseline
Simulation
ISP parameters
Data
ISP parameters
Compute
Augments capturing of the object on physical
level (different physical level of noise, readout,
motion blur)
Physically accurated simulation of oject
representation at different lighting condition
RAW 12-16bit (pre-ISP)
Augmented based on exposure change in
realistic
Real-time, hw-accelerated
Only changes image data after ISP
processing
Trying to mimic contrast, gamma, etc on
existing processed image.
8 bit RGB/YUB (post ISP)
ISP parameters are not change/augmented
Uses software, GPU
ISP augmentation for imaging, post processing for geometry

41. © 2020 Pony.AI
Resource
1. Exposure in photography (Wikipedia)
https://en.wikipedia.org/wiki/Exposure_(photography)
2.The Exposure Triangle: Making Sense of Aperture, Shutter Speed, and ISO (Petapixel)
https://petapixel.com/2017/03/25/exposure-triangle-making-sense-aperture-shutter-speed-iso
3. Boosting sensitivity (Vision research, Phantom ):
https://phantomhighspeed-service.force.com/servlet/servlet.FileDownload?file=00P1N00000dSuGDUA0
4. Contrast Resolution chart and analysis (Imatest)
https://www.imatest.com/docs/contrast-resolution/
5. Color Temperature (Wikipedia)
https://en.wikipedia.org/wiki/Color_temperature
6. Understanding White Balance (Cambridgecolor)
https://www.cambridgeincolour.com/tutorials/white-balance.htm
7. AutoML for Data Augmentation (Baris Ozmen)
https://blog.insightdatascience.com/automl-for-data-augmentation-e87cf692c366
8. Learning Data Augmentation Strategies for Object Detection (Google Research):
https://arxiv.org/abs/1906.11172
41

42. © 2020 Pony.AI
Thank you!
42