For the full video of this presentation, please visit: https://www.edge-ai-vision.com/2023/07/optimized-image-processing-for-automotive-image-sensors-with-novel-color-filter-arrays-a-presentation-from-nextchip/ Young-Jun Yoo, Vice President of the Automotive Business and Operations Unit at Nextchip, presents the “Optimized Image Processing for Automotive Image Sensors with Novel Color Filter Arrays” tutorial at the May 2023 Embedded Vision Summit. Traditionally, image sensors have been optimized to produce images that look natural to humans. For images consumed by algorithms, what matters is capturing the most information. We can achieve this via higher resolution, but higher resolution means lower sensitivity. To increase resolution and maintain high sensitivity, color information can be sacrificed—but in automotive applications, color is critical. In response, suppliers offer image sensors that capture color information using novel color filter arrays (CFAs). Instead of the traditional RGGB array, these sensors use patterns like red-clear-clear-green (RCCG). These approaches yield good results for perception algorithms, but what about cases where images are used by both algorithms and humans? Can we reconstruct a natural-looking image from an image sensor using a non-standard CFA? In this talk, Yoo explores novel CFAs and introduces Nextchip’s vision processor, which supports reconstruction of natural-looking images from image sensors with novel CFAs, including RGB-IR sensors.

1. Optimized Image Processing
for Automotive Image Sensors
with Novel Color Filter Arrays
Young-Jun Yoo
VP
Nextchip Co., Ltd.

2. Today’s contents
2
© 2023 Nextchip Co.,Ltd.
• Nextchip: Vision Professional
• Challenge for Both Human & Machine Vision
• Nextchip’s ISP Topology
• Test Results of Nextchip’s Image Processing for Each CFA
• Sophisticated methodology for design and tuning

3. 3
© 2023 Nextchip Co.,Ltd.
Nextchip: Vision Professional

4. Nextchip overview
World-class ISP (ASIC/IP/Tuning)
Automotive reliability
• 420million shipping units
• 51+ MP car models
Listed on the KOSDAQ
Market Proven Technology
• Went public in July 2022
Advanced ADAS technology
Customer support
• Know-how in tuning against various customers’ criteria
• AEC-Q100 Gr.2/CMMI Lv.-3/ISO26262/A-Spice process
• Developing image signal processing tech. for 26 years
• Open architecture with various imagers
• Edge processor /w hard-wired ML algorithms
• ADAS SoCs with CNN capability
• Tuning capability human & machine vision
© 2023 Nextchip Co.,Ltd. 4

5. 5
Challenge for both
Human & Machine Vision
© 2023 Nextchip Co.,Ltd.

6. 6
Key factor of machine vision
<Ref. Basler “High-Sensitivity Image Processing Cameras”>
• High sensitivity is the key factor of machine vision for achieving a quality video image with low image noise even
in poor lighting conditions.
• The higher the signal-to-noise ratio, the better the object can be detected.
• Sensitivitiy is more important than color in machine vision.
<Ref. gla.ac.uk “Sensitivity and noise”>

7. How to get higher sensitivity
7
Diverse bayer pattern with pixels
Rear Blind Spot
Guidance
Rear Camera
Fully Automatic Parking
Function
MOD Camera Motion
Object Detection
LDWS(Lane Departure
Warning System)
Night Vision Function
Stereo/Lidar Camera
System
Front Camera
Adaptive Camera
Drowsy Driving
Monitoring
R G B
G
R G B
I
R
High
sensitivity;
Moderate
color
Good
sensitivity;
Moderate
color
R Y C
y
Y
R C C B
High
sensitivity;
Less color
default
• Special CFAs have popped in a variety of
scenarios depending on how the camera
is used.
• All about the color & sensitivity for
human & machine vision.
• HOWEVER, newly proposed CFAs have
an issue in color reproduction.
© 2023 Nextchip Co.,Ltd.
< Ref1. MCNEX official website>
Moderate
sensitivity;
Strong
color

8. This is Machine’s view, but are you (human) ok with that?
© 2023 Nextchip Co.,Ltd.
What is the difference between
human vision & machine vision?

9. Issues when color reproduction fails
9
• Hard to distinguish red light. And, it is directly linked to malfunction in autonomous driving.
• Even for sensing application, color reproduction is required at the very least.
© 2023 Nextchip Co.,Ltd.
< Ref3. Non-RGB color filter options and traffic signal detection capabilities>

10. 10
Nextchip’s ISP Topology
Attempts to get sensitivity and color at the same time
© 2023 Nextchip Co.,Ltd.

11. What is special?
11
R G
G B
R C
C G
R Y
Y Cy
Front Sensing
With Higher Resolution
General Purpose
R G
B IR
In-cabin sensing
for Occupant Monitoring
• While maintaining Red,
R G gets more color than RGGB.
R Cy subtracts varying degrees of red, green, and blue from white light to produce a full spectrum of colors.
RGBIR uses IR to get higher sensitivity under the dark cabin environment.
• There is a trend to change color filter array from RGB to others.
It is based on applications like RGB-IR for in-cabin camera system, RCCG and RYYCy for exterior cameras in
automotive market.
© 2023 Nextchip Co.,Ltd.
<Ref4. Image Sensor World>

12. The key is
how to fill the lack of pixel information
12
IR
G
IR
G
G
B
G
R
IR
G
G
R
IR
G
G
B
Color Reproduction Edge Enhancer Noise Reduction
G:8 B:2 R:2 IR:4
• Since several pixels had been converted to
another pixels compared to RGGB, the key is
how to handle and improve the color and edge
characteristics given the lack of pixel
information.
• With specialty in image signal processing
technology as the fundamental vision
technology of Nextchip, it provides the
differentiated image quality for embedded
systems.v
© 2023 Nextchip Co.,Ltd.
G
C
G
C
C
R
C
R
G
C
C
R
G
C
C
R
G:4 B:0 R:4 C:4
Cy
Y
Cy
Y
Y
R
Y
R
Cy
Y
Y
R
Cy
Y
Y
R
G:0 Y:8 B:0 R:4 Cy:4

13. In RGB-IR case
13
R G
B IR
In-cabin sensing
for Occupant Monitoring
OMS
Application
Dual Band
Pass Filter
Interpolation & RGB IR
separation
R = (R + IR) – IR = R
G = (G + IR) – IR = G
B = (B + IR) – IR = B
IR = IR
Day
Vision
Night
Vision
G B
R
I
R
© 2023 Nextchip Co.,Ltd.
• RGB-IR is the most complicated case because it needs to extract IR from each pixels.
• Dedicated processing block for extracting IR is required.

14. 14
Test Results of Nextchip’s
Image Processing for Each CFA
© 2023 Nextchip Co.,Ltd.

15. RGBIR image
15
RAW RGB
© 2023 Nextchip Co.,Ltd.
• Despite the lack of pixel information, the Nextchip ISP is able to reproduce color at the same level as RGGB.
• If you want both RAW and YUV, you can get 2 separate image streams.
• They can be used for both human vision and machine vision.

16. RCCG image
16
RAW RCCG
• Despite the lack of pixel information, the Nextchip ISP is able to reproduce color at the same level as RGGB.
• If you want both RAW and YUV, you can get 2 separate image streams.
• They can be used for both human vision and machine vision.
© 2023 Nextchip Co.,Ltd.

17. RCCG image
17
RAW RCCG
© 2023 Nextchip Co.,Ltd.
• Despite the lack of pixel information, the Nextchip ISP is able to reproduce color at the same level as RGGB.
• If you want both RAW and YUV, you can get 2 separate image streams.
• They can be used for both human vision and machine vision.

18. RYYCy image
18
RAW RYYCy
© 2023 Nextchip Co.,Ltd.
• Despite the lack of pixel information, the Nextchip ISP is able to reproduce color at the same level as RGGB.
• If you want both RAW and YUV, you can get 2 separate image streams.
• They can be used for both human vision and machine vision.

19. Sophisticated methodology
for design and tuning
19

20. Realtime processing through well partitioning
20
© 2023 Nextchip Co.,Ltd.
• Real-time constraint is one of the most common challenge found in many critical embedded applications,
namely image and video processing. However, software tools and general purpose microprocessor are not
suitable for deals with such as problems. The implementation method is based on Hardware/Software (HW/SW)
codesign approach.
• To achieve a partition that will give us the required performance within the overall system requirements (in
size, weight, power, cost, etc.)
• In addition to the advantage of easily modifying parameters to suit various environments, blocks that require
fast image processing speed are designed with hardware that uses as little calculation line memory as possible,
so they have the advantage of being processed in real time and displayed on the screen without delay.

21. Image analysis for machine vision
21
© 2023 Nextchip Co.,Ltd.
• Key ISP modules which impact CV performance are MTF and Low light sensibility by sensor side,
and Sharpening, HDR, Tone mapping and Color contrast by ISP side.
• ISP vendor should understand each algorithms and trys to find out ultimate setting values for better detection.
<Ref. “Overview and Empirical Analysis of ISP Parameter Tuning for Visual Perception in Autonomous Driving”>
Best for MV
Best for HV

22. 22
Takeaways
© 2023 Nextchip Co.,Ltd.

23. Market proven ISP technology
23
Well-partitioned flexible ISP core technology
2 dedicated processing topologies for human vision and machine vision
Capable of analyzing the image based on understanding of HV and MV
© 2023 Nextchip Co.,Ltd.
ISP is not a one function. It is a bunch of functions.

24. Resources
24
High-Sensitivity Image Processing Cameras
https://www.baslerweb.com/en/vision-campus/camera-technology/high-sensitivity-industrial-cameras/
Sensitivity and noise
https://radio.astro.gla.ac.uk/a2_oa/a2oa_sec4.pdf
MCNEX AUTOMOTIVE DRIVING- RELATED TECHNOLOGY
https://www.mcnex.com/en/business/020201.automotive-camera
Robot Bionic Vision Technologies: A Review
https://www.mdpi.com/2076-3417/12/16/7970
Non-RGB color filter options and traffic signal detection capabilities
https://library.imaging.org/ei/articles/34/16/AVM-215
Aptina Explains Clarity+ Technology, Reveals 1.1um Pixel Product
http://image-sensors-world.blogspot.com/2013/07/aptina-explains-clarity-technology.html
Design of Image Signal Processor greatly reduced chip area
by role sharing of hardware and software
https://koreascience.kr/article/JAKO201031559288071.pdf
Other images
www.nextchip.com/en/ & Nextchip’s internal reports
Please visit booth #109 and enjoy the latest image processing technologies.
© 2023 Nextchip Co.,Ltd.