Fisheye/Omnidirectional View in Autonomous Driving IV

Fisheye/Omnidirectional View in
Autonomous Driving IV
Yu Huang

Outline
• FisheyeMultiNet: Real-time Multi-task Learning Architecture for
Surround-view Automated Parking System
• Generalized Object Detection on Fisheye Cameras for Autonomous
Driving: Dataset, Representations and Baseline
• SynWoodScape: Synthetic Surround-view Fisheye Camera Dataset for
Autonomous Driving
• Feasible Self-Calibration of Larger Field-of-View (FOV) Camera Sensors
for the ADAS

FisheyeMultiNet: Real-time Multi-task Learning Architecture
for Surround-view Automated Parking System
• Automated Parking is a low speed maneuvering scenario which is quite
unstructured and complex, requiring full 360° near-field sensing around the
vehicle.
• In this paper, discuss the design and implementation of an automated parking
system from the perspective of camera based deep learning algorithms.
• provide a holistic overview of an industrial system covering the embedded
system, use cases and the deep learning architecture.
• demonstrate a real-time multi-task deep learning network called
FisheyeMultiNet, which detects all the necessary objects for parking on a low-
power embedded system.
• FisheyeMultiNet runs at 15 fps for 4 cameras and it has three tasks namely object
detection, semantic segmentation and soiling detection.
• release a partial dataset of 5,000 images containing semantic segmentation and
bounding box detection ground truth via WoodScape project.

Classification of Parking scenarios - (a) Parallel Backward Parking (b) Perpendicular Backward Parking (c)
Perpendicular Forward Parking (d) Ambiguous Parking and (e) Fishbone Parking with roadmarkings.

Illustration of FisheyeMultiNet architecture comprising of object detection, semantic segmentation and soiling detection tasks.

Generalized Object Detection on Fisheye Cameras for
Autonomous Driving: Dataset, Representations and Baseline
• Object detection is a comprehensively studied problem in autonomous driving.
• However, it has been relatively less explored in the case of fisheye cameras.
• The standard bounding box fails in fisheye cameras due to the strong radial distortion,
particularly in the image’s periphery.
• explore better representations like oriented bounding box, ellipse, and generic polygon
for object detection in fisheye images in this work.
• use the IoU metric to compare these representations using accurate instance
segmentation ground truth.
• design a novel curved bounding box model that has optimal properties for fisheye
distortion models.
• also design a curvature adaptive perimeter sampling method for obtaining polygon
vertices, improving relative mAP score by 4.9% compared to uniform sampling.
• Overall, the proposed polygon model improves mIoU relative accuracy by 40.3%.
• The dataset comprising of 10,000 images along with ground truth will be made public.

Left: Illustration of fisheye distortion of projection of an open cube. A 4th-degree polynomial model radial
distortion. can visually notice that box matures to a curved box. Right: propose the Curved Bounding Box
using a circle with an arbitrary center and radius, as illustrated. It captures the radial distortion and obtains a
better footpoint. The center of the circle can be equivalently reparameterized using the object center (xˆ, yˆ).

Generic Polygon Representations. Left: Uniform angular sampling where the intersection of the polygon with the
radial line is represented by one parameter per point (r). Middle: Uniform contour sampling using L2 distance. It can
be parameterized in polar co-ordinates using 3 parameters (r, θ, α). α denotes the number of polygon vertices within
the sector, and it may be used to simplify the training. Alternatively, 2 parameters (x,y) can be used, as shown in the
figure on the right. Right: Variable step contour sampling. It is shown that the straight line in the bottom has less
number of points than curved points such as the wheel. This representation allows to maximize the utilization of
vertices according to local curvature.

FisheyeYOLO is an extension of YOLOv3 which
can output different output representation

SynWoodScape: Synthetic Surround-view Fisheye
Camera Dataset for Autonomous Driving
• Four fisheye cameras with a 190° field of view cover the 360° around the vehicle.
• Due to its high radial distortion, the standard algorithms do not extend easily.
• In this work, release a synthetic version of the surround-view dataset, covering many of its
weaknesses and extending it.
• Firstly, it is not possible to obtain ground truth for pixel-wise optical flow and depth.
• Secondly, WoodScape did not have all four cameras simultaneously in order to sample diverse
frames.
• However, this means that multi-camera algorithms cannot be designed, which is enabled in the
new dataset.
• implemented surround-view fisheye geometric projections in CARLA Simulator matching
WoodScape’s configuration and created SynWoodScape.
• release 80k images with annotations for 10+ tasks.
• also release the baseline code and supporting scripts.

SynWoodScape: Synthetic Surround-view Fisheye Camera Dataset for Autonomous Driving

Overview of Surround View cameras based multi-task
visual perception framework. The distance estimation task
(blue block) makes use of semantic guidance and dynamic
object masking from semantic/motion estimation (green
and blue haze block) and camera-geometry adaptive
convolutions (orange block). Additionally, guide the
detection decoder features (gray block) with the semantic
features. The encoder block (shown in the same color) is
common for all the tasks. The framework consists of
processing blocks to train the self-supervised distance
estimation (blue blocks) and semantic segmentation
(green blocks), motion segmentation (blue haze blocks),
and polygon-based fisheye object detection (gray blocks).
obtain Surround View geometric information by post-
processing the predicted distance maps in 3D space
(perano block). The camera tensor Ct (orange block) helps
OmniDet yield distance maps on multiple camera-
viewpoints and make the network camera independent.

Feasible Self-Calibration of Larger Field-of-
View (FOV) Camera Sensors for ADAS
• This paper proposes a self-calibration method that can be applied for multiple larger
field-of-view (FOV) camera models on ADAS.
• Firstly, perform steps such as edge detection, length thresholding, and edge grouping for
the segregation of robust line candidates from the pool of initial distortion line segments.
• A straightness cost constraint with a cross-entropy loss was imposed on the selected line
candidates, thereby exploiting that loss to optimize the lens-distortion parameters using
the Levenberg–Marquardt (LM) optimization approach.
• The best-fit distortion parameters are used for the undistortion of an image frame,
thereby employing various high-end vision-based tasks on the distortion-rectified frame.
• investigation on experimental approaches such as parameter sharing between multiple
camera systems and model-specific empirical γ-residual rectification factor.
• The quantitative comparisons between the proposed method and traditional OpenCV
method on KITTI dataset with synthetically generated distortion ranges.
• a pragmatic approach of qualitative analysis has been conducted through streamlining
high-end vision-based tasks such as object detection, localization, and mapping, and
auto-parking on undistorted frames.

Proposed Pipeline on ADAS workbench (a) ADAS Platform: Camera sensors setup and image acquisition

Proposed Pipeline on ADAS workbench (b) Proposed method with block schematics.

Structural anomaly induced into a
scene due to heavy lens distortion
caused by wide-angle cameras with
field-of-view 120◦ < FOV < 140◦.

Lens Projection Models: (a) Standard Camera Pinhole Projection Model. (b) Larger FOV Lens Orthogonal Projection Model.

Proposed Self-calibration design

Pre-processing of line candidates and
Estimation of Straightness constraint.

Schematic of distortion parameter
estimation using LM-optimization in normal
mode and parameter sharing mode.

Severe distortion cases rectified
using several approaches
[28,29], proposed method with
and without empirical γ-hyper
parameter.

Data acquisition scenarios using
various camera models.

Auto-parking scenario on rear fisheye camera: Real-time visual SLAM pipeline on lens distortion rectified sensor data.

Fisheye/Omnidirectional View in Autonomous Driving IV

Fisheye/Omnidirectional View in Autonomous Driving IV

More Related Content

What's hot

Similar to Fisheye/Omnidirectional View in Autonomous Driving IV

More from Yu Huang

Recently uploaded

Fisheye/Omnidirectional View in Autonomous Driving IV