Lane Detection Project Report in Udacity’s Self-Driving Car Nanodegree

1. Lane Detection
Fei Fei Zheng
December 2018
Abstract
Lane detection is an important component of intelligent transportation systems.
First, lane marks are extracted based on color information and the resultant binary
image with Canny line detection. Next, the perspective transformation and define the
Region of Interest(ROI) helps eliminate irrelevant elements in the image. The starting
points of lane lines can be roughly determined by calculating the histogram of the binary
image. Finally, the pixels of lane lines are detected by applying sliding windows from
detected starting points. The proposed algorithm can be used to detect lane lines on
both images and videos.
Keywords— Computer vision, lane detection
1 Introduction
Image
Frames
Image
Color
Extraction
Image
Color
Analysis
Lane Mark
Extraction
Sobel
Filter
Region
of
Interest
Lane Position
Determination
Perspective
Transform
Histogram
Lane Mark
Recognition
Sliding
Windows
Polynomial
Fit
Figure 1: Flow diagram of the algorithm
Detecting and localizing lanes from a road image is an important component of intelligent
transportation system applications. Intelligent vehicles are expected to be able to follow
the lane direction to ensure the safety of both passengers and pedestrians. The algorithm
is built from scratch. While I try different methods to get the optimal result, I learn how
those methods work and how to integrate all the separate pieces into one.
The proposed algorithm is illustrated in Fig.1. The algorithm assumes that the lane
lines on the road are marked with white or yellow. Therefore, we utilize color information
as the basis of our lane-detection algorithm. The lane mark extraction results in the binary
image. When extracting lane marks based on the result of Sobel and color filters, I define
the region of interest to eliminate the irrelevant elements on the image such as trees and
vehicles. In order to get ”birds-eye view”, I apply a perspective transform to rectify binary
image. The potential starting points of the lane marks are determined by finding peaks on
the histogram of the lower half of the rectified binary image. After getting the pixels of lane
1

2. marks with sliding windows moving upward from starting points, I can fit a polynomial line
to those points to find the direction of the lane. After all, I can pass back an overlay to the
original frame of the area between the curves to visualize the result.
2 Datasets
It’s a 4-hour road trip video from YouTube. The video is converted into thousands of frames.
I randomly select one thousand continuous frames and divided them into four groups. Each
frame is 1920 by 1080 pixels. The frames are used to test the accuracy of the algorithm.
3 Algorithm
3.1 Lane Mark Extraction
Given an image in RGB color space, it first converts to LAB color space Fig.2. It will apply
Sobel filter in both x-axis and y-axis to the L channel image, then use the combination
of color and gradient thresholds to obtain a magnitude-based and direction-based binary
image where the lane lines are clearly visible Fig.3.
0 250 500 750 1000 1250 1500 1750
0
200
400
600
800
1000
Original Image
0 250 500 750 1000 1250 1500 1750
0
200
400
600
800
1000
L Channel
0 250 500 750 1000 1250 1500 1750
0
200
400
600
800
1000
A Channel
0 250 500 750 1000 1250 1500 1750
0
200
400
600
800
1000
B Channel
Figure 2: Original Image and LAB color space
3.2 Lane Position Determination
Pick four points in a trapezoidal shape lying along the lines that would represent a rectangle
when looking down on the road from above. Apply perspective transform by using the
warpPerspective function in OpenCV to generate the rectified binary image with ”birds-
eye view”. The white pixels on the image potentially belong to the lane marks. Peak
detection function takes the lower half of the warped binary image, the highest points of
2

3. 0 250 500 750 1000 1250 1500 1750
0
200
400
600
800
1000
Binary Image with Sobel Filter and color threshold
0 250 500 750 1000 1250 1500 1750
0
200
400
600
800
1000
Binary Image with ROI mask
Figure 3: Binary Images
the accumulated values in each side are potential starting points to detect lane marks Fig.4.
When the car deviates from the center of the lane, if the histogram is simply divided into
left and right half, the peaks detection result will be incorrect. To solve this problem, an
assertion loop is implemented in the function. Assume the lane width is not less than 500
pixels compare to the image size, if the distance between two detected peak is less than the
lane width, it will adjust the center of histogram based on the location of current peaks and
re-detect the peaks.
0 250 500 750 1000 1250 1500 1750
0
200
400
600
800
1000
Warped Binary Image (Perspective Transformation)
0 250 500 750 1000 1250 1500 1750 2000
x axis
0
100
200
300
400
500
pixelcount
Histogram of Binary Warped Image
Left peak
Right peak
Figure 4: Peak Detection
3.3 Lane Mark Recognition
0 250 500 750 1000 1250 1500 1750
0
200
400
600
800
1000
Sliding Windows Lane Detection
0 250 500 750 1000 1250 1500 1750
0
200
400
600
800
1000
Lane Region
Figure 5: Peak Detection
3

4. The sliding window detection function begins tracing lanes in the binary images with
series of left and right ”windows” moving upwards, show in Fig.5. The search windows
start from the peaks detected in the histogram. If the pixels in a search window is beyond
the threshold value, the window will be re-centered to the mean position. When the lane
marked is dashed, it is possible no activated pixels in a search window. In this case, the
window will make a guess based on the other lane lines and a lane mark width. Since the
frames are continuous, the previous lane position can be used as a reference to improve the
efficiency of the sliding windows detection. After the window sliding search, a list of left
and right pixel locations are collected. Fit a quadratic equation to all the pixels by using
polyfit function in NumPy. Detected lane can overlay on the original image by using the
inverse matrix of the perspective transform to visualize the algorithm result Fig. 6.
0 250 500 750 1000 1250 1500 1750
0
200
400
600
800
1000
Original Image
0 250 500 750 1000 1250 1500 1750
0
200
400
600
800
1000
Original Image with Detected Lane Overlay
Figure 6: Lane Detection
4 Results
0 250 500 750 1000 1250 1500 1750
0
200
400
600
800
1000
Original Image
0 250 500 750 1000 1250 1500 1750
0
200
400
600
800
1000
Original Image with Detected Lane Overlay
0 250 500 750 1000 1250 1500 1750
0
200
400
600
800
1000
Original Image
0 250 500 750 1000 1250 1500 1750
0
200
400
600
800
1000
Original Image with Detected Lane Overlay
0 250 500 750 1000 1250 1500 1750
0
200
400
600
800
1000
Original Image
0 250 500 750 1000 1250 1500 1750
0
200
400
600
800
1000
Original Image with Detected Lane Overlay
0 250 500 750 1000 1250 1500 1750
0
200
400
600
800
1000
Original Image
0 250 500 750 1000 1250 1500 1750
0
200
400
600
800
1000
Original Image with Detected Lane Overlay
0 250 500 750 1000 1250 1500 1750
0
200
400
600
800
1000
Original Image
0 250 500 750 1000 1250 1500 1750
0
200
400
600
800
1000
Original Image with Detected Lane Overlay
0 250 500 750 1000 1250 1500 1750
0
200
400
600
800
1000
Original Image
0 250 500 750 1000 1250 1500 1750
0
200
400
600
800
1000
Original Image with Detected Lane Overlay
Figure 7: Lane Detection Results
The test results of the algorithm are shown in Fig. 7. The run time of processing single
frame is 1.24 seconds.
4

5. 5 Conclusion
This project covers the color and gradient threshold, perspective transform and sliding win-
dows to identify lane marks. The data set is chosen randomly with no camera specification.
For the binary image of perspective transformation Fig.3, the lane marks are not exactly
parallel. Generally, we calibrate the camera before applying the algorithm. This algorithm
is surprisingly robust to detect the lane marks accurately without camera parameters. There
are many improvements could be made to this project. In practice, the lane width may
be changing over time where the lane marks may not appear on the bottom of the frame.
Auto increasing search window width may help in this situation. Develop more flexible lane
mark extraction method to produce better binary image under various lighting conditions.
Apply other relevant computer vision techniques to reinforce the algorithm when noisy in
the image increases.
5