This document discusses a method for detecting and classifying road defects using image processing and machine learning algorithms. The method involves collecting road pavement images, segmenting the images using graph cuts to identify regions of interest, extracting features from those regions, and classifying the defects using a random forest algorithm. Experimental results on road images from Irkutsk showed encouraging performance for automatic identification of cracks, potholes and other pavement defects.

1. Introduction
Data Collection
Construction of Map of Defects
Defects Detection and Classification Method on Road Pavement
Conclusion
Irkutsk State Technical University
On road defects detection and classification
Thu Huong Nguyen, Aleksei Zhukov, The Long Nguyen
Irkutsk State Technical University
The 7th to 9th April 2016
Nguyen Thu Huong, Aleksei Zhukov, The Long NguyenAIST 2016 On road defects detection and classification 1 / 21

2. Introduction
Data Collection
Construction of Map of Defects
Defects Detection and Classification Method on Road Pavement
Conclusion
Irkutsk State Technical University
Outline
Introduction
Data Collection
Construction of Map of Defects
Defects Detection and Classification Method on Road Pavement
Conclusion
Nguyen Thu Huong, Aleksei Zhukov, The Long NguyenAIST 2016 On road defects detection and classification 2 / 21

3. Introduction
Data Collection
Construction of Map of Defects
Defects Detection and Classification Method on Road Pavement
Conclusion
Irkutsk State Technical University
Introduction
• In this talk proposes an automatic defect pavement detection
and classification system capable of identifying and retrieving
pavement surface images containing block cracks, longitudinal
cracks, potholes from a road pavement survey image database.
• The experimental results, achieved using images from Irkutst
roads, are encouraging for the development of automatic
pavement defects detection systems.
Nguyen Thu Huong, Aleksei Zhukov, The Long NguyenAIST 2016 On road defects detection and classification 3 / 21

4. Introduction
Data Collection
Construction of Map of Defects
Defects Detection and Classification Method on Road Pavement
Conclusion
Irkutsk State Technical University
Motivation
Nguyen Thu Huong, Aleksei Zhukov, The Long NguyenAIST 2016 On road defects detection and classification 4 / 21

5. Introduction
Data Collection
Construction of Map of Defects
Defects Detection and Classification Method on Road Pavement
Conclusion
Irkutsk State Technical University
Objective
• We proposes algorithms: Graph cuts method for images
segmentation, Random forest for classification and image
processing algorithms for features extraction.
• Automatic learning methods are used, capable of learning the
image statistical features from texture variations of the road
background and of the defects areas.
• The features studied are Histogram, Histogram chain code,
Moments-hull, shape of features.
Nguyen Thu Huong, Aleksei Zhukov, The Long NguyenAIST 2016 On road defects detection and classification 5 / 21

6. Introduction
Data Collection
Construction of Map of Defects
Defects Detection and Classification Method on Road Pavement
Conclusion
Irkutsk State Technical University
Main steps
1.To detect defect position (ROI).
2.Defect is described by its features.
3.To classify defect each using these different defect features such
as Chain Code Histogram, Hu-Moments, size of defect
region(width and length, area) and histogram of image.
Our approach
The following algorithms have been used: Graph cuts method for
image segmentation, Random Forests algorithm for data
classification.
Nguyen Thu Huong, Aleksei Zhukov, The Long NguyenAIST 2016 On road defects detection and classification 6 / 21

7. Introduction
Data Collection
Construction of Map of Defects
Defects Detection and Classification Method on Road Pavement
Conclusion
Irkutsk State Technical University
Data Collection
We used two datasets:
1 Our own dataset include:
• 500 images are collected by camera (Canon D100 16 mega
pixel).
• Images are captured in conventional daylight condition.
• Distance from camera to surface of road is 1m-1.2m.
2 SARA
• More 700 images.
• Collection by Center for Telecommunications and Multimedia,
INESC TEC, Portugal.
Nguyen Thu Huong, Aleksei Zhukov, The Long NguyenAIST 2016 On road defects detection and classification 7 / 21

8. Introduction
Data Collection
Construction of Map of Defects
Defects Detection and Classification Method on Road Pavement
Conclusion
Irkutsk State Technical University
Segmentation by Graph cuts method
• Graph cuts uses power optimization algorithm, which is
applied specifically to those models which employ a
max-flow/min-cut optimization (other graph cutting
algorithms may be considered as graph partitioning
algorithms).
• Finds strong local minima of our np-complete energy function.
• Graph-cuts have been around in computer vision for quite
some time (e.g. [Roy,ICCV98]).
Nguyen Thu Huong, Aleksei Zhukov, The Long NguyenAIST 2016 On road defects detection and classification 8 / 21

9. Introduction
Data Collection
Construction of Map of Defects
Defects Detection and Classification Method on Road Pavement
Conclusion
Irkutsk State Technical University
Segmentation by Graph cuts method
Figure: Example Graph cuts segmentation method
Nguyen Thu Huong, Aleksei Zhukov, The Long NguyenAIST 2016 On road defects detection and classification 9 / 21

10. Introduction
Data Collection
Construction of Map of Defects
Defects Detection and Classification Method on Road Pavement
Conclusion
Irkutsk State Technical University
Max-Flow Problem
Task : Maximize the flow from the sink to the source such that:
• The flow it conserved for each node
• The flow for each pipe does not exceed the capacity
Nguyen Thu Huong, Aleksei Zhukov, The Long NguyenAIST 2016 On road defects detection and classification 10 / 21

11. Introduction
Data Collection
Construction of Map of Defects
Defects Detection and Classification Method on Road Pavement
Conclusion
Irkutsk State Technical University
Ford Fulkerson algorithm (1956)
Input: Given a network G = (V , E) with flow capacity c, a source
node s, and a sink node t
Output: Compute a flow f from s to t of maximum value.
f (u, v) ← 0 for all edges (u, v)
while there exists a path p from s to t in the residual network Gj
do
Find cf (p) = min {cf (u, v) | (u, v) ∈ p} for each
edges(u, v) ∈ p do
f [u, v] = f [u, v] + cf (p);
f [v, u] = f [v, u] − cf (p);
end
end
Algorithm 1: Ford Fulkerson algorithm (1956)
Nguyen Thu Huong, Aleksei Zhukov, The Long NguyenAIST 2016 On road defects detection and classification 11 / 21

12. Introduction
Data Collection
Construction of Map of Defects
Defects Detection and Classification Method on Road Pavement
Conclusion
Irkutsk State Technical University
Result of max flow
Nguyen Thu Huong, Aleksei Zhukov, The Long NguyenAIST 2016 On road defects detection and classification 12 / 21

13. Introduction
Data Collection
Construction of Map of Defects
Defects Detection and Classification Method on Road Pavement
Conclusion
Irkutsk State Technical University
Min-Cut Problem
Task : Minimize the cost of the cut
• Each node is either assigned to the source S or sink T
• The cost of the edge (i, j) is taken if (i ∈ S) and (j ∈ T)
Finding min-cut |C| = e∈C we
Nguyen Thu Huong, Aleksei Zhukov, The Long NguyenAIST 2016 On road defects detection and classification 13 / 21

14. Introduction
Data Collection
Construction of Map of Defects
Defects Detection and Classification Method on Road Pavement
Conclusion
Irkutsk State Technical University
Result of Graph cut segmentation
Figure: Fig(a) Result of Graph cut segmentation method. Fig(b) Result
of Random forest algorithm
Nguyen Thu Huong, Aleksei Zhukov, The Long NguyenAIST 2016 On road defects detection and classification 14 / 21

15. Introduction
Data Collection
Construction of Map of Defects
Defects Detection and Classification Method on Road Pavement
Conclusion
Irkutsk State Technical University
Flowchart
End
Load model classification of
machine learning
Load road pavement image
database
Classification based on
RandomForest algorithm
Return type of defect road
pavement
Create features vector
Features extraction
Preprocessing image
Begin
Nguyen Thu Huong, Aleksei Zhukov, The Long NguyenAIST 2016 On road defects detection and classification 15 / 21

16. Introduction
Data Collection
Construction of Map of Defects
Defects Detection and Classification Method on Road Pavement
Conclusion
Irkutsk State Technical University
Random Forest algorithm
• Random forest (or random forests) is an ensemble classifier
that consists of many decision trees and outputs the class that
is the mode of the class’s output by individual trees.
• The method combines Breiman’s ”bagging” idea and the
random selection of features.
Nguyen Thu Huong, Aleksei Zhukov, The Long NguyenAIST 2016 On road defects detection and classification 16 / 21

17. Introduction
Data Collection
Construction of Map of Defects
Defects Detection and Classification Method on Road Pavement
Conclusion
Irkutsk State Technical University
Flowchart
Figure: Flow chart of Random Forest algorithm [Girish (2015)].
Nguyen Thu Huong, Aleksei Zhukov, The Long NguyenAIST 2016 On road defects detection and classification 17 / 21

18. Introduction
Data Collection
Construction of Map of Defects
Defects Detection and Classification Method on Road Pavement
Conclusion
Irkutsk State Technical University
Random Forest practical consideration
• Splits are chosen according to a purity measure:
E.g. squared error (regression), Gini index or devinace
(classification)
• How to select N?
Build trees until the error no longer decreases
• How to select M?
Try to recommend defaults, half of them and twice of them
and pick the best.
Nguyen Thu Huong, Aleksei Zhukov, The Long NguyenAIST 2016 On road defects detection and classification 18 / 21

19. Introduction
Data Collection
Construction of Map of Defects
Defects Detection and Classification Method on Road Pavement
Conclusion
Irkutsk State Technical University
Training time, Correct rate and Error test of Random
Forest classification algorithm
Random Forest 100 trees 50 trees 100 trees 100 trees
depth:2 depth:2 depth:5 depth:10
Training time(sec) 250 150 50 140
Correct rate (%) 91.45 80.5 93.29 96.66
MSE 0.393 0.516 0.366 0.3
Table: Training time, Correct rate and Error test of Random Forest
classification algorithm
Nguyen Thu Huong, Aleksei Zhukov, The Long NguyenAIST 2016 On road defects detection and classification 19 / 21

20. Introduction
Data Collection
Construction of Map of Defects
Defects Detection and Classification Method on Road Pavement
Conclusion
Irkutsk State Technical University
Conclusion
• In this talk we suggested the novel approach for road
pavements defects automatic detection and classification.
This method is based on the construction of an irregular
lattice derived from the original image. The lattice is
composed only by straight line segments.
• We also propose to use to Graph cut method, which improve
quality of image segmentation. From this we can detection
part of pavement defect - non defect.
• The classification algorithm - Random Forest was able to
correctly classify all the images contained in the two first sets.
In the test set simulating the real environment the achieved
classification results were 95,5%.
Nguyen Thu Huong, Aleksei Zhukov, The Long NguyenAIST 2016 On road defects detection and classification 20 / 21

21. Introduction
Data Collection
Construction of Map of Defects
Defects Detection and Classification Method on Road Pavement
Conclusion
Irkutsk State Technical University
THANK YOU SO MUCH !!!
Nguyen Thu Huong, Aleksei Zhukov, The Long NguyenAIST 2016 On road defects detection and classification 21 / 21