This document summarizes a thesis presentation on an autonomous robotic system for nondestructive evaluation of asphalt pavement using deep learning. The system uses a robot equipped with vision sensors and an impact echo sensor. Deep learning models are used to detect cracks from images and classify crack severity from impact echo signals. The robot can autonomously collect data, perform real-time crack detection using onboard processing, and present severity maps quantifying the cracks. The system provides a low-cost way to inspect roads and quantify cracking issues. Future work could improve low-light crack detection, evaluate subsurface conditions, and integrate additional sensors to cover more area faster.

1. Md.Al-Masrur Khan1
Advisor: Seong-Hoon Kee1,2*
1Department of ICT Integrated Ocean Smart Cities, Dong-A University, Busan, Republic of Korea
2University Core Research Center for Disaster-free & Safe Ocean City Construction, Dong-A University,
Busan, Republic of Korea
THESIS PRESENTATION
28th November 2022
Autonomous Robotic Assisted System for Nondestructive
Evaluation of Asphalt Pavement using Deep Learning
Department of ICT Integrated Ocean Smart Cities Engineering, Dong-A University

2. Presentation Layout
Introduction (Background & Research Motivation)
01
02 Literature Review & Novelty
03 Methodology
04 Results
05 Conclusion
Robotic System
Top View
Front View

3. Research Motivation
1/11
Background & Research Motivation
 Roads in Korea are constituted by a length of
105673 Km of which 89,701 Km are the paved
roads (91.6%).
 These Pavement roads can be damaged due to
various reasons
• Surface cracking
• Honey-Comb
• Delamination
• Exposure to the sun
• Rain erosion
• Natural Weathering
• Long-term driving of the vehicles
 If these cracks cannot be found and repaired in time , it will have a negative im
pact on the safe driving of vehicles.
Objectives of this research
• To build a robotic assisted automated system for performing NDE on road pavements.
Vision Sensors: Cheap, Easy to Use
Image Quality sensitive to light
Image processing method
sensitive shadow problem, Noise
Impact Echo: Easy to use, not
sensitive to environment
Slow operation process
Data (Image) Data (Elastic Wave)

4. Research Motivation
2/11
Literature Review & Novelty
Novelty:
 Combining automated data acquisition, crack detection using deep learning on the robot’s
onboard computer and finally presenting severity maps by crack measurement.
Literature review:
Researchers Inspected Structure Robot – Platform Deep Learning Remarks
Yu et al. [1] Concrete Tunnel Mobile robot No
Images were collected by the robotic system. An ima
ge processing algorithm was utilized in an external c
omputer for detecting cracks and crack information
Oyekola et al. [2] Concrete Tank Mobile robot No
Images were collected by the robotic system. A thre
shold-based algorithm was used in another compute
r for detecting the cracks. No postprocessing techni
ques were applied for obtaining geometrical informa
tion about the cracks.
La et al. [3] Bridge deck Seekur robot No
Combined visual sensor and NDE sensors for crack i
nspection. Presented stitched images after crack det
ection and delamination map.
Ramalingam et al.
[4]
Concrete pavement Panthera robot Yes
A SegNet-based model was developed to detect cra
cks and garbage. The system detects cracks on the o
nboard computer (Nvidia Jetson nano). A Mobile Ma
pping System was also utilized to localize the cracks.
Gui et al. [5] Airport pavement ARIR robot Yes
Both surface and subsurface data were collected by
a camera and GPR interfaced into the robotic syste
m. An intensity-based algorithm and voting-based C
NN were applied for processing image and GPR dat
a. A large-scale stitched image was presented to vis
ualize the cracks.
Gap of Knowledge:
No previous works use Deep learning methods for inspecting cracks from multiple sensors for real-time
monitoring and presenting severity maps of the detected cracks.

5. Research Motivation
3/11
Methodology
Step-1
Autonomous Robotic
System

6. Research Motivation
4/11
Methodology
Step 2 (HOST PC) : Graphical User Interface to Control Robot
Step 3 : Data Collection
Image Impact Echo Signal

7. Research Motivation
5/11
Methodology
VIDEO
The diagram of survey area for
the AMSEL robot.

8. Research Motivation
6/11
Methodology (Processing)
Deep learning model for crack segmentation from image
Deep learning model for IE signal prediction

9. Research Motivation
7/11
Methodology (Quantifying)

10. Research Motivation
8/11
Results
Stitched original image 2.5m × 1m grid Stitched predicted image 2.5m × 1m grid
original image predicted image image after quantifying

11. Research Motivation
9/11
Results
Severity distribution from image 2.5 × 1 grid Severity distribution from impact echo 2.5 x 1 grid
Total Area Maximum Area Minimum Area Density
15231.88mm2 1741.35mm2
Location(x=3,y=0.25)
308.2025mm2
Location(x=4,y=0.5)
0.60%
The grid is 0.60% cracked of its total area
We also defined severity level based on our
data set

12. Research Motivation
10/11
Results
Comparison between the manually measured and digitally measured crack size outdoor.
Linear regression between manually measured data and digitally measure data
(a) Length of the cracks (b) Width of the cracks

13. Research Motivation
11/11
Conclusions
Conclusion Remarks:
Developed Robotics platform has following features:
1. Low-cost deep learning model for implementing it on the robot to detect cracks from
the RGB images in real-time.
2. Presents an Impact Echo dataset for classifying the crack severity using deep learning.
3. Presents a deep learning classifier for classifying the crack severity type from elastic
wave signals collected by the impact echo method.
4. Presents a crack quantification algorithm for finding out crack length, width, and area.
5. Finally, presents a visualization of the crack severity map.
• Better segmentation model, which can detect cracks even in low illumination condition as
well as in extreme shadow.
• To improve the capability of the IE system for evaluating subsurface conditions (i.e., depth
of cracks, delamination etc.).
• To integrate other NDE sensors including GPR, USW, ER, etc. To add multiple visual sensors
for covering a large area quickly to make the inspection process faster.
Prospects:

14. Research Motivation