This document proposes using vehicle-mounted cameras on public vehicles like garbage trucks to detect blurred road markings as a way to efficiently monitor road infrastructure deterioration. An object detection approach is used that predicts the location, size, class and confidence of different types of road markings using a YOLO network with VGG16 features trained on a new dataset from drive recorder videos. Evaluation shows the approach detects white lines with 51.7% mAP but struggles with color lines and marks. Future work to improve detection accuracy includes expanding the dataset and exploring semi-supervised learning.

1. Road Marking Blur Detection
with Drive Recorder
Makoto Kawano, Kazuhiro Mikami, Satoshi Yokoyama,
Takuro Yonezawa and Jin Nakazawa
Keio University
makora@ht.sfc.keio.ac.jp
hide tokuda lab.

2. Outline
• Introduction
• Related work
• Our basic idea
• Problem setting
• Our approach
• Experiments
• Conclusion
2

3. Introduction 3
Roads are one of the important city infrastructures
Road markings Road appendages

4. Introduction 4
Road markings Road appendages
Roads are one of the important city infrastructures BUT become deteriorated

5. Introduction 5
Road markings Road appendages
Use special vehicles or visual inspection
Inspection and Repair doesn’t catch up
0
140,000
280,000
420,000
560,000
700,000
2008 2009 2010 2011 2012 2013 2014 2015
The cost of repair in Fujisawa (1,000JPY)
Increasing
Roads are one of the important city infrastructures BUT become deteriorated

6. Related Work
Road flatness estimation [1][2][3]
• Use accelerometer/gyro sensor on bicycle with factor analysis
Potholes detection [4][5]
• Use laser image via exclusive camera with naïve bayes
Road surface damage detection [6]
• Use image from participatory sensing with deep CNN
6
Road flatness
Potholes

7. Related Work
Road flatness estimation [1][2][3]
• Use accelerometer/gyro sensor on bicycle with factor analysis
Potholes detection [4][5]
• Use laser image via exclusive camera with naïve bayes
Road surface damage detection [6]
• Use image from participatory sensing with deep CNN
7
Road flatness
Potholes
In our study
Road markings damage/blur detection
• It is sufficient for not only cars but also citizens
• Cause traffic accident
• (To our knowledge) no other work tackle this issue

8. a
Basic Idea 8
Install sensors in the city Ask citizens to provide information[6]
Difficult to sustain the road inspection by
or
a.k.a Participatory Sensinge.g. Smart Santander Project

9. Basic Idea 9
Spotlight the public vehicles: garbage trucks with cameras or sensors
Piggybacking on their daily work takes no additional cost!
Coverage area of garbage trucks
3days
aInstall sensors in the city Ask citizens to provide information[6]or
a.k.a Participatory Sensinge.g. Smart Santander Project
Difficult to sustain the road inspection by

10. Our Previous Work 10
Split the images to patches
Normal marking
No marking
Blurred marking
3-class
Simple CNNTake photos
Web Camera
Completed system in a garbage truck alone

11. Our Previous Work 11
Split the images to patches
• Detect only for blurred road lane markings
• Require installing additional camera
Too hard constraints to deploy it to real world
Normal marking
No marking
Blurred marking
3-class
Simple CNN
{Assumption
The result was 98% accuracy but
Take photos
Web Camera
Completed system in a garbage truck alone

12. Problem Setting: Road Marking Blur Detection 12
Actual drive recorder image (a) white line (b) white mark (c) color mark (d) color line (e) crosswalk
Detecting various road markings
Using actual drive recorder video images
Estimate the location where the marking is blurred
• 5 types: white line / white mark / color mark / color line / crosswalk
• Add guardrail to discriminate lane markings

13. Location (x-axis and y-axis)
Size (width and height)
Increase confidence of object existing box
Decrease confidence of no object box
Class (classification)
Object Detection Approach 13
448×448×3
VGG16VGG16
Fully Connected
448×448×64
112×112×256
224×224×128
56×56×512
14×14×512
7×7×20
28×28×512
7×7×1024
convolution + ReLU
max pooling
output
YOLO[Redmon et al. 2015] + VGG16[Simonyan and Zisserman, 2015]
(ˆxi, ˆyi)
ˆwi
ˆhi
ˆCi(= 1), ˆpi(c)
(xi, yi)
Ci, pi(c)
wi
hi
Prediction
Ground truth

14. Create Dataset for Object Detection Approach 14
RectLabel
• Three people annotated using RectLabel
• 1 sunny day and 1 rainy day in the morning
• Split train and test set by time
Two trucks videos in 18 frames/sec Extract every 25 frames

15. Compare the Object Detection with Other Approach15
v.s our previous work
(simple CNN classifier)
v.s. full convolution network
[Shelhamer et al.]
Pros. Pros.
Cons.
• Seeing surround context • Computational cost is lower
• Annotation cost is lower
• Annotations is not precise
Cons.
• Computational cost is higher
Network would detect
almost vanished markings
FCN is too expensiveSimple classifier is not enough
here

16. Experiment 16
then the blur detection is correct
Evaluate our network how can detect the road markings blur
Area of Union
Area of Overlap
IoU Score =
Ground-truth bounding box
Predicted bounding box
> TmAP
Intersection of union
Prediction
blur none
Ground
truth
blur TP FN
none FP TN
Precision =
TP
TP + FP
mAP =
1
L
X
label2L
APlabel
APlabel =
1
n
nX
Precisioni
label
• Calculate mAP score:
(Note: in Contest)TmAP = 0.5

17. Quantitative Result 17
Not a good result…
• Did not detect the color line and color mark
(Note: SOTA mAP score is about 80 on PascalVOC 2007/2012)
• The best detection is white line with 51.7 mAP score

18. Quantitative Result 18
Not a good result…
• Did not detect the color line and color mark
(Note: SOTA mAP score is about 80 on PascalVOC 2007/2012)
• The best detection is white line with 51.7 mAP score
But what does the network detect actually?

19. 19

20. Qualitative Result 20
(a)
(c)
(b)
(d)
(a)
(c)
(e)
(b)
(d)
(f)
Correct
Prediction
Wrong
Prediction

21. Qualitative Result 21
(a)
(c)
(b)
(d)
The prediction box is much smaller than GT

22. Qualitative Result 22
(a)
(c)
(b)
(d)
The prediction box is much smaller than GT

23. Qualitative Result 23
(a)
(c)
(b)
(d)
The prediction box is much smaller than GT
TmAP
0
17.5
35
52.5
70
0.5 0.3 0.1 0.0
White line
White mark
Color line
Crosswalk
Guarrail
mAP
The lower threshold is, the higher mAP score is

24. Discussion and Future Work 24
Drive recorder image
Detecting on Edge Computer
Feedback to
citizens and
city administrator
Cloud computer
city analysis
Detection results
Road marking blur detection
Improve the mAP score
Deploy to real world
Brush up the dataset
• Add more images
• Define annotation rules
Propose the new approach
• Semi-supervised learning
Visualize the results to maps
• Tuning recall/precision to be useful
Implement on actual trucks
• Use embedding computers
• Connect to drive recorder

25. Conclusion 25
Detecting the blurred road markings by applying object detection approach
Actual drive recorder image (a) white line (b) white mark (c) color mark (d) color line (e) crosswalk
448×448×3
VGG16VGG16
Fully Connected
448×448×64
112×112×256
224×224×128
56×56×512
14×14×512
7×7×20
28×28×512
7×7×1024
convolution + ReLU
max pooling
output
object detection
neural network
• Create the annotated dataset for object detection for actual drive recorder
• Evaluate our network by mean average precision score

26. Road Marking Blur Detection
with Drive Recorder
Makoto Kawano, Kazuhiro Mikami, Satoshi Yokoyama,
Takuro Yonezawa and Jin Nakazawa
Keio University
makora@ht.sfc.keio.ac.jp
Thank you for listening!
Any questions?

27. 27

28. 28