Improved license plate recognition for low resolution cctv forensics by integrating sparse representation-based super-resolution.

1. Improved License Plate Recognition for
Low-Resolution CCTV Forensics by Integrating
Sparse Representation-based Super-Resolution
12th International Workshop on
Digital Forensics and Watermarking
Auckland, New Zealand
October 4, 2013
Hyun-seok Min, Seung Ho Lee,
Wesley De Neve, Yong Man Ro
Image and Video Systems Lab
Department of Electrical Engineering
Korea Advanced Institute of Science and Technology (KAIST)
Daejeon, Republic of Korea

2. Improved License Plate Recognition for Low-Resolution CCTV Forensics by
Integrating Sparse Representation-based Super-Resolution
Overview
• Introduction
• Proposed method for license plate recognition
• Experimental results
• Conclusions
2/20

3. Improved License Plate Recognition for Low-Resolution CCTV Forensics by
Integrating Sparse Representation-based Super-Resolution
License Plate Recognition
• Automatic license plate recognition (LPR)
– important for CCTV forensics
– important for privacy protection
3/20
Large-scale Privacy Protection
in Google Street View
LPR in a CCTV system

4. Improved License Plate Recognition for Low-Resolution CCTV Forensics by
Integrating Sparse Representation-based Super-Resolution
Challenges
• Low effectiveness of LPR for real-world imagery
– due to difficult capturing conditions
• E.g., frequent use of long-distance CCTV cameras
– output noisy LP images with a low resolution
4/20
Example input images captured by long-distance CCTV cameras in Korea

5. Improved License Plate Recognition for Low-Resolution CCTV Forensics by
Integrating Sparse Representation-based Super-Resolution
Super-Resolution (1/2)
• Super-resolution can be used to improve the
effectiveness of current LPR systems
– enhances images by producing high-resolution
images from one or more low-resolution images
5/20

6. Improved License Plate Recognition for Low-Resolution CCTV Forensics by
Integrating Sparse Representation-based Super-Resolution
Super-Resolution (2/2)
• Shortcomings of current approaches
– focus on maximizing the SNR between the original
high-resolution image and the reconstructed image
• do not focus on maximizing the effectiveness of LPR
– focus on the use of general image information
• do not take advantage of specific image priors, such as the
font, location, and color of LP characters
6/20
Super-resolution
Training images
Similar?

7. Improved License Plate Recognition for Low-Resolution CCTV Forensics by
Integrating Sparse Representation-based Super-Resolution
Proposed LPR Method (1/2)
• Uses high-resolution LP images for
1) creating a dictionary for sparse representation-based
(SR-based) super-resolution of the test LP images
2) the training of LP character classifiers (SVM)
7/20
Proposed LPR system
License
plate
detection
Sparse
representation-
based super-
resolution
Image patchesInput image
High-quality
training images
...
...
...
...
...
...
...
...
Classifier for 1
Classifier for 2
...
Classifier for 0
Recognition of
license plate
characters
...
Output
CCTV

8. Improved License Plate Recognition for Low-Resolution CCTV Forensics by
Integrating Sparse Representation-based Super-Resolution
Proposed LPR Method (2/2)
• Advantages
– takes into account specific image priors
• e.g., the font, location, and color of LP characters
– shares specific image priors among different steps
• SR-based resolution enhancement of the test LP images
• SVM-based LP character recognition
8/20

9. Improved License Plate Recognition for Low-Resolution CCTV Forensics by
Integrating Sparse Representation-based Super-Resolution
Sparse Representation-based
Super-Resolution for LPR (1/2)
• We enhance the quality of noisy and low-resolution
LP images through SR-based super-resolution
– to that end, we make use of a dictionary that contains a
large number of LP image patches that have been
extracted from high-quality LP images
9/20
Kd
Ki
×
= R],...,,...,[ 1 ∈xxxD
xi : the feature vector of the ith LP image patch extracted
from the set of high-resolution training images
d : the dimensionality of the feature vectors
K: the number of LP image patches in the dictionary

10. Improved License Plate Recognition for Low-Resolution CCTV Forensics by
Integrating Sparse Representation-based Super-Resolution
Sparse Representation-based
Super-Resolution for LPR (2/2)
10/20
License
plate
detection
Sparse
representation-
based super-
resolution
Image patches
High-quality
training images
...
...
...
...
...
...
...
...

11. Improved License Plate Recognition for Low-Resolution CCTV Forensics by
Integrating Sparse Representation-based Super-Resolution
Experimental Setup:
Test Set (1/2)
• Test set characteristics
– 600 input images captured by CCTV cameras in Korea
– resolution of input images: 1280×720
– we extracted 772 LP images from the 600 input images
by making use of sliding concentric windows (SCW)
• each input image contains one or more LP images
11/20
Example input images captured by long-distance CCTV cameras in Korea

12. Improved License Plate Recognition for Low-Resolution CCTV Forensics by
Integrating Sparse Representation-based Super-Resolution
Experimental Setup:
Test Dataset (2/2)
• We divided the LP images in the test set into four
groups, taking into account their spatial resolution
1) Resolution Group 1: LP image width ≤ 40 pixels
2) Resolution Group 2: 40 pixels < LP image width ≤ 55 pixels
3) Resolution Group 3: 55 pixels < LP image width ≤ 70 pixels
4) Resolution Group 4: 70 pixels < LP image width
12/20
Resolution Group 1 Resolution Group 2 Resolution Group 3 Resolution Group 4

13. Improved License Plate Recognition for Low-Resolution CCTV Forensics by
Integrating Sparse Representation-based Super-Resolution
Experimental Setup:
Dictionary Construction
• Dictionary construction (& classifier training)
– we made use of 100 high-resolution LP images that
have been downloaded via Google Image Search
– we randomly extracted 10,000 image patches from the
training images
• image patches have a size of 10×10 pixels
– we made use of a dictionary with 10,000 image patches
13/20
Example high-resolution LP images

14. Improved License Plate Recognition for Low-Resolution CCTV Forensics by
Integrating Sparse Representation-based Super-Resolution
Experimental Results:
Influence of Resolution Enhancement (1/2)
14/20

15. Improved License Plate Recognition for Low-Resolution CCTV Forensics by
Integrating Sparse Representation-based Super-Resolution
Experimental Results:
Influence of Resolution Enhancement (2/2)
• Observation
– the effectiveness of the proposed LPR method is
better than the effectiveness of
• LPR without resolution enhancement
• LPR using bicubic interpolation
• Visual examples
15/20
(a) original low-resolution LP images (b) LP images that are the
output of bicubic
interpolation
(c) LP images that are the
output of SR-based super-
resolution
45×22 56×25

16. Improved License Plate Recognition for Low-Resolution CCTV Forensics by
Integrating Sparse Representation-based Super-Resolution
Experimental Results:
Dictionary Influence (1/2)
• We enhanced the resolution of the test LP
images by means of two types of dictionaries
1) a dictionary consisting of high-resolution LP images
2) a dictionary consisting of general images
16/20
Examples of general imagesExample high-resolution LP images

17. Improved License Plate Recognition for Low-Resolution CCTV Forensics by
Integrating Sparse Representation-based Super-Resolution
Experimental Results:
Dictionary Influence (2/2)
• Observation
– SR-based super-resolution by means of high-quality
LP training images is more effective than SR-based
super-resolution by means of general images
17/20

18. Improved License Plate Recognition for Low-Resolution CCTV Forensics by
Integrating Sparse Representation-based Super-Resolution
Experimental Results:
Influence of Low-Level Visual Features
• Observation
– color Gabor wavelet features are most effective
18/20

19. Improved License Plate Recognition for Low-Resolution CCTV Forensics by
Integrating Sparse Representation-based Super-Resolution
Conclusions
• We introduced a novel LPR method
– applies SR-based super-resolution to test LP images
– uses a common set of high-quality LP images to
facilitate
• SR-based super-resolution of LP test images
• SVM-based LP character recognition
• Our results indicate that the proposed method
allows improving the effectiveness of LPR in
real-world imagery
19/20

20. Any questions or comments?
Contact information
Hyun-seok Min: hsmin@kaist.ac.kr
Web: http://ivylab.kaist.ac.kr
20/20