License Plate Recognition

5,858 views

Published on

License Plate Recognition

Published in: Technology, Automotive

License Plate Recognition

  1. 2. <ul><li>Why read license plates? </li></ul><ul><li>Is this challenging? </li></ul><ul><li>Probably if solution is to be versatile… </li></ul>Introduction
  2. 3. <ul><li>Many methods </li></ul><ul><ul><li>Connected Component Analysis </li></ul></ul><ul><ul><li>Kwaśnicka, Wawrzyniak [2002] </li></ul></ul><ul><ul><li>Filtering, Edge Finding, Adaptive Threshold </li></ul></ul><ul><ul><li>Enyedi, Konyha, Fazekas [2006] </li></ul></ul><ul><ul><li>Sliding Windows, Histogram </li></ul></ul><ul><ul><li>Kaushik, Chae, Jo [2009] </li></ul></ul><ul><ul><li>Machine Learning, Binarization & Extension </li></ul></ul><ul><ul><li>Chen, Yuille [2004] </li></ul></ul><ul><ul><li>Machine Learning, Gradient-Based </li></ul></ul><ul><ul><li>Escalera et al. </li></ul></ul>Lit. Review
  3. 4. <ul><li>Localize license plate(s) </li></ul><ul><li>Binarize license plate </li></ul><ul><li>Extract license plate text </li></ul><ul><li>Transform text to flat orientation </li></ul><ul><li>Perform OCR on text </li></ul><ul><li>Post-process OCR output </li></ul>Methodology
  4. 5. <ul><li>Many solutions assume near-horizontal license plate </li></ul><ul><li>Chosen approach: Localize by training on text fragments </li></ul><ul><li>Idea: Be greedy in identifying possible license plate text, and discard non-license plate text during post-processing </li></ul>Methodology
  5. 6. <ul><li>What is “license plate text”? </li></ul><ul><li>Characteristics: </li></ul><ul><ul><li>Colour </li></ul></ul><ul><ul><li>Intensity Distribution </li></ul></ul><ul><ul><li>Entropy </li></ul></ul><ul><ul><li>Energy </li></ul></ul><ul><ul><li>Others… </li></ul></ul>1. Localization
  6. 7. <ul><li>First Attempt: Train on large blocks </li></ul><ul><li>Failed </li></ul><ul><li>Characteristics too varied to make good judgments on </li></ul>1. Localization
  7. 8. <ul><li>Second Attempt: Train on small blocks </li></ul><ul><li>Success! (well… mostly) </li></ul><ul><li>Some amount of false positives </li></ul><ul><li>from reflections, grilles, etc </li></ul>1. Localization Positive Negative (for white-on-black) ?
  8. 9. <ul><li>Classification </li></ul><ul><li>AdaBoost : Combines multiple weak classifiers linearly </li></ul><ul><li>Our eventual classifier: </li></ul><ul><ul><li>Colour </li></ul></ul><ul><ul><li>Number of regions in binarized edge thumbnail </li></ul></ul><ul><ul><li>Size distribution of regions </li></ul></ul>1. Localization (31 regions) (2nd 0.87 times 1 st ) (7 regions) (2nd 0.50 times 1 st ) (1 region)
  9. 10. <ul><li>Matching back to the input image… </li></ul><ul><li>Pre-process image by increasing contrast, sharpening to try and reduce blur* </li></ul><ul><li>Recursively partition image into quadrants </li></ul><ul><li>Each quadrant is resized to a small thumbnail </li></ul><ul><li>Each thumbnail is checked against classifier, and judged to be either TEXT or NONTEXT </li></ul>1. Localization *Possible blur removal approach: http://cs.nyu.edu/~fergus/research/deblur.html
  10. 11. <ul><li>Example Result 1 </li></ul>1. Localization
  11. 12. <ul><li>Example Result 2 </li></ul>1. Localization
  12. 13. <ul><li>Example Result 3 </li></ul>1. Localization
  13. 14. <ul><li>So how do we actually localize? </li></ul><ul><li>“ Intelligent guessing” </li></ul><ul><li>Connected region (of same granularity) </li></ul>1. Localization
  14. 15. <ul><li>2. K-means clustering (used K=2, multiple iterations) </li></ul>1. Localization
  15. 16. <ul><li>How many candidate regions do we get? </li></ul><ul><li>Usually, around 10 to 50 </li></ul><ul><li>More if no license plates are found at the default scale, and we have to subdivide the image further </li></ul>1. Localization
  16. 17. 1. Localization (35 candidate regions for this example)
  17. 18. <ul><li>Why go to all this trouble? </li></ul><ul><li>For each candidate region, we have to attempt to extract text </li></ul><ul><li>Therefore, the localization procedure cuts down tremendously on the amount of processing needed </li></ul><ul><li>A brute force search would otherwise yield astronomical numbers of candidate regions, due to the sheer number of combinations </li></ul>1. Localization
  18. 19. <ul><li>We resize each candidate region to around 40000 pixels, and convert it to black and white </li></ul><ul><li>Thankfully, this is relatively simple since a license plate is supposed to have high contrast between the plate colour and letter colour </li></ul><ul><li>We simply convert to grayscale, quantize the image to two colours, and normalize it </li></ul>2. Binarization
  19. 20. <ul><li>We create an edge image (high contrast, remember) (also works whatever the license plate colour) </li></ul><ul><li>We flood fill and keep only regions within certain dimensions relative to the edge image </li></ul>3. Extraction
  20. 21. <ul><li>We expect at most some skew of the license plate (no complex nonlinear distortions) </li></ul><ul><li>Eliminate remaining blobs unlikely to be characters </li></ul>4. Transformation Too far Too short
  21. 22. <ul><li>Study the character blobs to predict whether the license plate has one or two lines </li></ul><ul><li>Draw lines between all midpoints and see if there is a line that is close enough to all other midpoints (Justified by linearity of characters) </li></ul>4. Transformation min_d_max = 68.5 px min_d_max = 2.11 px
  22. 23. <ul><li>Map the corners to a standard template long rectangle if one-lined, and fat rectangle if two-lined (a simple homography, as in Lab 3) </li></ul>4. Transformation
  23. 24. <ul><li>We used Google Tesseract as suggested </li></ul><ul><li>Unfortunately, the raw results were not as good as expected </li></ul><ul><li>Possible explanation: License plate fonts are in general distinct from normal text font, which caused poor performance </li></ul><ul><li>Possible solution: Train Tesseract exclusively on license plate fonts </li></ul>5. OCR produces “ SGZ 35BBG ”
  24. 25. <ul><li>Standard Singapore license plates consist of two or three letters, then one to four digits, and one final letter </li></ul><ul><li>The final letter is a checksum for the other characters </li></ul>6. Post-processing
  25. 26. <ul><li>The formula is*: Let the license plate have the format [S] AB CDEF X </li></ul><ul><li>Then, we take the value V = ((A*9)+(B*4)+(C*5)+(D*4)+(E*3)+(F*2)) mod 19 </li></ul><ul><li>We can then determine X as follows: </li></ul><ul><li>It is therefore very easy to check if an OCR output is a valid license plate number </li></ul>6. Post-processing *Formula from: http://gangasudhan.com/blog/2008/07/nric-fin-number-suffix-checker.html B C D E G H J K L M P R S T U X Y Z A X 18 17 16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1 0 V
  26. 27. <ul><li>Given the accuracy (or lack thereof) of OCR when it comes to license plate fonts, it might be worthwhile to use this knowledge to repair poor OCR output </li></ul><ul><li>We note that OCR performance seems to be better on letters than on digits in general </li></ul><ul><li>Idea: If exactly one of the digits is misinterpreted as a letter, we can use the other characters (including the checksum) to correct it, since there are only ten possibilities for a digit </li></ul><ul><li>This is less useful once two or more digits are misinterpreted… </li></ul>6. Post-processing
  27. 28. <ul><li>We used Perl (with ImageMagick) for most of the coding </li></ul><ul><li>Test system is an Intel Core 2 Duo 3GHz CPU running Windows, with 3GB of RAM </li></ul><ul><ul><li>Runtime for Localization: ~2 or 3 minutes for standard granularity (6 levels of recursion) </li></ul></ul><ul><ul><li>Runtime for Binarization, Extraction: Varies; a few seconds per candidate region </li></ul></ul><ul><ul><li>Runtime for Transformation: Varies; ~10 seconds per candidate region </li></ul></ul><ul><ul><li>Runtime for OCR, Post-processing: negligible </li></ul></ul><ul><ul><li>Total expected runtime: 5-10 minutes </li></ul></ul>Implementation
  28. 29. <ul><li>A framework for a robust license plate recognition program has been implemented </li></ul><ul><li>It has the following properties: </li></ul><ul><ul><li>Scale-invariant </li></ul></ul><ul><ul><li>Deals with moderate skew </li></ul></ul><ul><ul><li>Deals with multiple license plates </li></ul></ul><ul><ul><li>Possibly corrects erroneous reads </li></ul></ul><ul><li>Future challenges: </li></ul><ul><ul><li>Characters that run into each other </li></ul></ul><ul><ul><li>More sophisticated blur removal </li></ul></ul><ul><ul><li>Increase speed by reimplementation in a faster programming language </li></ul></ul>Conclusion
  29. 30. Questions?

×