Final Thesis Presentation Licenseplaterecognitionincomplexscenes


Published on

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Final Thesis Presentation Licenseplaterecognitionincomplexscenes

  1. 1. Department of Electrical and Computer Engineering Illinois Institute of Technology, Chicago Masters’s Thesis Defense Presentation 2011 November 23 Dhawal S Wazalwar Examination Committee Dr Erdal Oruklu (advisor) Dr Jafar Saniie Dr Tricha Anjali
  2. 2.  Introduction to license plate recognition system Background of previously implemented systems• A robust flow for license plate recognition • License plate detection • Character segmentation • Feature extraction and character recognition Experimental results Conclusion Further improvements
  3. 3.  A rapidly evolving technology used in surveillance systems like management of road traffic, speed checks, vehicle parking etc. Types • Fixed camera systems • Example:- Vehicle tracking at toll centers • Mobile camera systems • Example: - Camera mounted on police vehicles, which constantly scan the license plates in all directions, during patrol operation.
  4. 4. Image Frame Optical Character License Plate Acquisition Processing Recognition Image Database Focus is on improving license plate detection and recognition performance for complex scenes.
  5. 5.  Binary level processing • Edge detection and mathematical morphological operations are used to detect license plate information. • Faster execution speed but sensitivity to unwanted edges is an issue. Gray level processing • It uses local contrast information and thresholding techniques to differentiate between all possible object region. Color level processing • Unique color information of the license plate characters and its background details is used. These methods have more accuracy but relatively slow in exceution. • Greater accuracy but the huge amount of color data needs to be processed, so slow in comparison with previous two methods.
  6. 6.  Horizontal and vertical projection data based techniques • Image column and row data is added to obtain a vector. Binary algorithms and morphology based techniques • Connected component analysis along with object measurements like height, width, area is used. Contour based techniques • These techniques are based on shape descriptors like curvature, gradient information. Gray level histogram techniques • Histogram valley is used for thresholding purposes, which is a key aspect of character segmentation
  7. 7.  Feature vector • Uniqueness of feature decides the character recognition performance. • Horizontal and vertical projection data, region properties like area, perimeter, orientation are commonly used features. • Concept of stroke direction and elastic mesh showed an accuracy of about 97.8-99.5 %• Selection of Classifiers • Statistical classifiers like support vector machine, Markov models have been widely with success rate of about 95-99 %. • Various architectures of artificial neural networks, mostly working on binary data have been proposed.
  8. 8.  The efficiency of any license plate recognition system depends on how accurately we can detect the license plate location. Key steps in the process • Edge detection: The main objective is to obtain clear and distinct edges of license plate with minimal noise content. • Morphological dilation operation: This step helps to thicken the edges obtained after the edge detection step. • Region growing segmentation: It basically divides the entire region into small segments based on pixel connectivity information. • Criterion for license plate selection: The main aim here is to determine the most optimal criteria for identifying license plate region after region growing segmentation.
  9. 9.  Simple edge detection techniques using Prewitt, Sobel operator work well for cases where there exists a clear and distinct contrast between the license plate and region surrounding it. However, for cases where there exists a partial contrast between license plate region and surrounding part, more complex edge detection using Mexican hat operator is required. Mexican hat operator first performs smoothing operation and then extracts edges. Mathematically represented as r2  r 2   2   2 2  h r     2 4  e  
  10. 10.  Case 1: Clear and distinct contrast Edge detection using Example Image 1 Prewitt operator Case 2: Partial Contrast Edge detection using Edge detection using Example Image 2 Prewitt operator Mexican Hat operator
  11. 11.  Dilation is considered as local maximum operation used to add pixel information to boundaries in an image. It is mathematically represented as A⊕B = { z | B ∩ A ≠ Φ} where A is a binary image and B is the structuring element used. The size and nature of structuring element is important since smaller size can negate the desired effect and larger size can cause the license plate regions to be connected to the surrounding area. For our algorithm, best results were achieved with 3x3 ones matrix as a structuring element.
  12. 12. Blur license plate image Edge detection output Detection result without dilation Dilation operation output Detection result with dilation
  13. 13.  For detecting a license plate, we first need to come up with all possible plate region. Region growing with pixel connectivity information is used to divide entire image into several sub-regions.• The performance of region growing depends on selection of • Type of pixel connectivity information used • Starting point for algorithm, seed point selection • Stopping rule selected for the implementation
  14. 14. 4-pixel connectivity 8-pixel connectivity8-pixel connectivity will have better performance, however,computation time required will be more.
  15. 15. Original Image Seed point= 200Seed point= 230 Seed point= 245
  16. 16. Original Image Stopping rule=25 pixelsStopping rule =55 Stopping rule=75 pixels pixels
  17. 17.  For every bright pixel in the binary output after dilation, its neighboring four pixel connectivity information was checked and region was grown based on this information. In order to speed up the process, sub-sampling was used, where algorithm operation was performed for every two pixels, instead of performing for each and every pixel in the image. A lower and upper bound was defined to remove the unwanted regions (non license plate regions).
  18. 18. Before region Output after regiongrowing step growing
  19. 19.  The main aim here is to determine the most optimal criteria for identifying license plate region after region growing segmentation. Criterion 1: License plate dimensions • Easy to implement • Disadvantage is that the algorithm becomes extremely sensitive to license plate dimensions. Criterion 2: Euler number of a binary image • Euler number of a binary image gives the number of objects minus the closed curves (holes). • Experimentally, it was found that in alphanumeric regions, value of Euler number is zero or negative, while for other regions it has a positive value. • Algorithm may fail if vehicle image has alphanumeric characters in more than one region.
  20. 20.  Criterion 3 : Combination of Euler number of a binary image and license plate dimensions • A combined approach of using Euler number criterion as well as license plate dimensions helps to overcome problem raised in case of Criterion 2 Output using Criterion 2 Output using Criterion 3
  21. 21.  After locating the license plate, the next important step is to segment each character individually, avoiding any possibility of joint characters. Key steps in the process • Image pre-processing: In this step, we apply median filter followed by contrast enhancement operation. • Thresholding: In this we apply local thresholding algorithm which converts grayscale license plate into binary image for further processing. • Erosion operation: This operation basically helps to remove dirt/shadow present in the license plate image. • Region growing segmentation
  22. 22. Original Image Gaussian noiseSalt and pepper noise Speckle noise
  23. 23.  Median filter is a non-linear spatial filter and is used widely for image denoising. In median filtering, we replace the original pixel data by median of pixels contained in a prescribed window. Median extremely useful because it important characteristics of the image like boundaries, edges.
  24. 24.  If the image acquired is blurred, it will be difficult to process it further, since most of its character information is not properly visible. It was found that the character information for most of the images in database is usually in the intensity range of 0-0.4 on a scale of 1. In order to obtain further contrast enhancement, we increased the dynamic range for pixels in this (0-0.4) intensity range.
  25. 25. Original license plate image Enhanced version after preprocessing
  26. 26.  Its basically a segmentation technique used to separate object and background information. Global thresholding • A single threshold value is applied for entire image.• g ( x, y)  1, otherwiseT 0, f ( x, y )  • It is easy to implement, however single dark or bright region can corrupt the threshold value. Local thresholding • The entire image is divided into several parts by using windowing operation and then perform the thresholding for each case separately. For our algorithm, we have used Otsu’ threshold method. In this method, gray level histogram is used to determine the optimal threshold value for which the intra class variance between the two classes is minimal.
  27. 27. Original ImageGlobal threshold operation Local threshold operation
  28. 28.  In cases where there exist dirt/shadow on license plate, local thresholding may not be sufficient all the time. Erosion operation is required to remove the dirt/shadow part. Erosion operation mathematically can be represented as X ΘB   x |  B x  X The structuring element is selected such that character information remains intact and only the noise part is removed.
  29. 29. Original Image Local threshold operationOutput after erosion operation Segmented characters after region growing
  30. 30.  Two stage process • Firstly we have to extract key features associated with the characters. • Secondly, a mechanism has to be defined to recognize these characters based on the extracted features. Feature extraction • The robustness of character recognition algorithm depends on how well it can recognize different versions of a single character. • Algorithm should be able to distinguish between characters like D and 0 or 5 and S because of the similarity in their appearance as well as spatial information.
  31. 31. Different version of ‘G’
  32. 32.  Feature 1: Region properties like area, perimeter, orientation, minor-major axis length. Feature 2: Projection information about X and Y axes Feature 3: Some of the characters like 0, M, N are symmetric about X-Y axes. This information can be used to distinguish different characters especially 0 and D. This symmetry information can be quantified by a coefficient called as reflection symmetry coefficient.
  33. 33.  Calculation • Consider Cin as the original character matrix where information is stored in binary format. The center of matrix is considered as the origin. Cx and Cy are the transformed matrix obtained after flipping about the X and Y axes respectively. • Calculate the area of original matrix Cin • Determine the union of transformed matrixes Cx and Cy with original matrix Cin • Calculate the areas of regions (Cin ∪ Cx) and (Cin ∪ Cy) • The reflection coefficient can be determined as Area  Cin  Xcoefficient  Area  Cin  C x  Area  Cin  Ycoefficient  Area  Cin  C y 
  34. 34.  A mathematical model called artificial neural network based on interconnection of artificial neurons, which can be trained by flow of external and internal information was developed. Extensively used for solving pattern classification, clustering, function approximation. Classification • Feed forward neural networks • Recurrent neural networks
  35. 35.  Input layers to the neural network are the feature vectors extracted for the all the characters. The overall size is decided by the character set and the feature vector size. Output nodes are the set of all possible characters that are present in license plate information. The number of output layers for our implementation is 33. There is no generic rule as such for deciding the number of hidden layers. In our design, the number of hidden layers was experimentally found to be 25. We obtained a set of 50 images for each character from the database images and trained the neural network using them.
  36. 36. Samples of character ‘5’ Samples of character ‘S’If the characters are possibly between 5, 0, S and D,then region properties like orientation, reflectioncoefficient were again used in the 2nd stageidentification process and the final possible characterwas identified
  37. 37.  The proposed algorithm was tested on images with varying illumination conditions taken during day as well as night times, at different angles and difficult shadows. Entire database was divided in following types:- • Type 1: In these samples, license plate is fully visible and all its characters are clear and distinct. • Type 2: Images in these samples are little blurred due to variations in illumination conditions. • Type 3: These samples have little dirt or shadows on or near license plate region. • Type 4: In these samples, another text block apart from license plate is present in the frame. • Type 5: Plate details are present in two rows
  38. 38. License plate localization video demoSource: CitySync single lane PAL demo video,
  39. 39. Type 1 Type 2 Type 3 Type 4 Type 5
  40. 40. Plate Character Image Total images detection recognition type tested accuracy accuracy Type 1 879 99.4% 97.81% Type 2 442 96.3% 96.63% Type 3 153 94.77% 95.69% Type 4 44 100% 99.25% Type 5 11 100% 97%Overall Results 1529 98.1% 97.05%
  41. 41. OpenCV execution MATLAB execution Task time timeEdge detection 0.69s 0.4s Dilation 0.05s 0.03sSegmentation 0.384s 0.3s Total 1.13s 0.731s
  42. 42.  Algorithm fails when license plate is black in color Image with license plate Region growing output fail black in color to detect license plate
  43. 43.  The main objective is to segment the individual characters and extract appropriate features from it to recognize them. A Neural network based approach is then used to train a network using these feature vectors. Our algorithm is currently able to recognize license plate characters with an accuracy of 90-93%.
  44. 44.  The algorithm presented in this paper can be extremely useful in dealing with complex cases, while building a real time license plate detection system. The use of Mexican hat operator helps to improve the performance of edge detection step, when there is only partial contrast between license plate region and region surrounding it. Euler number criterion for a binary image helps to reduce the sensitivity of algorithm to license plate dimensions.
  45. 45.  Preprocessing step using median filter and contrast enhancement ensures performance in case of low resolution and blurred images. Reflection symmetry coefficient along with two stage identification process improves the character recognition performance significantly especially in dealing with complex cases like recognizing 0 and D. Local threshold operation prevents a single brighter or darker region from corrupting the threshold value and thereby improves binarization process.
  46. 46.  Algorithm can be extended to mobile license plate recognition systems. Hardware implementation in FPGA