Successfully reported this slideshow.

Thesis Licenseplaterecognitionincomplexscenes Fulldocument


Published on

1)  Implemented an algorithm based on Mexican hat operator and Euler number of binary image for vehicle license plate localization.
2)  Back propagation artificial neural network classifier has been used to train and test the neural network based on the extracted feature.
3)  A thorough testing of algorithm is performed on a database with varying illumination and plate conditions. Results are encouraging with success rate of 98.1% for license plate localization and 97.05% for character recognition.

Thesis Licenseplaterecognitionincomplexscenes Fulldocument

  1. 1. LICE NSE PLATE RE COG NITIO N IN COr.ifP LEX SCENES BY DHAWAL S WAZALV,TAR Submit t ed in partial fulfillment of t he requirement s for t he degree of Master of Science in Electrical Engineering in t he Gra duate College of t he Illinois Institute of Technology Approved ----("Lf-f/;Io.q---;4/----/;4~--- Chicago, Illinois Decemb er 2011
  2. 2. ACKNOWLEDGMENT I would like to sincerely thank my advisors Dr Erdal Oruklu and Dr JafarSaniie for their constant encouragement and support throughout my thesis work.They always ensured that my research work is heading in proper direction and alwaysguided me in planning each stage of my research work. I would also like to thank DrTricha Anjali for her encouragement and insightful comments during thesis reviewmeeting. I would like to thank my family and friends for their constant support through-out my course. iii
  3. 3. TABLE OF CONTENTS PageACKNOWLEDGEMENT . . . . . . . . . . . . . . . . . . . . . . . . . iiiLIST OF TABLES . . . . . . . . . . . . . . . . . . . . . . . . . . . . viLIST OF FIGURES . . . . . . . . . . . . . . . . . . . . . . . . . . . . viiiLIST OF SYMBOLS . . . . . . . . . . . . . . . . . . . . . . . . . . . ixABSTRACT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xCHAPTER 1. INTRODUCTION . . . . . . . . . . . . . . . . . . . . . . . 1 1.1. Typical license plate recognition system . . . . . . . . . . 2 1.2. Problem statement . . . . . . . . . . . . . . . . . . . . 3 1.3. Thesis structure . . . . . . . . . . . . . . . . . . . . . 3 2. LITERATURE REVIEW AND PROPOSED ARCHITECTURE 5 2.1. License plate localization . . . . . . . . . . . . . . . . . 5 2.2. Character segmentation . . . . . . . . . . . . . . . . . . 7 2.3. Character recognition . . . . . . . . . . . . . . . . . . . 8 2.4. Proposed license plate recognition system . . . . . . . . . 10 3. LICENSE PLATE LOCALIZATION . . . . . . . . . . . . . . 13 3.1. Selection of edge detection method . . . . . . . . . . . . 13 3.2. Morphological dilation operation . . . . . . . . . . . . . 16 3.3. Region growing segmentation . . . . . . . . . . . . . . . 19 3.4. Criterion for license plate detection . . . . . . . . . . . . 22 4. CHARACTER SEGMENTATION . . . . . . . . . . . . . . . 28 4.1. Image preprocessing . . . . . . . . . . . . . . . . . . . 28 4.2. Threshold operation . . . . . . . . . . . . . . . . . . . 32 4.3. Morphological erosion operation . . . . . . . . . . . . . . 35 4.4. Region growing segmentation . . . . . . . . . . . . . . . 35 5. FEATURE EXTRACTION AND CHARACTER RECOGNITION 37 5.1. Feature extraction . . . . . . . . . . . . . . . . . . . . 37 iv
  4. 4. 5.2. Artificial neural networks . . . . . . . . . . . . . . . . . 40 6. IMPLEMENTATION IN MATLAB AND OPENCV . . . . . . 47 6.1. MATLAB implementation . . . . . . . . . . . . . . . . 47 6.2. OpenCV . . . . . . . . . . . . . . . . . . . . . . . . . 47 6.3. Comparison of MATLAB and OpenCV . . . . . . . . . . 48 7. EXPERIMENTAL RESULTS . . . . . . . . . . . . . . . . . 50 7.1. Database for testing of license plate recognition . . . . . . 50 7.2. Results . . . . . . . . . . . . . . . . . . . . . . . . . . 51 8. CONCLUSION AND FUTURE WORK . . . . . . . . . . . . 54 8.1. Conclusion . . . . . . . . . . . . . . . . . . . . . . . . 54 8.2. Future recommendation . . . . . . . . . . . . . . . . . . 55APPENDIX . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56 A. MATLAB CODE FOR LICENSE PLATE RECOGNITION . . . 56 B. OPENCV CODE FOR LICENSE PLATE LOCALIZATION . . 66BIBLIOGRAPHY . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75 v
  5. 5. LIST OF TABLESTable Page 6.1 Comparison of MATLAB and OpenCV execution time for license plate detection of a single frame . . . . . . . . . . . . . . . . . 49 7.1 Summary of license plate localization and detection results . . . . . 51 7.2 Number of frames detected per vehicle . . . . . . . . . . . . . . 52 vi
  6. 6. LIST OF FIGURESFigure Page 1.1 A typical license plate recognition system . . . . . . . . . . . . 2 2.1 Proposed license plate recognition system . . . . . . . . . . . . 11 3.1 (a) Sample image with distinct contrast; (b) Prewitt edge detection; (c) Sobel edge detection . . . . . . . . . . . . . . . . . . . . . 15 3.2 (a) Sample image with partial contrast; (b) Prewitt edge detection; (c) Sobel edge detection . . . . . . . . . . . . . . . . . . . . . 17 3.3 Wavelet Mexican hat . . . . . . . . . . . . . . . . . . . . . . 18 3.4 Mexican hat edge detection output for partial contrast case . . . . 18 3.5 (a) Blur image; (b) Edge detection output; (c) Plate detection with- out dilation; (d) Dilation output; (e) Plate detection with dilation 20 3.6 Pixel connectivity types . . . . . . . . . . . . . . . . . . . . . 21 3.7 Region growing segmentation output for different seed points (a) Sample image; (b) Seed point=200; (c) Seed point=230; (d) Seed point=245 . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 3.8 Region growing segmentation output with different stopping rule (a) Original image ; (b) Stopping rule=25; (c) Stopping rule=55; (d) Stopping rule=75 . . . . . . . . . . . . . . . . . . . . . . 24 3.9 Region growing for license plate detection (a) Output after dilation; (b) Region growing output . . . . . . . . . . . . . . . . . . . 25 3.10 (a) Failure case for Criterion 2; (b) Criterion 3 output for part (a) case . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 4.1 Design flow for character segmentation . . . . . . . . . . . . . . 29 4.2 Various forms of noise in an image (a) Original image; (b) Gaussian Noise; (c) Salt pepper noise; (d) Speckle noise . . . . . . . . . . 31 4.3 (a) Original license plate image; (b) Enhanced version after image preprocessing . . . . . . . . . . . . . . . . . . . . . . . . . . 32 4.4 (a) Sample image; (b) Global thresholding; (c) Local thresholding . 34 4.5 Steps in character segmentation (a) Image with shadow; (b) Local thresholding; (c) Erosion operation; (d) Segmented characters . . . 36 vii
  7. 7. 5.1 Different versions of character ’G’ . . . . . . . . . . . . . . . . 385.2 Basic feed forward model, Perceptron . . . . . . . . . . . . . . 425.3 Three layer Perceptron . . . . . . . . . . . . . . . . . . . . . 437.1 Types of images included in license plate database (a) Type 1; (b) Type 2; (c) Type 3; (d) Type 4; (e) Type 5 . . . . . . . . . . . . 53 viii
  8. 8. LIST OF SYMBOLS Symbol Definition σ Standard deviation E Euler number of binary imageXcoef f icient Reflection coefficient about X axisYcoef f icient Reflection coefficient about Y axis ix
  9. 9. ABSTRACT License plate recognition is considered to be one of the fastest growing tech-nologies in the field of surveillance and control. In this project, we present a newdesign flow for robust license plate localization and recognition. The algorithm con-sists of three stages: i) license plate localization ii) character segmentation and iii)feature extraction and character recognition. The algorithm uses Mexican hat opera-tor for edge detection and Euler number of a binary image for identifying the licenseplate region. A pre-processing step using median filter and contrast enhancement isemployed to improve the character segmentation performance in case of low resolutionand blur images. A unique feature vector comprised of region properties, projectiondata and reflection symmetry coefficient has been proposed. Back propagation artifi-cial neural network classifier has been used to train and test the neural network basedon the extracted feature. A thorough testing of algorithm is performed on a databasewith varying test cases in terms of illumination and different plate conditions. Theresults are encouraging with success rate of 98.10% for license plate localization and97.05% for character recognition. x
  10. 10. 1 CHAPTER 1 INTRODUCTION License plate recognition is one of the most rapidly evolving technologies inthe field of video based surveillance and control. It is sometimes also referred to asAutomatic number plate recognition (ANPR) or automatic vehicle detection system.As the name implies, it first detects the license plate from the input frame andrecognizes the vehicle as well its owner based on its license plate information. Sinceits first use in 1976 by UK Police, this technology has gone through a lot of changesand is now being extensively used in law enforcement field. One of the key applications has been in access control for gated communitiesand other such restricted access locations. In such places, license plate information ofall the authorized vehicles is stored in a database and only these vehicles are allowedto enter the premises. This can be extremely useful in safeguarding intelligence agen-cies and other such key government buildings from any external threat. Automatedtraffic management is another field where license plate recognition has contributedimmensely. It is widely used for toll collection, speed control and accident scenereconstruction. The success for fixed camera applications encouraged authorities to use thistechnology for mobile camera based surveillance. The use for mobile applications hadits own challenges especially in terms of power efficiency since vehicle battery was usedas a power source. Another challenge was processor speed as all the systems needto be real time. In recent years, the semiconductor technology has grown immenselywhich has contributed in improving power performance of electronic equipments aswell as reducing its size. This has helped in developing portable surveillance systemswhich can be easily mounted on vehicle top. This has proven to be an important toolfor police officials in solving traditional criminal investigations. Recently, a Maryland
  11. 11. 2state police officer traveling along I-70 nabbed alleged drug dealer with the help oflicense plate technology.1.1 Typical license plate recognition system A generic representation of license plate recognition system is shown in Figure1.1. It mainly has following key stages. Op cal Character License Plate Image Frame Recogni on Informa on Acquisi on Processing Engine Database Figure 1.1. A typical license plate recognition system1.1.1 Image acquisition. Surveillance cameras are used to capture real time videoor images of the front or rear of the vehicle with its license plate. Nowadays, mostlyvideos are captured and then a frame grabber is used as an interface to make therequired frame available for processing to the software. Cameras used are speciallydesigned which have important features like faster frame rates, weatherproof andbetter power efficiency. For better performance in low vision or night times, infra-red (IR) illumination is used. A lot of companies constantly work on developing highend intelligent cameras for better image acquisition. For example PIPS technology,one of the key vendors of license plate recognition systems, uses a dual lens camerawith triple flash technology to produce highest quality image.1.1.2 Frame processing. After acquiring the input frame of vehicle with its licenseplate, a series of image enhancement and processing steps are performed to detectthe license plate region. The complexity of these steps depends upon the applicationrequirements. Once we obtain the license plate, we need to segment its individualcharacters which then can be sent to optical character recognition engine.1.1.3 Optical character recognition engine. For recognizing the segmented
  12. 12. 3characters, we extract some of its key features and use statistical classifiers to differ-entiate and therein recognize the character information.1.1.4 License plate database. This is also an important part of overall systemwherein we store the output obtained from OCR engine in a database. This databaseinformation then can be used by police officials or other law enforcement authoritiesto use for various applications like stolen vehicle identification, speed checks etc.1.2 Problem statement Although many license plate recognition systems already are in market anda lot of research has been done in this field, all these systems perform efficientlyonly in a controlled environment. For example, in case of license plate detection,most of the systems claim to have an accuracy of above 95 %. However, the actualsystem performance was not as per the expectations when dealing with practical caseslike presence of shadow/dirt on license plates, sensitivity to license plate dimensionsand many such other factors. Our emphasis in this project has been to come upwith solutions which will help to improve performance of license plate detection andrecognition in dealing with real time practical scenarios. The performance analysis of license plate recognition systems is also an issuesince there is no common benchmark. In this thesis, system performance is evaluatedon a real time demo video taken from City Sync license plate recognition camera.This video has all the practical scenarios with varied license plate sizes, night/twilightconditions and dirty/distorted license plate images. Our focus has been on dealingwith performance issues related to fixed camera applications; however the algorithmscan also be extended for mobile camera surveillance systems with some modifications.1.3 Thesis structure Chapter 2 gives an insight on the previous research work done in the license
  13. 13. 4plate recognition field. In Chapter 3, a design flow for robust license plate recognitionhas been presented and performance of license plate localization for real time casesis analyzed. The character segmentation aspect and all the associated steps with itare explained in Chapter 4. The challenging issue of feature extraction and characterrecognition is explored in Chapter 5. Software implementation part in MATLABand OpenCV is explained in Chapter 6. Performance evaluation of license platelocalization and recognition on test database is explained in Chapter 7. A detailedsummary with some future recommendations is given in Chapter 8. Finally, MATLABand OpenCV codes are provided in Appendix A and B for reference purpose.
  14. 14. 5 CHAPTER 2 LITERATURE REVIEW AND PROPOSED ARCHITECTURE License plate recognition can be seen as a three stage process: i) Licenseplate localization; ii) Character segmentation; and finally iii) Feature extraction andcharacter recognition. The first two stages are mostly based on image processing andenhancement techniques, while the final stage requires use of statistical classifiers.A detailed survey of all previous implementation based on issues like performance,execution time, and the platform used has been done in [20].2.1 License plate localization License plate localization is very crucial step in that the overall system ac-curacy depends on how accurately we can detect the exact license plate location.The input can be in the form of still images or video frames from surveillance cam-eras. Based on the type of processing method used, it can be classified in followingcategories.2.1.1 Binary image processing. All these methods mostly use edge informationand mathematical morphological operations to detect the license plate region. Forinstance, in [16] a morphological operator called ‘top-hat’ is used to locate smallobjects of significantly different brightness. This method had a success rate of about80%, however it was sensitive to license plate dimensions. Another method based onedge statistics and morphological operation was proposed in [8]. This method hadan accuracy of about 99.6 % but requires prior information about candidate regionbeing in a specific position. Connected component analysis (CCA) which uses pixelconnectivity information for labeling the image is one of the widely used techniques inbinary image processing. It scans the image data and depending on whether we use4-connected or 8-connected methods it labels the entire image pixel by pixel. Spatial
  15. 15. 6measurements like area, perimeter, and aspect ratio are used for extracting the licenseplate region. These methods are relatively fast in execution as the processing is doneat binary level. However, sensitivity of algorithm to unwanted edges is an issue.2.1.2 Gray level processing. In these methods, processing is done at graylevel and the algorithm uses the local contrast information to differentiate betweenpossible regions in the image. For example, in [6] the algorithm continuously scansthe image and any region which has contrast changes on a scale of 15 pixels or moreis detected as license plate region. The value 15 was selected based on experimentalanalysis and depends on distance between vehicle and the camera. Instead of fullimage scanning, a method based on partial image analysis was proposed in [1] . Acertain threshold value is assumed and if the number of edges is greater that value, itis considered license plate region. This method is comparatively faster as it requiresscanning of few rows only. Another method based on block based gray level processingis presented in [13] . In this blocks with high edge magnitude or high edge varianceare identified as possible license plate region.2.1.3 Color processing. These methods use color information of the licenseplate characters and the plate background for differentiating it from the rest of theimage. One such method based on using unique color combination of license plateregion was proposed in [2]. Some methods make use of fuzzy set theory for locatinglicense plate. In [19], segmentation method incorporates the fuzzy sets of yellownessand texture to identify license plate region. The accuracy of this method was about75.4%. A Korean license plate recognition system [3], uses an edge detection techniquewhich responds to only three types of edges, black-white, red-white, and green-white.It then transforms RGB model into hue-saturation-intensity (HIS) model and therespective H, S and I maps are generated. After integrating these individual maps ina single map M, region with locally maximal values is identified as probable license
  16. 16. 7plate region. This method was fairly successful with 97.9% detection accuracy. Colorhistogram information with Gaussian weighted algorithm [27] can also be used tolocate license plate region. All these color processing methods are relatively moreaccurate as compared to standard gray level or binary level implementations. Theissue lies in the processing speed as the color data is huge and so real time processingis difficult. Another important aspect is the color information is not stable in differentlighting conditions. This affects robustness of the system since most of these systemsperform well only in the originally defined environment.2.2 Character segmentation After detecting the license plate region, the next important task is to obtainclean and isolated individual characters. The method used depends on the applicationrequirements and sometimes combination of two or more techniques is used to getthe desired results.2.2.1 Horizontal and vertical projection data based techniques. The basicidea is to add up image columns or rows and obtain a vector whose minimum valuesallows us to segment characters. In [28], a method using vertical projection data waspresented, however this technique requires prior knowledge of license plate size as wellas the number of characters in it. This sensitivity to license plate dimensions wasreduced to an extent by employing region growing segmentation method along withpixel connectivity information [15].2.2.2 Binary algorithms and morphology based techniques. CCA usedin conjunction with binary object measurements such as height, width, area andorientation can also effectively segment the license plate characters. In order totackle complex scenes like degraded plates, a morphology based adaptive approachwas suggested in [21]. Building on this technique, a novel approach of thickening and
  17. 17. 8pruning algorithm to identify noise region was proposed in [7]. All these methods hadprior knowledge of the number of characters in the license plate.2.2.3 Contour based techniques. In [11], an algorithm based on the contourtracking principle called ‘bug following’ was implemented for character segmentation.A shape driven active contour model along with calculation of a special fast marchingalgorithm was proposed in [24]. Gradient, curvature and other such shape descriptionwas embedded with these contour models to perform accurate character segmentation.2.2.4 Gray level histogram based techniques. Threshold operation is a keyaspect of character segmentation algorithm. In [6], intensity-gradient scattergramsor histogram valley technique was employed to determine threshold value. Similarmethod was proposed in [1], where the threshold value was calculated a formulathat employs the entropy values of background and foreground or by cumulativedistribution functions.2.2.5 Local/Adaptive thresholding and transformations. Binarization ofa gray level image is a key aspect in character segmentation and researchers havebeen using both local as well as global thresholding as a means of achieving it. Theimage can be divided in several blocks of mxn size and a separate threshold valuecan be obtained based on certain image properties. One such method based on Otsubinarization technique is implemented in [18].2.3 Character recognition After isolating the individual characters, it is important we extract the uniquefeatures of it and develop a mechanism to recognize them. For this, normally we gofor either statistical/hybrid classifiers or artificial neural networks.2.3.1 Statistical/Hybrid classifiers. Statistical classification is a machine learn-ing problem where we try to specify a relationship between the explanatory variables
  18. 18. 9and the dependent variable. A hidden Markov model for each character was employedin [15] with a complex procedure for preprocessing and parametrization of the regionof interest (ROI). This method performed well with a predefined effective distancebetween the vehicle and camera. The authors in [12] then proposed a system basedon support vector machine model. Four separate models to recognize upper numerals,upper characters, lower characters and lower numerals were designed and success ratewas about 97.2%. Instead of using single statistical model for classification, researchers cameup with hybrid models. Horizontal and vertical projection data, zoning density andcontour features were combined to form a feature vector [4] with an accuracy of 95.7%.In this two stage hybrid recognition system, firstly four statistical sub-classifiers( SC1,SC2, SC3 and SC4) separately recognize the input character and then results arecombined using Bayes method. If the characters obtained in first stage output aredifferent, a second stage is initiated in which structural features are fed to a decisionclassifier and final decision is made. Recently, a method based on the concept ofstroke direction and elastic mesh [26] was proposed with an accuracy in the range of97.8- 99.5%.2.3.2 Artificial neural networks (ANN). Artificial neural network is non-linearadaptive statistical data models which can be trained by using already know infor-mation and then can be used to classify unknown test data. The model type, numberof hidden layers and the training method determine the overall performance of ANN.In [19], A multi layered neural network that contains 24 input, 15 hidden, and 36output neurons was trained to recognize 36 Latin alphabet characters. This methodwas on binary input data and was reported to have accuracy of 98.5%. In order todeal with problematic cases (I/1, B/8 and O/0) [23], training data included had moresuch cases so that ANN can be effectively trained.
  19. 19. 10 Adaptive resonance theory (ART), an unsupervised learning technique, whichdynamically changes the number of hidden layers based on convergence requirements,was presented in [11]. To further improve the system performance, a unique methodbased on Kohonen’s self organized feature maps was utilized in [3] and this workedextremely well for difficult cases like bent/deformed license plates. Probabilistic neu-ral networks (PNN) were first used for license plate recognition in [20]. The mainadvantage was these models required only one time training, which saved a lot ofexecution time. Two separate models for alphabet and number recognition were de-signed and were reported to have success rate of about 99.5 % in general cases andof about 90% in complex scenes like tilted/deformed license plates.2.4 Proposed license plate recognition system A significant research has already been done, however the system should workeffectively to different types of plate conditions, weather conditions and illuminationeffects. Another important aspect is most of previous implementations are eitherregion or country specific. Therefore, in order to use these systems at different lo-cation, redesign of whole system is required at times. Ideally, while designing itselfcare should be taken that algorithm remains generic and can be easily used at anylocation. A new design flow for license plate recognition in complex scenes is shownin Figure 2.1. The system can be categorized into three key stages i) license plate localization;ii) character segmentation and iii) character recognition. The main objective of licenseplate localization stage is to come up with a near perfect algorithm which will addressto some of the shortcomings in previous implementations like i) presence of anothertext block in an image, ii) existence of partial contrast between the license plateregion and surrounding area, iii) license plate region located anywhere in the frame. Incharacter recognition stage, the emphasis is on obtaining clean and isolated characters.
  20. 20. 11 Creating and training a Input frame Median filtering neural network Edge detection using Contrast enhancement Mexican hat operator Probable character string Local thresholding Morphological dilation operation Subtracting morphological erosion output No Is character S, 5, 0 or D Region growing segmentation Region growing segmentation YesNo License Checking for specific plate Criterion 3 using Euler property of S, 5, 0 or details No D number of binary image Probable character region Yes Yes Correction in obtained Segmented license string plate region Segmented characters License plate Character Character detection segmentation recognition Figure 2.1. Proposed license plate recognition system
  21. 21. 12Practical issues like presence of shadow or dirt on license plate, blur images need tobe considered. In character recognition, the main focus is on obtaining unique featurevector as well as on training the back propagation artificial neural network classifier.
  22. 22. 13 CHAPTER 3 LICENSE PLATE LOCALIZATION Plate localization is the first stage for any license plate recognition system.It is an important step as the success rate at this level will decide the performanceof rest of the system. The key issue is to achieve a robust system which will workefficiently in general as well as difficult scenarios like varying illumination and licenseplate conditions. The main steps involved are edge detection, morphological dilationoperation and region growing segmentation.3.1 Selection of edge detection method An edge can be defined as a set of pixels which define a boundary betweentwo regions. Edge detection is one of the widely used techniques in detecting mean-ingful discontinuities in an image. A detailed study of edge detection techniques hasbeen made in [17], where a comparison based on well known measuring metrics hasbeen made. The performance of any edge detection technique depends on lightingconditions, the presence of similar intensities objects and noise content in an image.Manual adjustment of masks and threshold value for edge consideration has to bedone in order to run edge detector with different set of data. Edge detection can beperformed at gray level as well color level. Gray level is faster as it has to operateonly on one channel as compared to three channels in color processing. However,color edge detection provides more information about the scene since it can detectedges in region with high color variation but low intensity variations. Some of thewidely used edge detection techniques at gray level are Canny edge detector, Mexicanhat operator based edge detection and also Boolean function based edge detection.Color edge detection using Euclidean distance and vector angle is also used in somecases.
  23. 23. 14 In case of license plate detection, edge detection has an important role sincewe need to extract clean and continuous edges to reduce any possibility of false licenseplate detection. The performance of edge detectors need to be analyzed by consideringtwo important practical cases. 1. Clear and distinct contrast between the license plate and surrounding region. In such cases, simple gradient based methods like Prewitt, Sobel operators can be used to perform the edge detection task. These operators are used to obtain gradient components along x and y direction. For Prewitt operator, the gradient components are Gx = (z7 + z8 + z9 ) − (z1 + z2 + z3 )Gy = (z3 + z6 + z9 ) − (z1 + z4 + z7 ) (3.1) In case of Sobel operator, a weight value of 2 is used to achieve better smoothing and noise suppression characteristics. The gradient components in this case are Gx = (z7 + 2z8 + z9 ) − (z1 + 2z2 + z3 )Gy = (z3 + 2z6 + z9 ) − (z1 + 2z4 + z7 ) (3.2) The final approximation is then obtained by performing ▽ ≈ |Gx| + |Gy| (3.3) Figure 3.1(a) shows a sample image where we can visually identify the contrast difference between license plate and region around it. Figure 3.1(b) and Figure 3.1(c) shows the result of application of Prewitt and Sobel mask on the sample image respectively. It can be clearly seen that license plate edges are properly separated from surrounding region and no further modification is required. 2. Partial contrast between the license plate and surrounding region. Consider a case shown in Figure 3.2(a), where the contrast difference is less as compared to case shown in Figure 3.1(a). In this situation, the performance of general
  24. 24. 15 (a) (b) (c)Figure 3.1. (a) Sample image with distinct contrast; (b) Prewitt edge detection; (c) Sobel edge detection
  25. 25. 16 gradient operators like Prewitt and Sobel is not as desired, see Figure 3.2(b) and Figure 3.2(c). It can be seen the license plate edges are still connected to region around it; this can cause license plate detection failure at later stage. In order to deal with such cases, advanced operators like Mexican hat operatorhave to be used. It is called so because of its similarity in appearance to a Mexicanhat, see Figure 3.3. Mexican hat operator first performs smoothing operation andthen extracts edges. This function is also called as Laplacian of Gaussian (LoG) andcan be mathematically expressed [7] as r 2 − σ 2 −r2 2 ▽2 h(r) = − e 2σ (3.4) σ4 Masks of different sizes 3x3, 5x5 and 9x9 were tested and the best results wereseen for 9x9 operator. Figure 3.4 shows the edge detection output using Mexican hat operator forFigure 3.2(a). As compared to Prewitt and Sobel operator output, the edge detectionoutput using Mexican hat seems to have proper and continuous edges. Therefore, wehave selected Mexican hat operator for edge detection in our implementation.3.2 Morphological dilation operation In general, mathematical morphological operations are used to extract com-ponents which describe a particular region, such as boundaries, skeletons, and theconvex hull. Dilation is morphological operation which can be considered as a localmaximum operator used to add pixel information to boundaries in an image. Dilationoperation can be mathematically expressed [7] as X ⊕ B ≡ {X : Bx ∩ X = φ} (3.5)where X is the object and B is the structuring element.
  26. 26. 17 (a) (b) (c)Figure 3.2. (a) Sample image with partial contrast; (b) Prewitt edge detection; (c) Sobel edge detection
  27. 27. 18 Figure 3.3. Wavelet Mexican hatFigure 3.4. Mexican hat edge detection output for partial contrast case
  28. 28. 19 If the license plate image is blurred or distorted, edge detection may not besufficient to obtain proper edges. Figure 3.5(a) shows one such case with a blur licenseplate image. The Mexican hat operator based edge detection output can be seen inFigure 3.5(b). The discontinuity in license plate edges result in improper detectionoutput, see Figure 3.5(c). In order to prevent such false detections and make theseedges thick and continuous, a dilation operation is performed after the edge detectionstage. The improved results edge detection and the corresponding detection outputcan be seen in Figure 3.5(d) and Figure 3.5(e) respectively. The size and nature of structuring element is important since smaller size cannegate the desired effect and larger size can cause the license plate regions to beconnected to the surrounding area. For our algorithm, best results were achievedwith 3x3 ones matrix as a structuring element.3.3 Region growing segmentation In computer vision, segmentation is used to partition the digital image intodistinct segments by using characteristic properties of an image like color, intensity ortexture. For example, in medical imaging applications, segmentation can be used tolocate tumors, tissues which then can be used for diagnosis purpose. Segmentation iswidely used to locate objects and boundaries in a digital image. Image segmentationcan be classified based on two basic properties of image: discontinuity and similarity[7]. In first type, abrupt changes in an image like edges are first detected and thena edge linking procedure is used to locate meaningful objects in the image. Houghtransform is an example of this category and can be used to segment regular curveslike lines, circles, ellipses. The second category divides the image into different subregions that are similar based on certain predefined criterion. Thresholding, regiongrowing and region splitting are some of the techniques which belong to this category.
  29. 29. 20 (a) (b) (c) (d) (e)Figure 3.5. (a) Blur image; (b) Edge detection output; (c) Plate detection without dilation; (d) Dilation output; (e) Plate detection with dilation
  30. 30. 21 (a) 4 pixel connectivity (b) 8 pixel connectivity Figure 3.6. Pixel connectivity types For detecting a license plate, first we have to obtain all probable license plateregions in an image. The output of dilation operation is a binary image and so itwill be easy to grow regions based on pixel connectivity information. Region growingsegmentation is ideally suited for this purpose. In region growing, we group pixelsbased on some predefined pixel connectivity information to form sub regions. De-pending upon the accuracy requirements, we can either go for 4-pixel connectivityFigure 3.6(a) or 8-pixel connectivity Figure 3.6(b). Although ideally we would like touse 8-pixel connectivity every time, on occasions it can be computationally extensive. The performance of region growing algorithm depends on the starting point iethe seed point and the stopping rule selected for the implementation.3.3.1 Seed point selection. This is an important aspect of region growingespecially when only one starting point is defined for the implementation. Considera sample image of fluid scatter (splash). For this image, we have to segment thesplash part from the rest of image using region algorithm. Figure 3.7 shows originalimage and region growing segmentation output by using different seed points. It
  31. 31. 22can be visually confirmed that seed point=230 gives the best segmentation results ascompared to seed point=200 and Stopping rule. In region growing, we need to define a stopping rule whichwill terminate the region growing process. The selection of rule depends on theimage characteristics like size, and likeliness of candidate pixel. The stopping rulewill determine the execution time required for region growing process. For instance,consider a medical image, where we need to segment the tumor part from rest of theimage. Suppose, the stopping criterion used is maximum number of pixels in theimage, M. Figure 3.8 shows the original image with the results for M=25, 55 and 75.Based on the application requirements, a tradeoff between accuracy and computationtime is made and suitable stopping rule is selected. For our implementation, we have to find all possible license plate regions.This means region growing has to be performed on the entire image. For every brightpixel, its neighboring four pixel connectivity information was checked and region wasgrown based on this information. In order to avoid region growing for noise part inthe image, we defined a threshold which decides whether the region is object partor noise content. A lot of analysis was done and it was seen that regions with arealess than 75 pixels need not be considered. In order to increase the algorithm speed,sub-sampling was used, where operation was performed for every two pixels, insteadof performing for each and every pixel in the image. The final output will be severalsub regions labeled based on 4-pixel connectivity information, see Figure Criterion for license plate detection Once we obtain all probable regions in an image after performing region grow-ing segmentation, we need to define a criterion to select accurately the license platepart from it. Following different criteria can be used for this purpose.
  32. 32. 23 (a) (b) (c) (d)Figure 3.7. Region growing segmentation output for different seed points (a) Sample image; (b) Seed point=200; (c) Seed point=230; (d) Seed point=245
  33. 33. 24 (a) (b) (c) (d)Figure 3.8. Region growing segmentation output with different stopping rule (a) Original image ; (b) Stopping rule=25; (c) Stopping rule=55; (d) Stopping rule=75
  34. 34. 25 (a) (b)Figure 3.9. Region growing for license plate detection (a) Output after dilation; (b) Region growing output3.4.1 Criterion 1 License plate dimensions. Most of the countries/statesgenerally fix license plate dimensions for all vehicles. This information can be usedto separate license plate region from rest of the probable segments. It is very easyto implement and computationally less extensive. However, by using this criterion,we make algorithm sensitive to license plate dimensions, which depend on the dis-tance between the vehicle and camera. In real time scenarios, this distance can varydepending upon the application used. Practically it is extremely important that al-gorithm works at least on 15% tolerance range of license plate dimensions. Therefore,this criterion does not meet performance requirements.3.4.2 Criterion 2 Euler number of binary image. For a digital image, Eulernumber helps to find the structure information of object without relating to thegeometric shape of object [14]. It is defined as total number of objects in an image’C’ minus the number of holes in those objects ’H’. E =C−H (3.6)
  35. 35. 26 Characters used in license plate region are alphanumeric (A-Z and 0-9). If wecarefully observe these characters in an image, this can be seen as closed curves likein 0, 9, 8, P etc. Thus clearly license plate region will have more closed curves thanin any other part of an image. Euler number of an image can be used to distinguishthe different regions obtained after the region growing output. Experimentally, itwas found that in alphanumeric regions, value of Euler number is zero or negative,while for other regions it has a positive value. Using Euler number criterion willovercome the sensitivity issue encountered in using Criterion 1. This criterion worksaccurately and efficiently for majority of test cases. The only failure case was seenwhen the vehicle image has alphanumeric characters in more than one region, seeFigure 3.10(a).3.4.3 Criterion 3 Combination of Criterion 1 and Criterion 2. In thisimplementation, we have used a combined approach of Euler number criterion aswell as license plate dimensions to overcome problem raised in case of Criterion 2.Instead of predefining exact license plate dimensions, we specify a range of acceptableminimum and maximum license plate sizes. This meets the real time performancerequirements of algorithm working for cases with 15 % tolerance over exact licenseplate dimensions. Figure 3.10(b) shows the application of this criterion on failurecase of criterion 2. It can be seen that license plate is accurately detected using thiscriterion.
  36. 36. 27 (a) (b)Figure 3.10. (a) Failure case for Criterion 2; (b) Criterion 3 output for part (a) case
  37. 37. 28 CHAPTER 4 CHARACTER SEGMENTATION After locating the license plate, the next important step is to segment eachcharacter individually, avoiding any possibility of joint characters. The main chal-lenge is ensuring performance in complex scenes like blur image, tilted license plateand presence of shadow/dirt on the license plate region. Figure 4.1 shows all thenecessary steps that need to be performed in order to ensure accurate character seg-mentation. Firstly, the license plate image obtained after localization stage needs tobe preprocessed for better accuracy. Threshold operation is then performed in orderto convert the gray level image into binary format for further processing. Morpholog-ical erosion operation is done to remove the noise content and finally region growingsegmentation is used to segment the character region.4.1 Image preprocessing Even though we assume that the image acquired during acquisition stage isperfect, it may not be the case always. The image may have some noise content,which can cause performance issue in later analysis.4.1.1 Image noise. The quality of imaging sensors used during the image acqui-sition and the variations in environment conditions decides the noise content in theacquired image. In Figure 4.2, some of the common forms of noise that can be seenin a digital image are shown on image of Lena, most widely used test image for imageprocessing algorithms. Gaussian Noise. Figure 4.2(b) shows Lena image with Gaussian noise init. In this case, each pixel will be distorted by small amount compared to the originalimage. If we plot a histogram for amount of distortion of a pixel with respect to itsfrequency content, it can be seen that it follows a Gaussian distribution.
  38. 38. 29 Median filtering Contrast enhancement Local thresholding Subtracting morphological erosion output Region growing segmentation No Probable character region Yes Segmented charactersFigure 4.1. Design flow for character segmentation
  39. 39. 304.1.1.2 Salt and pepper noise. In this case, we can see dark and white dots onthe image everywhere and so it is called as salt and pepper noise, see Figure 4.2(c).This is usually caused because of dust flecks inside the camera or faulty image sensingelements. Speckle noise. Speckle noise is a granular noise usually seen in the formof increased mean gray level of a local area, see Figure 4.2(d). It is usually present incases where we acquire images from multiple sources and combine all the images inone.4.1.2 Median filtering. In order to improve character segmentation performance,we need to first remove the noise content from the image and reconstruct a betterversion of original image. While doing so, we need to make sure that we preserveimportant characteristics of the image like boundaries, edges. Median filter is a non-linear spatial filter and is used widely for image denoising [25], [5]. In median filtering,we replace the original pixel data by median of pixels contained in a prescribedwindow. The median filter output for 3x3 window can be obtained by performingfollowing steps. 1. Compute the mean value y0 of the pixels in the filter window. 2. Subtract y0 from each pixel within the filter window. 3. Find the median ym of the nine modified points. ˆ 4. The final output is X = ym + y0 . In order to ensure that the edges of the characters are preserved, analysiswas done to determine the optimal median filter size. In this case, best results wereobtained for 3x3 filter window.
  40. 40. 31 (a) (b) (c) (d)Figure 4.2. Various forms of noise in an image (a) Original image; (b) Gaussian Noise; (c) Salt pepper noise; (d) Speckle noise
  41. 41. 32 (a) (b)Figure 4.3. (a) Original license plate image; (b) Enhanced version after image pre- processing4.1.3 Contrast enhancement. If the image acquired is blurred, it will be difficultto process it further, since most of its character information is not properly visible.Therefore, we have to locally enhance it in smaller regions while preserving its globalcontrast. This can be done by increasing the dynamic range for pixels in certainintensity range. After observing the pixel data in the license plate region, it wasfound that the character information for most of the images in database is usuallyin the intensity range of 0-0.4 on a scale of 1. In order to obtain further contrastenhancement, we increased the dynamic range for pixels in this (0-0.4) intensity range. Figure 4.3 shows one sample license plate image and enhanced output afterpreprocessing step. The improved contrast enhancement results can be seen in Figure4.3(b).4.2 Threshold operation Thresholding is a segmentation process used to separate operate object andbackground information. It is also used for converting grayscale image into a binary
  42. 42. 33image. There are basically two types of threshold operation.4.2.1 Global thresholding. In this case, we select one single threshold value forentire image. Mathematically it can be represented as   1, if f (x, y) ≥ 0     g(x, y) = (4.1)   0, otherwise.    where f(x,y) is the original grayscale image, g(x,y) is the threshold outputimage and T is the threshold value obtained. This operation is easy to implementand also less time consuming. However, it is sensitive to illumination conditions andfails in cases if there is a brighter or darker region other than object. One such case isshown in Figure 4.4(b), where we can see character ‘O’ and ‘2’ are connected becauseof unwanted dark region in original grayscale image, see Figure 4.4(a).4.2.2 Local thresholding. In the local threshold operation, we divide the grayscaleimage into several parts by using a windowing operation and then perform the thresh-old operation separately for each case. Since, we perform threshold operation foreach region separately; a single brighter or darker region cannot corrupt the thresh-old value. Local thresholding ensures that in most cases, we get clear and separatedcharacters. In Figure 4.4(c), we can see all the characters are separated compared toFigure 4.4(b), where global thresholding was performed. The performance of localthreshold operation will vary depending on the selected window size. Experimentally,best results were obtained when the window size was chosen approximately equal tothe general individual character size in the license plate image. For our algorithm, the threshold value for local region is obtained using Otsu’smethod for gray level histogram [22]. In this method, optimal threshold value isobtained such that the intra class variance between the two classes(foreground and
  43. 43. 34 (a) (b) (c)Figure 4.4. (a) Sample image; (b) Global thresholding; (c) Local thresholding
  44. 44. 35background) is minimal.4.3 Morphological erosion operation In certain situations if some dirt or shadow is present on the license plateregion, the output of local threshold operation may not be sufficient. For example inFigure 4.5(b), it can be seen even after the local threshold operation some amount ofunwanted noise is still present near character ‘4’ and ‘X’. This noise can be removedby performing morphological erosion operation see Figure 4.5(c). Mathematically,erosion operation can be represented [7] as X ⊖ B ≡ {X|Bx ⊆ X} (4.2) where X is the object and B is the structuring element. The size and na-ture of structuring element should be selected with an aim of keeping the characterinformation intact and only removing unwanted parts in the binary image.4.4 Region growing segmentation The pixel connectivity information can be used to separate the characters.Region growing segmentation along with four pixel connectivity approach is used inthis case. This is the same algorithm which was used for license plate detection pur-pose. Figure 4.5(d) shows the separated characters after performing region growingsegmentation.
  45. 45. 36 (a) (b) (c) (d)Figure 4.5. Steps in character segmentation (a) Image with shadow; (b) Local thresh- olding; (c) Erosion operation; (d) Segmented characters
  46. 46. 37 CHAPTER 5 FEATURE EXTRACTION AND CHARACTER RECOGNITION Character recognition, often referred in general as optical character recogni-tion, has been a major revolution in the electronics world. It is widely used forscanning handwritten notes, online exams evaluation and techniques such as machinetranslation, text to speech conversion and text mining. The accuracy requirementsdepend on the intended application and in our case it is used to recognize the seg-mented characters. It can be considered as a two stage process; firstly we have toextract key features associated with the characters and secondly, we need to define amechanism to recognize these characters based on the extracted feature information.For the latter purpose, we use statistical classifiers in general and in our case we haveused artificial neural networks.5.1 Feature extraction Feature extraction is a process in which we define and characterize the availabledata by a set of features called as feature vector. Following are two important pointswe need to consider while selecting feature vector for character segmentation. 1. The robustness of character recognition algorithm depends on how well it can recognize different versions of a single character. In Figure 5.1, all the images are for same character G of size 24x10, however it varies in terms of spatial information, noise content and thickness of edges. This is a common case espe- cially when dealing with different illumination conditions. The feature vector should be such that algorithm can still accurately recognize all these varied representations of a single character. 2. Sometimes it is difficult to distinguish between characters like D and 0 or 5 and S because of the similarity in their appearance as well as spatial information. It
  47. 47. 38 Figure 5.1. Different versions of character ’G’ is critical that our feature vector contain certain characteristics which will help us differentiate between these complex cases.5.1.1 Selection of feature vector . In order to deal with these challenges,our algorithm uses a combination of spatial information as well region properties.Following is the description of these key features. 1. Feature 1: Analysis of variation of all region properties with respect to the characters was performed and from them six properties were selected. Following are the image properties selected in feature vector. (a) Area: This gives the actual number of pixels in the region. (b) Perimeter: Perimeter is calculated as the distance between each adjoining pair of pixels around the border of the region. (c) Minor-major axis length: These specify the length of pixels associated with the minor and major axis of the ellipse that has the same normalized second central moments as the region. (d) Orientation: Orientation is the angle between the x-axis and the major axis of the ellipse that has same secondary moments as that of region. Orientation value is the angle between the horizontal dotted line and the major axis.
  48. 48. 39 (e) Euler number of binary image: This gives a value which gives the number of objects minus the number of holes present in the object [14].2. Feature 2. The second part of feature vector includes projection information about X and Y axes. Projection is mapping of binary image into one dimensional array. A horizontal projection is sum of white pixels (background being black) projected onto vertical axis x, while vertical projection is the sum of white pixels onto horizontal axis y. Projection data forms major part of feature vector and is unique for each character.3. Feature 3: Some of the characters like 0, M, N are symmetric about X-Y axes. This information can be used to distinguish different characters especially 0 and D. This symmetry information can be quantified by a coefficient called as reflection symmetry coefficient. For 2D images, a way of calculating this coefficient is proposed in [20]. In this implementation for a character of size 24x10, we calculate measure of symmetry about X and Y axis in following way. (a) Consider Cin as the original character matrix where information is stored in binary format. The center of matrix is considered as the origin. Cx and Cy are the transformed matrix obtained after flipping about the X and Y axes respectively. (b) Calculate the area of original matrix Cin . (c) Determine the union of transformed matrixes Cx and Cy with original matrix Cin . (d) Calculate the areas of regions (Cin ∪ Cx ) and (Cin ∪ Cy ). (e) The reflection coefficient can be determined as Area(Cin ) Area(Cin ) Xcoef f icient = ; Ycoef f icient = (5.1) Area(Cin ∪ Cx ) Area(Cin ∪ Cy )
  49. 49. 40 Feature 3 has significantly improved algorithm’s performance in dealing withcomplex cases like D and 0. Experimentally it was found that for 0, the value ofXcoef f icient and Ycoef f icient are approximately similar. On the other hand, since D hassymmetry only along X axis, value of Xcoef f icient is larger compared to Ycoef f icient .5.2 Artificial neural networks5.2.1 Basis concept and applications. Humans have the capability to solvecomplex perceptional problems at high speed. This is mainly possible because ofbiological neural networks which give them ability to learn adaptively, inherent con-textual information and massive parallelism capability. Inspired by these performancecharacteristics of biological neuron, a mathematical model called artificial neural net-work based on interconnection of artificial neurons, which can be trained by flow ofexternal and internal information was developed. Since its first use, ANN models hasproven to be powerful tool for solving complex engineering problems [9] like 1. Pattern classification. In these problems, we first define a feature vector which represents the input pattern and then finally we use this feature information to assign it to a set of predefined output classes. Character recognition, speech recognition, blood cell classification are some examples in this category. 2. Clustering/categorization. Clustering refers to grouping of input data based on some predefined similarity criterion. This falls in an unsupervised leaning paradigm as there is not data defined for training purposes. Common examples are data mining, data compression. 3. Function approximation. Function approximation is used to represent target data with set of well defined available functions. Interpolation, extrapolation, regression analysis, and curve fitting can be considered to be function approxi- mation problems.
  50. 50. 41 4. Prediction/forecasting. In this, for the given set of data at a particular time instant we need to predict sample value at future time interval. Machine learn- ing techniques applied in stock market prediction is well known example in this category.5.2.2 Classification of artificial neural networks. Based on the connectionpattern in neural network architecture, artificial neural networks are classified intotwo categories. 1. Feed forward neural networks. These are the simplest form in which data flow is only in one direction. It flows from input nodes to the hidden nodes (if any) and then finally to the output nodes. Multi-layer Perceptron, radial basis function nets are examples of this category. 2. Feedback/recurrent neural networks. Contrary to the feed forward neural net- works, these architectures are bidirectional and have loops in it. Kohonen’s SOM, Hopfield networks are some examples. For our implementation, we have used feed-forward based neural network de-sign.5.2.3 Feed forward neural networks. Basic model Perceptron. Figure 5.2 shows a simplest form of feed-forward neuron model is called as ‘Perceptron’. In this model, (x1, x2,...xn) are theinput vectors, (w1, w2,...wn) are the weights associated with the input vectors, h isthe summation of all inputs with its weights and y is the output. Activation functiondecides the amplitude range for output. For example, if step function is used, therange is between 0 and 1, while if a signum function is used, the output lies between -1
  51. 51. 42and 1. The selection of activation function is decided by the complexity of proposedproblem. For our design, we have used a log sigmoid activation function. weights inputs x1 w1j ac va on x2 w2j func on net input net j ∑ Oj x3 ac va on . w3j . transfer . . func on . . . . θj x4 threshold w4j Figure 5.2. Basic feed forward model, Perceptron The basic algorithm for this model [9] can be explained as follows. 1. Initialize the associated weights and the activation function for the model. 2. Evaluate the output response for given input pattern (x1 , x2 , xn )t . 3. Update the weights based on following rule wj (t + 1) = wj (t) + η(d − y)xj (5.2) Based on the requirements, this principle can be extended to multiple layers.In general, a standard L-layer model has L input nodes, (L-1) hidden layers and
  52. 52. 43connected set of predefined output nodes. A typical three layer feed forward model isshown in Figure 5.3. It can be seen that the data flow is unidirectional and the outputshave no connection between the units in the same layer. Multi-layer perceptronbecame extremely popular with the use of back propagation concept. In case of back-propagation model, the error at the output is propagated backwards and accordinglythe weights for inputs are adjusted. xq wqi(1) wij(2) wjk(l) yk(l) Inputs Hidden Output layer layers layer Figure 5.3. Three layer Perceptron5.2.4 Design of neural network for character recognition problem. Followingis the description of some of these key design parameters in reference to characterrecognition problem. 1. Input Layers. Input layers to the neural network are the feature vectors ex- tracted for the all the characters. The overall size is decided by the character set and the feature vector size. 2. Output Layers. Output nodes are the set of all possible characters that are
  53. 53. 44 present in license plate information. The number of output nodes depends upon the numbers of characters that need to be classified. Typically, it consists of 26 alphabets ‘A-Z’ and 10 numbers ‘0-9’. However, to reduce algorithm complexity, certain countries avoid using both 0 and O together. In our test database, we have all the characters except O, 1 and Q. Therefore, the number of output layers is 33 in our implementation. 3. Hidden Layers. Hidden layers are the intermediate layers between the input and output layers. There is no generic rule as such for deciding the number of hidden layers. In our design, the number of hidden layers was experimentally found to be 25. For our implementation, we have used a supervised learning paradigm. Weobtained a set of 50 images for each character from the database images and trainedthe neural network using them. For better results, it is important to include images ofall types of character representation in the training dataset. We have used MATLABNeural network tool box for training the neural network. Following are some of thekey steps in MATLAB implementation.%Training the neural network using ideal target vectors[R,Q] = size(alphabet);[S2,Q] = size(targets);P = alphabet;net = newff(minmax(P),[S1 S2],{’logsig’ ’logsig’},’traingdx’);net.performFcn = ’sse’;net.LW{2,1} = net.LW{2,1}*0.01;%Layer weightsnet.b{2} = net.b{2}*0.01;%Bias vectornet.trainParam.goal = 0.5;
  54. 54. = 20;net.trainParam.epochs = 5500; = 0.7;[net,tr] = train(net,P,T);%Training the neural network using training image data setnetn=net;netn.trainParam.goal = 0.6;netn.trainParam.epochs = 300;%P1=new alphabet;%T1=Target vector;[netn,tr] = train(netn,P1,T1);% Simulating the network performance% Rp - > Feature vector of the character to be indentifiedA2=sim(netn,Rp);A2=compet(A2);answer = find(compet(A2) == 1);%alphamat= set of all possible characterscharacter=alphamat(answer); It was observed that for certain characters like 0-D and 5-S, error rate was morecompared to other characters, because of similar spatial information content. In orderto improve the overall accuracy, a two stage detection process for these characters isused. If the characters are possibly between 5, 0, S and D, then region propertieslike orientation, reflection coefficient were again used in the 2nd stage identification
  55. 55. 46process and the final possible character was identified. This two stage process hasimproved the overall algorithm accuracy by 1-2%.
  56. 56. 47 CHAPTER 6 IMPLEMENTATION IN MATLAB AND OPENCV The major focus of this project has been to create a software platform for ro-bust license plate recognition system. This platform then can be used as a frameworkfor hardware implementation in future. Initially, all the algorithms were implementedin MATLAB. Once complete system was tested and analyzed in MATLAB, OpenCVplatform was then used with aim of building real time embedded system in future.6.1 MATLAB implementation For MATLAB, all algorithms were made from scratch and then tested with thepredefined MATLAB functions. This was done in order to make our implementationcan be ported for future hardware implementation. For character recognition purpose,we have made use of artificial neural network toolbox. Following are some of theimportant leanings from MATLAB implementation. 1. For computer vision and image processing applications, it is always good to start first with MATLAB implementations because of its built-in function capability. Most of the functions like segmentation algorithms, morphological operations, region properties are already there and so ist easy to verify our custom code. 2. While doing this, we need to make sure that we don’t overuse MATLAB func- tionality. If we do so, the mapping of software and hardware becomes complex. 3. MATLAB is not the ideal platform to analyze algorithm speed.6.2 OpenCV OpenCV was first developed by Intel research team with a strong focus of realtime embedded applications. It is an open source library with over 500 functions in
  57. 57. 48the field of computer vision, machine learning and statistical pattern recognition.Thelibrary is written in optimized C language and therefore can be used with C as wellC++ environment. For our project, we have implemented license plate detection part in OpenCV.Following are some of the important aspects in OpenCV. 1. OpenCV has primitive data types like Cvpoint(), CvSize(), CvRect()and CvS- calar. These are used while accessing matrix or image details. 2. OpenCV has a object oriented design, especially C++ inheritance concept is the base of it. Therefore, IpIImage, which is used for accessing image data is basically derived from CvMat class, which is used to deal with matrix elements. 3. For our algorithm, we have used in-built functions like morphological dilation operator, contour finder.6.3 Comparison of MATLAB and OpenCV 1. OpenCV was specifically designed for image processing applications, while in comparison MATLAB is generic. Therefore, it is comparatively faster and easier to use. 2. MATLAB is based on Java and so there is delay associated with respect to inter- preting MATLAB code and then converting into Java for execution. OpenCV is written in C and so it has no such issues.The above points can be confirmed from Table 6.1. It shows execution speed for allthe steps in license plate detection for MATLAB and OpenCV implementation.
  58. 58. 49Table 6.1. Comparison of MATLAB and OpenCV execution time for license plate detection of a single frame Task MATLAB code execution time OpenCV code execution time Edge detection 0.69s 0.4 Dilation 0.05s 0.03s Segmentation 0.384s 0.3 Total time 1.13sec 0.731sec
  59. 59. 50 CHAPTER 7 EXPERIMENTAL RESULTS Performance evaluation for the license plate recognition algorithm is a chal-lenge in itself, since there is no common reference point for benchmarking [10]. Manysystems claim an accuracy of more than 95% however truth is all of them work onlyon restricted environmental conditions. In fact, some of these systems when usedin real time environment conditions, system performance was actually found in therange of 60 to 80%. It is important that while evaluating system performance, wemake sure that we test on all practical test cases.7.1 Database for testing of license plate recognition It is important that while evaluating system performance, we make sure thatwe test on all practical cases. We tested our algorithm on 1500 different framesobtained from a demo video freely available online, taken from City Sync licenseplate recognition camera. The resolution of all frames is 480X640. All these frameswere then classified into following types. 1. Type 1: In these samples, license plate is fully visible and all its characters are clear and distinct, see Figure 7.1(a). Also, the license plate location in the image is in the center portion of the image. 2. Type 2: Images in these samples are little blurred due to variations in illumina- tion conditions; see Figure 7.1(b). This is mainly because of motion blur effect during image acquisition stage. 3. Type 3: These samples have little dirt or shadows on or near license plate region, Figure 7.1(c). These include cases where even with presence of dirt/shadow, license plate information is still visible. Cases where license plate details are
  60. 60. 51 hidden need not be considered in analysis, as its impossible to correctly recognize such cases. 4. Type 4: In these samples, another text block apart from license plate is present in the frame, Figure 7.1(d). 5. Type 5: License plate details are present in two rows, Figure 7.1(e).7.2 Results Table 7.1 summarizes the results obtained for all the above mentioned testcases. Table 7.1. Summary of license plate localization and detection results Image type Images tested Plate detection success Character recognition success Type 1 879 99.4% 97.81% Type 2 442 96.3% 96.63% Type 3 153 94.8% 95.69% Type 4 44 100% 99.25% Type 5 11 100% 97% Total 1529 98.1% 97.05% Overall, the algorithm has been tested on images of 335 vehicles. On an average5 frames were available for each vehicle. A summary of license plate detection resultsin terms of number of frames per vehicle is shown in Table 7.2, where n is the numberof input frames available for each vehicle (maximum value of n=5).
  61. 61. 52 Table 7.2. Number of frames detected per vehicleNumber of frames per vehicle Number of vehicles falling in this category 5 291 3≤N ≤5 32 1≤N ≤3 5 None 7
  62. 62. 53 (a) (b) (c) (d) (e)Figure 7.1. Types of images included in license plate database (a) Type 1; (b) Type 2; (c) Type 3; (d) Type 4; (e) Type 5
  63. 63. 54 CHAPTER 8 CONCLUSION AND FUTURE WORK8.1 Conclusion In this project, a framework has been proposed to localize and recognize licenseplate in complex scenes. A detailed study of all previous implementations was per-formed and key implementation issues at license plate detection as well as recognitionstage were listed. Following are some of the key highlights of our algorithm. 1. We ensured to include all practical test cases in our database including complex scenarios like tilted license plate, license plate images taken in different illumi- nation conditions, presence of dirt/shadows on license plate. This serves as a common benchmarking for future implementations. 2. The use of Mexican hat operator helps to improve the performance of edge detection step, when there is only partial contrast between license plate region and region surrounding it. 3. Euler number criterion for a binary image helps to reduce the sensitivity of algorithm to license plate dimensions. 4. Preprocessing step using median filter and contrast enhancement ensures per- formance in case of low resolution and blurred images. 5. Reflection symmetry coefficient along with two stage identification process im- proves the character recognition performance significantly especially in dealing with complex cases like recognizing 0 and D. 6. Local threshold operation prevents a single brighter or darker region from cor- rupting the threshold value and thereby improves binarization process.
  64. 64. 558.2 Future recommendation8.2.1 Extension of algorithm for mobile license plate recognition systems.The algorithm can be extended to mobile license plate recognition systems because ofits ability to provide exact license plate location in complex cases. Also, to increasethe intelligence of license plate detection algorithm while working with video input,motion analysis can be applied to the sequential frames and selection of proper framecan be done. The performance of algorithm will also improve if higher resolutioninput frames are used.8.2.2 Hardware implementation. This project mainly focused on softwareimplementation aspect and algorithms were implemented in MATLAB and openCVplatforms. This can serve as a framework for hardware implementation using FPGA.VHDL codes for key image processing stages like edge detection, morphological dila-tion and erosion operation need to be developed. The major challenge is to implementartificial neural networks in FPGA.
  66. 66. 571. License plate localization fi=imread(‘Path of input vehicle image with license plate’); fin=rgb2gray(fi); d=double(fin); [r c]=size(d); %Mexican Filter Operator filter= ... [ 0 0 0 -1 -1 -1 0 0 0; 0 -1 -1 -3 -3 -3 -1 -1 0; 0 -1 -3 -3 -1 -3 -3 -1 0; -1 -3 -3 6 13 6 -3 -3 -1; -1 -3 -1 13 24 13 -1 -3 -1; -1 -3 -3 6 13 6 -3 -3 -1; 0 -1 -3 -3 -1 -3 -3 -1 0; 0 -1 -1 -3 -3 -3 -1 -1 0; 0 0 0 -1 -1 -1 0 0 0]; %Creating image matrix for Mexican Hat operator gm=zeros(r,c); for i=5:2:r-5 for j=5:2:c-5 gm(i,j)=sum(sum(double(f(i-4:i+4,j-4:j+4)).*filter,2)); end; end; %Removing the unwanted edges by using a threshold fh=gm>1200; %Dilation Operation x=1;
  67. 67. 58y=1;fs=double(fh);se=ones(3,3);for x=3:3:r-20 for y=3:3:c-20 if(x+50<=r) xend=x+50; else xend=r; end; if(y+100<=r) yend=y+150; else yend=c; end;if(sum(fh(x:xend,y))<=35 || sum(fh(x,y:yend),2)<=60) if((sum(fh(x,y:y+3),2)<=3 && sum(fh(x,y:y+3),2)>2 )) fs(x-2:x+2,y-2:y+2)=bwmorph(fh(x-2:x+2,y-2:y+2),’dilate’,se); end;end; end;;end;% Image with Dilation Performedf=double(fs);% Determining the size of an image ie number of rows% and columns[ row col ]= size(f);
  68. 68. 59% Initializing a matrix for segmented Imageg=zeros(row,col);gl=zeros(row,col);label=1;n=1;x=1;iter=[];it=0;ss_prev=0;nn=[];sss_mat=[];for i=1:2:row for j=1:2:col r_pt=i; c_pt=j; if(g(r_pt,c_pt)==0) while(true) % using 4 Neighbor Rule if(f(r_pt(n),c_pt(n))==1 && g(r_pt(n),c_pt(n))==0) g(r_pt(n),c_pt(n))=label; if(r_pt(n)+1<=row) if( f(r_pt(n)+1,c_pt(n))==1) r_pt=[r_pt r_pt(n)+1]; c_pt=[c_pt c_pt(n)]; x=x+1; end; end;
  69. 69. 60if(c_pt(n)-1>=1)if(f(r_pt(n),c_pt(n)-1)==1) r_pt=[r_pt r_pt(n)]; c_pt=[c_pt c_pt(n)-1]; x=x+1;end;end;if(c_pt(n)+1<=col)if(f(r_pt(n),c_pt(n)+1)==1) r_pt=[r_pt r_pt(n)]; c_pt=[c_pt c_pt(n)+1]; x=x+1;end;end;if(r_pt(n)-1>=1)if(f(r_pt(n)-1,c_pt(n))==1) r_pt=[r_pt r_pt(n)-1]; c_pt=[c_pt c_pt(n)]; x=x+1;end;end;end;if(n>=x) break;end;n=n+1;end;
  70. 70. 61 y1=min(r_pt); y2=max(r_pt); x1=min(c_pt); x2=max(c_pt); a1=g(min(r_pt):max(r_pt),min(c_pt):max(c_pt)); f1=d(min(r_pt):max(r_pt),min(c_pt):max(c_pt)); [ra ca]=size(a1); if(n>=50 ) b1=bwlabel(a1); ss=regionprops(b1,’EulerNumber’); sss=struct2array(ss); sss=min(sss); sss_mat=[ sss_mat sss]; if(sss < ss_prev && sss<0 && ca<=190 ...&& ra <=60 && ca>=50 && ra>=15 && mean(mean(f1))<=220 ) x_cor1=x1; y_cor1=y1; x_cor2=x2; y_cor2=y2; ss_prev=sss; end; label=label+1; else g(r_pt,c_pt)=0; end; end; x=1;
  71. 71. 62 n=1; it=1; end; end; if( exist(’y_cor1’)==1) d(y_cor1:y_cor1+2,x_cor1:x_cor2)=255; d(y_cor2:y_cor2+2,x_cor1:x_cor2)=255; d(y_cor1:y_cor2,x_cor1:x_cor1+2)=255; d(y_cor1:y_cor2,x_cor2:x_cor2+2)=255; end; %Segmented license plate image d=mat2gray(d); lp= d(y_cor1:y_cor2,x_cor1:x_cor2);2. Character Segmentation %License plate image, characters of which are to be segmented lp1=d(y_cor1:y_cor2,x_cor1:x_cor2); [rl cl]=size(lp1); % Median Filtering lp=medfilt2(lp1,[3 3]); %Contrast enhancement lpf=imadjust(lp,stretchlim(lp, [0.1 0.5])); %Creating output image matrix output=zeros(rl,cl); %Window for local threshold operation dis=round(cl/7); %Local threshold operation
  72. 72. 63 for i=1:dis:cl if(i+dis-1<=cl) t=threshcal(lpf(:,i:i+dis-1),a); output(:,i:i+dis-1)=lpf(:,i:i+dis-1)<=t; else t=threshcal(lpf(:,i:cl),a); for z1=2:rl-1 for z2=i+5:cl-5 if(mean(mean(lpf(z1-1:z1+1,z2-5:z2+5)))<=t) output(z1,z2)=1; end; end; end; output(:,i:cl)=lpf(:,i:cl)<=t; end; end; %Structuring element for erosion operation se=[1 1 1 1 1 1 1 1 1 1 1 1 1 1 1]; output=output-imerode(output,se); [of lab lpdet]=reggrow1(logical(output),number); %Segmented characters lpdet=logical(lpdet);3. Character Recognition % Output string giving license plate iformation lpstr=[]; for i=1:lab-1
  73. 73. 64 R=lpdet(:,st:st+9); st=st+10; b=bwlabel(R);%Feature extraction ar=struct2array(regionprops(b,’area’)); or=struct2array(regionprops(b,’orientation’))/90; eu=struct2array(regionprops(b,’eulernumber’))/10; pe=struct2array(regionprops(b,’perimeter’)); mi=struct2array(regionprops(b,’minoraxislength’)); ma=struct2array(regionprops(b,’majoraxislength’)); temp=logical(R);%Reflection X and Y coefficient determination v1=temp; v1(:,6:10)=flipdim(temp(:,1:5),2); vx=(v1 + temp)/2; vx=vx>=0.5; xcoef=sum(sum(temp),2)/sum(sum(vx),2); v2=temp; v2(1:12,:)=flipdim(temp(13:24,:),1); vy=(v2 + temp)/2; vy=vy>=0.5; ycoef=sum(sum(temp),2)/sum(sum(vy),2); ed=struct2array(regionprops(b,’equivdiameter’))/100; [val pos]=max(fa); vcoeff=pe(pos)/ar(pos); mcoeff=ed(pos); Rp=[xcoef/ycoef;pe(pos)/ar(pos);mi(pos)/ma(pos);or(pos);...
  74. 74. 65 eu(pos);sum(temp,2)/norm(sum(temp,2));... transpose(sum(temp,1))/norm(sum(temp,1)) ]; A2=sim(netn,Rp); A2=compet(A2); answer = find(compet(A2) == 1); if(i<=numel(lpnum)) if(alphamat(answer)==lpnum(i)) numrc=numrc+1; else answ=find(alphamat==lpnum(i)); err(answ)=err(answ)+1; end; end; lpstr=[lpstr alphamat(answer)];end;numc=numc+numel(lpnum);if(strcmp(lpstr,lpnum)==1) tr=tr+1; sr=strcat(num2str(num),’/’,num2str(1),’//’); casep=[casep sr];else fr=fr+1; sr=strcat(num2str(num),’/’,num2str(1),...’/’,num2str(answer),’//’); casen=[casen sr];end;
  76. 76. 67#include "stdafx.h"#include <iostream>using namespace std;#include <cv.h>#include <cxcore.h>#include <highgui.h>#include <time.h>#include <windows.h>#include <vector>int _tmain(int argc, _TCHAR* argv[]){ double ts = GetTickCount(); IplImage *img1 = cvLoadImage("Input image path"); IplImage *img=cvCreateImage(cvSize(640,480),IPL_DEPTH_8U,1); cvCvtColor( img1, img, CV_RGB2GRAY ); IplImage *out=cvCreateImage(cvGetSize(img),IPL_DEPTH_8U,1); IplImage *out1=cvCreateImage(cvGetSize(img),IPL_DEPTH_8U,1); CvSize c= cvGetSize(img); CvScalar a1; a1.val[0]=0; IplImage *out2=cvCloneImage(img); IplImage *out3=cvCloneImage(img); IplImage *out4=cvCloneImage(img);for(int i=0;i<c.height;i++){ for(int j=0;j<c.width;j++) {
  77. 77. 68cvSet2D(out3,i,j,a1);cvSet2D(out4,i,j,a1); };}; CvMat *imdata=cvCreateMat( 9, 9, CV_16UC1); CvMat *matdata=cvCreateMat( 9, 9, CV_16UC1); CvMat *outdata=cvCreateMat( 9, 9, CV_16UC1); int it=0; double a[]={0, 0, 0, -1, -1, -1, 0, 0, 0, 0, -1, -1, -3, -3, -3, -1, -1, 0, 0, -1, -3, -3, -1, -3, -3, -1, 0, -1, -3, -3, 6, 13, 6, -3, -3, -1, -1, -3, -1, 13, 24, 13, -1, -3, -1, -1, -3, -3, 6, 13, 6, -3, -3, -1, 0, -1, -3, -3, -1, -3, -3, -1, 0, 0, -1, -1, -3, -3, -3, -1, -1, 0, 0, 0, 0, -1, -1, -1, 0, 0, 0,};if(i%9==0 && i>0)cout<<endl;else cout << string( 3, ’ ’ );int Mi = ((int*)( + F.step*0))[i];cout<< Mi;};*/