Text Detection and Recognition


Published on

Text Detection and Recognition from Images using Matlab, Final year Project Presentations

Published in: Technology
No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Text Detection and Recognition

  1. 1. Text Detection and Character Recognition from Images<br />BadruzNasrin Bin Basri<br />1051101534 <br />Supervisor : MohdHaris Lye Abdullah<br />1<br />
  2. 2. Contents<br />Introduction<br />1<br />Literature review  <br />2<br />Method Used <br />3<br />Experiment and Result<br />4<br />Future works   <br />5<br />2<br />
  3. 3. Aims and objectives<br />Student 2<br />Recognition -<br />Recognize each of the character in the detected text region using a suitable algorithm<br />Segmentation -<br />Separate the text region into its individual characters.<br /> <br />3<br />The aim of this project is to detect, extract and recognize text from images, particularly license car plate.<br />
  4. 4. Motivation <br />4<br />Text detection and recognition in general have quite a lot of relevant application for automatic indexing or information retrieval such document indexing, content-based image retrieval, and license car plate recognition which further opens up the possibility for more improved and advanced systems.<br />
  5. 5. Problem statement<br />5<br />To segment the image to individual characters, we need to find the characteristic to be used as boundary to segment the image. <br />To classify, we need to use the best template to compare with the segmented image and to determine how the template will be used to compare with the image.<br />
  6. 6. Literature review<br />6<br />Segmentation<br />Klouverin research on recognition text in PayPal HIP [1] proposed the use of vertical projection to segment the characters in images. There are several reasons vertical segmentation been proposed by Klouver:<br /><ul><li>PayPal HIP image is computer generated only minor preprocessing needed.
  7. 7. There are obvious separation between each character
  8. 8. Character size in PayPal HIP is fixed</li></ul>Study on Malaysian License Plate Recognition by Othman [3] proposed a model specifically to detect and recognize the text in Malaysian license plate. For segmentation, connected components method has been proposed however this method only can use if the license plates are single row license plate and the study only been made on single row license plate. Same method also been use by Ganapathy and Lui in [4]<br />
  9. 9. 7<br />Literature review<br />Recognition<br />Klouverin research on recognition text in PayPal HIP [1] and Ho C. H. et. al in research on License Plate Recognition(LPR) [2] used Templates Matching to recognize the characters in image. Klouver detailed the matching classifier into four types of classifier that are Pixel Counting, Horizontal Projection, Vertical Projection and Template Correlations. Klouver’s experiment proved that the best classifier is Vertical projection and Template Correlation where both of this classifier yield 100% accuracy.<br />Fixed type of font in image(PayPal HIP) makes it very easy to distinguish between different characters using templates matching. There are no other research that yield 100% accuracy.<br />
  10. 10. 8<br />Literature review<br />Recognition<br />Study on Malaysian License Plate Recognition by Othman [3],Ganapathy and Lui in [4], M.Fukumi et. Al [6] , AnasJ.A. Husain et. Al [7] used Neural Network to recognized the text. Compared to templates matching, neural network consume more time. Neural network also need training before it can be used and it only can achieve high accuracy if the sampled image is almost same with the training images. Andrew Vogt and Joe G. Bared [5] concluded the disadvantages of neural network are :<br />Minimizing overfitting in neural networks requires a great deal of computational effort <br />The individual relations between the input variables and the output variables are not developed by engineering judgment so the model tends to be a black box or input/output table without analytical basis and to make the accuracy level high the sample size has to be large. <br />
  11. 11. 9<br />Literature review<br />Recognition<br />Jared Hopkins and Tim Anderson in [9], used Fourier Descriptor to recognize text in image. In most of the researches, Fourier descriptor been used to recognize more complex shape such as for logo classification by Folkers and Samet [10] and for Sinhala Script by Rohana, Ruvan and Kevin [11].<br />Basically there are no research on LPR using Fourier descriptor(FD), hence, this research will also test the usage of FD to recognize text in Malaysian License Plate.<br />WisamAl Faqheri and SyamsiahMashohor in A Real-Time Malaysian Automatic License Plate Recognition (M-ALPR) using Hybrid Fuzzy [12] used the hybrid Fuzzy method to recognize the license number. Compared to other study previously done on license plate detection where almost all of the previous work relied on a single method like template matching or neural network Wisamand Syamsiah proposed combination of more than one method based on the type of license plate. <br />
  12. 12. Approach - Flowchart to recognize the characters in image<br />10<br />
  13. 13. Make template<br />To create templete.mat to be use for classification:<br />11<br />…<br />…<br />Matrix size 24 X 42 X 36 <br />Saved as template.mat<br />36 images of characters<br />Size = 42 X 24<br />
  14. 14. Preprocess<br />12<br />
  15. 15. Segmentation – Vertical Projections <br />Preprocessed Image<br />Vertical projection analysis<br />Segmented Image<br />Vertical Projection<br />13<br />
  16. 16. Segmentation – Vertical Projections <br />Weaknesses<br />14<br />Image that failed to be segmented by vertical projection<br />
  17. 17. The segmentation character involves the following steps:<br />Scan the image from left to right to find ‘on’ pixel.<br />If on pixel been found, all ‘on’ pixel connected to the detected on pixel will be extracted segmented as a pixel.<br />The process will be repeated until it reach end right of the image.<br />15<br />Segmentation – Connected Components <br />
  18. 18. Recognition - Pixel Counting<br />16<br />tempSum = sum(sum(templates(:,:,j))); inSum = sum(sum(chars(:,:,i))); allCorrs(j) = abs(tempSum - inSum);<br />allCorrs(j) = 1 - (allCorrs(j)/1008);<br />
  19. 19. Corr2<br />𝑟=mn𝑖𝑚𝑛−𝑖(𝑗𝑚𝑛−𝑗)(mn(𝑖mn− 𝑖)2)(mn(𝑗mn− 𝑗)2)<br />Where 𝑖 is the mean of the input matrix i and 𝑗 is the mean of the input matrix j.<br />0 < r < 1<br />1 mean i and j is exactly same while 0 mean the i and j not same at all.<br /> <br />17<br />
  20. 20. Recognition - Vertical projections<br />18<br />tempVP = sum(templates(:,:,j)); inVP = sum(chars(:,:,i)); allCorrs(j) = corr2(tempVP, inVP);<br />
  21. 21. Recognition - Horizontal projections <br />19<br />tempHP = sum(templates(:,:,j)'); inHP = sum(chars(:,:,i)'); allCorrs(j) = corr2(tempHP, inHP);<br />
  22. 22. 20<br />Recognition - Template Correlations<br />temp = templates(:,:,j); <br />in = chars(:,:,i); <br />allCorrs(j) = corr2(temp, in);<br />
  23. 23. Recognition – Fourier Descriptor<br />Following is the detailed steps on extracting and comparing the Fourier Descriptor (FD)<br />Edging<br />21<br />U= 𝑥0𝑦0𝑥1𝑦1...𝑥𝑛𝑦𝑛<br /> <br />
  24. 24. Recognition – Fourier Descriptor<br />Extracting FD – 1 D Discrete Fourier Transform (DFT) been done to the complex vector U to get the frequency domain of the boundaries using the following equation:<br /> <br />22<br />F=𝐹𝐹𝑇𝑈=𝑘=0𝑁−1Uk− 2πN𝑘𝜇<br /> <br />
  25. 25. Recognition – Fourier Descriptor<br />Normalize FD: <br />Translation invariant<br />Scale invariant<br />Rotation invariant<br />23<br />
  26. 26. Recognition – Fourier Descriptor<br />Images of ‘E’ reconstructed from (a) n = 4 (b) n = 8 (c) n = 10 (d) n = 15 (e) n = 25 (f) n = 30 (g) n = 278<br />Resize FD – As FD contains information of all information of the ‘on’ pixel, the size of FD is number of on pixel. To make it comparable with other FD it need to be resized to predefined number of descriptor, Figure 3.9 show different shape reconstructed using different number of descriptor. As to resize the FD to n descriptor, function shiftfft in Matlab will remove low frequency descriptor leaving only n-th highest descriptor. <br />24<br />
  27. 27. Recognition – Fourier Descriptor<br />Compare FD – CompareFD a measure of the difference between two inputs FD. It will quantify the difference between FDs. Higher values of different mean the two FDs are far apart in shape. The extracted FD, I can be compared with using the following algorithm:<br />CompareFD(I,T)<br />D ← ø<br />for each Templates 𝑡𝑖∈T<br /> do diff = -1<br /> if length(𝑡𝑖) = length (I)<br /> do diff ← sum(𝑎𝑏𝑠(𝑡𝑖−𝐼)2)<br />return k such that 𝑟𝑘=min(𝑅)<br /> <br />25<br />
  28. 28. Recognition – Heuristic Filter<br /><ul><li>Context Approach</li></ul>After considering all data in database, it was concluded that:<br />First two characters are always text.<br />Third character can be text or number.<br />All following characters are always numbers. <br />26<br />
  29. 29. Recognition – Heuristic Filter<br /><ul><li>Euler number - Euler number is equal to the number of connected elements (always equal to one) minus the number of holes.</li></ul>27<br />
  30. 30. Recognition – Heuristic Filter<br />28<br />Notice that for ‘H’, x line passed through 2 white line and y passed through 2 white line, for’ W’, x passed 3 white line and y passed 2 white line while ‘M’ is opposite.<br />
  31. 31. Experiment 1: Comparison between Different Segmentation Method and Different Templates Matching Classifier<br />29<br />Template with size 42 X 24 was created using images of 36 characters.<br />To conduct the experiment, all 125 images have been renamed as their ground truth and saved in a folder. <br />A Matlab script as included in appendices was created to load all the images in the folder as well as their name and then perform preprocess segmentation and recognition to all of the images. <br />Then the result of the segmentation and recognition as well as time needed to recognize a number plate were recorded and calculated using following equation. The result will also be analyzed automatically by the Matlab script.<br />Some of images that been used in the experiment<br />
  32. 32. Segmentation Accuracy was calculated using formula :<br />𝐴𝑆𝑒𝑔𝑚𝑒𝑛𝑡=𝑁𝑆𝑒𝑔𝑚𝑒𝑛𝑡𝑒𝑑𝑁𝑆𝑎𝑚𝑝𝑙𝑒𝐶h𝑎𝑟 x 100%<br />Where 𝐴𝑆𝑒𝑔𝑚𝑒𝑛𝑡 is segmentation accuracy, 𝑁𝑆𝑒𝑔𝑚𝑒𝑛𝑡𝑒𝑑 is number of correctly segmented character and 𝑁𝑆𝑎𝑚𝑝𝑙𝑒𝐶h𝑎𝑟 is number of characters in sample.<br />Classification Accuracy was calculated using formula :<br />𝐴𝐶h𝑎𝑟=𝑁𝑅𝑒𝑐𝑜𝑔𝑛𝑖𝑧𝑒𝑁𝑆𝑒𝑔𝑚𝑒𝑛𝑡𝑒𝑑x 100%<br />Where 𝐴𝐶h𝑎𝑟 is recognition accuracy, 𝑁𝑅𝑒𝑐𝑜𝑔𝑛𝑖𝑧𝑒 is number of correctly recognized character and 𝑁𝑆𝑒𝑔𝑚𝑒𝑛𝑡𝑒𝑑 is number of characters that had been correctly segmented.<br />Classification Accuracy was calculated using formula :<br />𝑡𝐶h𝑎𝑟=𝑇𝑁𝑆𝑎𝑚𝑝𝑙𝑒<br />Where 𝑡𝐶h𝑎𝑟 is average recognition time, 𝑇 total running time to recognize all sample images and 𝑁𝑆𝑎𝑚𝑝𝑙𝑒 is number of sample images.<br /> <br />30<br />Experiment 1: Comparison between Different Segmentation Method and Different Templates Matching Classifier<br />
  33. 33. Then the experiment repeated four times using connected components as segmentation method and the following as recognition classifier:<br />Pixel Count<br />Vertical Projection<br />Horizontal Projection<br />Templates Correlation<br />31<br />Experiment 1: Comparison between Different Segmentation Method and Different Templates Matching Classifier<br />
  34. 34. Result<br />32<br />Comparison on segmentation by Vertical Projection and Connected Components<br />Comparison on Template Matching using different classifier<br />
  35. 35. Experiment repeated two times using connected components as segmentation method and the following as recognition classifier:<br />Templates Correlation<br />Fourier Descriptor <br />33<br />Experiment 2: Comparison between Template Correlation and Fourier Descriptor<br />
  36. 36. Result<br />34<br />Comparison on Recognition by Templates Correlation and Fourier Descriptor<br />
  37. 37. Experiment repeated with introducing context in the algorithm<br />35<br />Experiment 3: Improvement on LPR Using Context Approach<br />Result<br />Comparison on recognition by Templates Correlation after context been introduced<br />
  38. 38. Experiment repeated with introducing hybrid in the algorithm<br />36<br />Experiment 4: Improvement on LPR Using Hybrid method<br />Result<br />Comparison on recognition by Templates Correlation after hybrid been introduced<br />
  39. 39. Discussion<br />37<br />Image that failed to be segmented using connected components<br />Preprocessed image after erosion<br />
  40. 40. Why heuristic filters failed ?<br />38<br />Image that failed to be recognized due to change in Euler number<br />
  41. 41. Why Fourier Descriptors failed ?<br />39<br />Image that failed to be recognized due to unsmoothed image<br />Image that failed to be recognized due to rotation invariant<br />
  42. 42. Conclusion<br />The objective of this paper is to segment and recognize characters in image have been achieved. Even the segmentation accuracy from the experiment is 100% the result during real application may be lower due to limited set of picture used in experiment. However, this shown that segmentation using connected components is best method to segmenting the image. <br />After several experiments been done to find the best method to recognize the characters with highest accuracy and considerable amount of time, the best way is by using templates correlations as main recognition method with Fourier Descriptor and several heuristic approach as filters. Experiments have found that this method’s recognition accuracy is 98.46%.<br />40<br />
  43. 43. References<br />41<br />[1] Kurt Alfred Kluever. (2008) Digital Media Library : RIT Scholars. [Online]. https://ritdml.rit.edu/bitstream/handle/1850/7813/KKlueverTechPaper05-20-2008.pdf<br />[2] C. H. Ho, S. B. Koay, M. H. Lee, M. Moghavvemi, and M. Tamjis, "License Plate Recognition (Software)," Universiti Malaya, No Date.<br />[3] SherozKhan, Rafiqul Islam Othman khalifa, "Malaysian Vehicle License Plate Recognition," The International Arab Journal of Information Technology , pp. 359-364, 2007.<br />[4] VelappaGanapathy and Wen Lik Dennis Lui, "A Malaysian Vehicle License Plate Localization and Recognition System," Monash University Malaysia,.<br />[5] Office of Safety and Traffic, Operations Research and Development, USA. (2010, February) Literature Review : Artificial Neural Network. [Online]. http://www.tfhrc.gov/safety/98133/ch02/body_ch02_05.html<br />[6] M. Fukumi, Y. Takeuchi, and M. Khalid, "Neural Network Based Threshold Determination for Malaysia License Plate Character Recognition," UniversitiTechnologi Malaysia, No Date.<br />
  44. 44. References<br />42<br />[7] K. Saleh Ali Al-Omari, Putra Sumari, A. Sadik Al-Taweel, and J.A. AnasHussain, "Digital Recognition using Neural Network," Journal of Computer Science, vol. 5, no. 6, pp. 427-434, 2009.<br />[8] VelappaGanapathy and Leong LiewKok, "Handwritten Character Recognition Using Multiscale Neural Network Training Technique," World Academy of Science, Engineering and Technology, vol. 39, 2008.<br />[9] Jared Hopskins and Tim Anderson, "A Fourier Descriptor Based Character Recognition Engine Implemented under the GameraOpen-Source Document Processing Framework," No Date.<br />[10] Andre Folkers and SametHanan, "Content-based Image Retrieval Using Fourier Descriptors on a Logo Database," in Proc of the 16th Int. Conf. on Pattern Recognition, vol. III, Quebec City, Canada, 2002, pp. 521-524.<br />[11] RajapakseK Rohana, Ruvan A Weerasinghe, and Kevin E Seneviratne, "A Neural Network Based Character Recognition System For Sinhala Script," Department of Statistics and Computer Science, University of Colombo, Colombo, No Date.<br />[12] WisamAl Faqheri and SyamsiahMashohor, "A Real-Time Malaysian Automatic License Plate Recognition (M-ALPR) using Hybrid Fuzzy," IJCSNS International Journal of Computer Science and Network Security, vol. VOL.9, no. No.2, pp. 333-340, February 2009.<br />
  45. 45. Thank You <br />43<br />