Statistical Analysis Of Myanmar Words On The World Wide Web For+ Search Engine Development+

Statistical Analysis of Myanmar Words on the World Wide Web for Search Engine Development Pann Yu Mon Maung Maung Thant Ohnmar Htun Pe s065402@ics.nagaokaut.ac.jp mmthant@gmail.com ohnmar.iuj@gmail.com San Ko Oo Yoshiki Mikami sankooo@gmail.com mikami@kjs.nagaokaut.ac.jp †Management and Information Systems Engineering Department Nagaoka University of Technology ††International University of Japan Abstract the Indian subcontinent between 5th Century B.C and 3rd Century AD. Myanmar language has 33 This paper introduces an automatic consonants and 12 vowels according to traditional Myanmar word analysis program for ongoing tones on grammar. research of Myanmar search engine development. Since 1990 Myanmar natural language In this research we collected Myanmar words from documents on the World Wide Web to know processing task has been done by Myanmar which words are frequently used. This program is Unicode & NLP Research Center. The first designed for encodings compatible with Unicode Myanmar Unicode font for GUI environment 5.1standard. Our program can automatically (Mac) was developed in 1988 and the one for generate Markov Chain matrix on the result Windows system was developed in 1992. In 1998, words. The program was written by using PHP Myanmar Language processing was first script. Myanmar head words that include in Myanmar-English dictionary are also used as discussed at ISO/IEC JTC1 and Unicode index words. Technical Committee and finally Myanmar Keywords character code set was included in ISO 10646. Until now, they keep on trying over Myanmar, Code conversion tools, Myanmar word Myanmar language processing tasks to cope well searching with all applications so as to complete all the tasks to cover the whole area which requires more 1. Introduction endeavors. In this research, the program that can Myanmar Language, a member of the automatically collect Myanmar words from the Tibeto-Burman language, subfamily of the Sino- Myanmar Web Pages is proposed. The main Tibetan family of language, is spoken as mother purpose of this research is to present the analysis language by more than 37 million Burmese and as of Myanmar words on the Myanmar Web pages to second language by about 20 million ethnic support Myanmar Search Engine Development. minorities in Myanmar. It is the only official To establish the Myanmar Search Engine, it is language of Myanmar which is formerly known as needed to do a lot of tasks such as indexing rule, Burma. Myanmar language is written in a script sorting algorithm, stemming algorithm, word shaped in circular and semi-circular letters, which breaking algorithm and so on. are adopted from the Mon script. And the mon In this study, we have collected script is derived from Indian Brahmi flourished in Myanmar Web pages from various Web sites

including Myanmar daily newspaper, community to multi-font converter to the Unicode 5.1. At last Web sites, news Web sites total of which accounts the program run for searching the word from input to 9,274 Kbytes. And then we extracted words text, and result words are saved in the Database. The process will be explained step by step in the from downloaded Myanmar Web pages. And next section in more detail. detail process of collecting words and analysis of result data will be discussed in following sections. 3.1.First Step : Downloading Myanmar Web Pages 2. Related Research World Wide Web is the most convenient A number of researchers not only from existing source of linguistic data providing the local but also from word wide have collected users abundance of texts in various types in a Myanmar words from different sources for their large number of languages. Already having in individual purposes. electronic forms, these texts are quite suitable for From 2007, Myanmar Unicode and NLP the corpus studies. Research Center has started the development task In order to download Myanmar Web pages, it needs very efficient crawler that can of Myanmar National Corpus (MNC) [5]. MNC collect only Myanmar Web pages selectively from includes all texts including written text and the World Wide Web. In this research, the spoken text from various resources. That project is Language Specific Crawler (LSC) developed by almost finished. one of the authors [3] was used. LSC runs Hla Hla Htay and colleagues [2] have concurrently with language identifier and collect developed Myanmar corpora based on various Myanmar Web pages efficiently. Following table explains the sources of the downloaded web sites. resources such as text from official newspapers in After downloading, downloaded pages were Myanmar, over 300 full books and Myanmar texts passed to converter. from various Web sites including news sites and on-line magazines. In their research they had Table 1. Detail Information for source data processed all their tasks based on ASCII format. 3. Methodology 3.2.Second Step : Conversion of various encoding to Unicode 5.1 Standard Myanmar texts on the Web are using various encoding which are not fully compliant with Unicode 5.1. So it is required to convert the crawled Web Pages to Unicode encoding. If the Web pages are encoded in Unicode then the work Figure. 1. Step by step Procedure of Analysis becomes easier. The step by step processes of our In order to convert various Myanmar analysis are shown in figure 1. Firstly it needs to encodings to Unicode, an efficient converter is collect Myanmar Web pages regardless of their needed. Currently, there are a number of fonts and encodings. Then, we have to pass them Myanmar font conversion tools available on the

Web. In this research, Kanaung converter 1 and match. If no such match is found in the word lists, Burglish converter2 were used. Although both of the character is simply segmented as a word. them work nicely, it is still needed to edit a little bit. For example, Kanaug converter could not 3.4. Fourth Step: Frequency Markov covert ‘ ’ and ‘ ’ properly and correctly. In case Chain Analysis of Burglish, it works correctly in the conversion from “Zawgyi-One” font to “Myanmar3” font. In the program, Word-based Markov But in the conversion from “Wininwa” font to models are also used to calculated word matrix “Myanmar3” font, it cannot covert accurately for table to know the adjacency word in the sentences ‘ ’. And it cannot correctly work on punctuation (This mean which word most frequently appears marks and quotation marks. Thus manual after one word.) It gives us high level background correction is needed in those cases though they are information for word boundary detection in somewhat perfect. parsing of the Myanmar language. Our program firstly finds the words on the given web pages and 3.3. Third Step: Word Searching calculates the frequency of that word to know how Algorithm many times that word appears on the Web sites. After that, Markov chain matrix table was Myanmar language is written in a syllabic generated automatically. system and there are no spaces always put between words or sentences. That is why word 4. Result segmenting algorithm and word searching algorithm for Myanmar Language are needed. We downloaded the various web sites Very little research in different approach has been including newspaper sites, blog sites, published on segmenting sentences into words in Myanmar language [1] [4]. entertainment sites, sport sites and collected 9,274 In our program, all of the Myanmar head Kbytes of text data. After running the program, words that included in Myanmar–English total 766,892 words are collected and 12,211 Dictionary 3 are used as indexed file. It includes unique head words found. 28,000 Myanmar words. Those head words are stored in the database and sorted in reverse order of syllable length to compare with the input data. 4.1. Distribution of Words on input string If the input word is matched with one of the head word, the program will retrieve that word. If the It is found that mono-syllable is most input word does not match with the head word frequently used because those words can be used lists, the program cannot retrieve the word in several ways. For Example, mono-syllable correctly. Thus the accuracy of this algorithm is “ ” was found more than 20,000 times. largely depends on the head word lists. Because it can be used in different ways. For In our algorithm the longest matching Example, in case 1: polite prefix to a young man’s algorithm, was used to find the word on the input name (as in “ ”), in case 2: postpositional data. It normally starts at the first character in a marker to indicate objective (as in text using a heard word list and attempts to find “ ”), in case 3: emphatic the longest word in the list. If such a word is particle suffixed to words (as in found, the longest-matching algorithm marks a “ ”) and in case 4: post boundary at the end of the longest word, and then positional marker indicating destination (as in it repeats the same process as to start searching “ ”). And then bi- longest match at the characters following the syllables words are second most and it is followed by the tri-syllables and so on. The top ten words sorted by frequency for mono-syllable, bi- syllables, tri-syllables and tetra-syllables are 1 http://code.google.com/p/kanaung/ shown in the following tables. 2http://burglish.googlepages.com/fontconv.htm 3 Myanmar-English dictionary produced by Department of the Myanmar Language Commission

Table 2. Top ten mono-syllable words Table 3. Top ten bi-syllable words Mono-Syllable Frequency Bi-Syllable Frequency [ko] 20070 [Kyun 3537 Postpositional marker to (2.61%) taw] (0.46%) indicate objective case I(male) [ma] 18181 [Kyun ma] 3332 Partical prefixed to a verb to (2.40%) I(female) (0.43%) the negative sence [Ka lay] 1994 [ka] 17469 Child (0.26%) Postpositional marker to (2.30%) 1981 [A twat] indicate nominative case (0.25%) For [tal] 14424 [Ae di] 1737 Colloquial form of the (1.90%) That (0.22%) sentence final [par] 12774 Particle denoting inclusion (1.70%) Table 4. Top ten tri-syllable words Table 5. Top ten tetra-syllable words Tetra-Syllable Frequency [sar yay sa 222 Tri-Syllable Frequency yar] (0.02%) [Tha yot 627 Author saung] (0.08%) [a nu pa nyar] 204 Actor Art (0.02%) [Pa ri thet] 500 [a chay a nay] 176 Audience (0.06%) Condition (0.02%) [Sa yar ma] 495 [a yay a tar] 157 Teacher(female) (0.06%) Writing (0.01%) [Thu nge 404 [a mhat ta ya] 138 chin] (0.5%) Remembrance (0.01%) Friend [Main ka lay] 400 Girl (0.05%) 600,000 581,355 500,000 number of collected words 400,000 300,000 200,000 147,100 100,000 27,770 9,752 758 117 16 5 17 2 - Mono- Bi- Tri- 4- 5- 6- 7- 8- 9- 10- Syllable Syllable Syllable Syllable Syllable Syllable Syllable Syllable Syllable Syllable Number of Syllables Figure. 2. Number of Syllables found in Test Data

4.2. Word Level Frequency Matrix Based on the input string, the program for parsing of the sentence into words. By applying generated word level Markov table. By using this this algorithm in character level we can also generate matrix we can know adjacency word pairs. It a character level Markov table. It can be used in gives us the high level background information Myanmar character input method to Mobile phone. Table 6 .Word-Level Matrix Sum of Second Frequency Word Grand First Word Total 1144 1144 722 1273 1217 4893 1564 2343 1339 1511 2850 934 934 1205 1717 2922 809 1754 Grand Total 1205 722 2651 1339 1273 2373 1511 1144 1217 16840 4.3. Distribution of characters on Input String It is found that the words begins with the “ ” is the over 90,000 and it is first ranking character. And it is followed by the “ ” and so on. No words are found We analyzed character level frequency of the that starting with the characters “ ”. We could not input data. The result is shown in Figure 3. find that words even in the Myanmar – English dictionary. 100000 90000 80000 70000 number of collected words 60000 50000 40000 30000 20000 10000 0 List of Characters Figure. 3. Total Frequency of Myanmar Characters found in Test Data

5. Error Analysis expect this ongoing research will yield benefits for our Myanmar search engine development task. In our test data of 9,274 Kbytes, we found 2,935,233 characters which excluding Acknowledgements punctuation marks, numerals and English words. In terms of words, we identified total 766,892 We acknowledge and highly appreciate Myanmar words (12,211 unique headwords). But the kind assistance and help given by Myanmar 5,861 words (0.76%) were not identified. The Unicode & NLP Research Center. We would like errors result from the incorrect spelling in the to express our thanks to Dr. Daw Myint Myint original text, undefined headwords (proper nouns Than and U Ngwe Tun as they kindly provided us which are not defined in the dictionary) and the data we are in need of. incorrect description of syllable length in the database. Moreover, some error results from the References words ending with some characters such as “ ” (Myanmar Sign Dot Below) and ambiguity in word segmentation. Some examples of errors are [1] Hla Hla Htay and et al., “Myanmar Word listed in Table 7. Segmentation using Syllable level Longest Matching”, Proceedings of the 6th Workshop on Asian Language Resources (ALR6), Hyderabad, Table 7. Some Examples of errors India, January 2008. [2] Hla Hla Htay, G. Bharadwaja Kumar and Kavi N. Murthy, “Constructing English-Myanmar Parallel Corpora”. The Fourth International Conference on Computer Application 2006. [3] Pann Yu Mon, Chew Yew Choong, Yoshiki Mikami, “Language Specific Crawler for Myanmar Pages”, Proceedings of the 11th International Conference on Humans and Computers (HC 2008), Nagaoka, Japan, November 2008. [4] Tun Thura Thet and et al., “Word Segmentaion of the Myanmar Language”, Journal of Information Science, Vol. 34, No.5, pp 688- 704. 2008 [5] Wunna Ko Ko and Thin Zar Phyo, “Selection of XML tag set for Myanmar National Corpus”, 6. Conclusion Proceedings of the 6th Workshop on Asian Language Resources (ALR6), Hyderabad, India, In this paper, we presented word January 2008. segmentation program for Myanmar text based on longest string matching algorithm and dictionary. Also we presented both word level and character level frequency distributions and word level Markov table generated by this program. The program performed segmentation work well and proved itself to be used as a practical word segmentation engine for various NLP applications, including Myanmar search engine (in particular word stemming engine). Statistical data generated by this program is useful as background information for designing various Myanmar NLP applications including input system etc. For future task, we plan to extend our program by collecting all possible Myanmar words including not only conversational words but also proper nouns. We

Statistical Analysis Of Myanmar Words On The World Wide Web For+ Search Engine Development+

by sianpu

Accessibility

Categories

Upload Details

Usage Rights

Statistics

Statistical Analysis Of Myanmar Words On The World Wide Web For+ Search Engine Development+ Document Transcript