This document presents a maximum-length comparison method for automatic word segmentation of Myanmar text. It consists of four main steps: (1) preprocessing the sentences, (2) detecting the first character (consonant), (3) extracting candidate words from a data dictionary by truncating the input based on word lengths, and (4) comparing word lengths to select the candidate with the maximum length as the segmented word. The method was tested on a dataset of 30147 sentences, correctly segmenting 21577 words (92% accuracy). Future work could involve increasing the data dictionary and analyzing word meanings to improve segmentation for natural language processing tasks.

1. Maximum-Length Comparison Method
Of Automatic Word Segmentation
for Myanmar Text
Htet Myet Lynn, Pankoo Kim, Junho Choi, Jenogin Kim
CAIPT2015 – June 23, 2015 CAIPT2015

2. What is NLP?
CAIPT2015
Natural
Language

3. Why Word Segmentation?
CAIPT2015

4. U n i v e rs i t y LO G O
Contents
1 Nature of Myanmar Script
CAIPT2015
2 Maximum-Length Comparsion Model
3 Experimental Result & Future Study

5. Nature Of Myanmar Script
CAIPT2015

6. Nature of Myanmar Script
CAIPT2015
Consonants
Digits

7. Nature of Myanmar Script
CAIPT2015
Consonants
Basic Vowels

8. Nature of Myanmar Script
CAIPT2015
Consonants
Consonant Combination Symbols

9. Nature of Myanmar Script
CAIPT2015
Consonants
Devowelization Consonants

10. Nature of Myanmar Script
CAIPT2015
 Lack of standard rules for distinct word delimiter (white-space) between words
become challenge
He is having a meal with three nephews.
He is nephew three persons with meal eating

11. Maximum-Length Comparison Model
CAIPT2015
1 Preprocessing Sentences
2 Detect First Character (Consonant)
3 Candidates Detection & Extraction
4 Maximum-Length Comparison

12. Preprocessing Sentences
CAIPT2015
Input Text
Preprocessing
Detect Consonant
Data Dictionary
Cadidate
Extraction
Candidates.txt
Maximum Length
Comparison
Output Result

13. Preprocessing Sentences
CAIPT2015
He joins the army.Input:
Preprocessing:
 Each and every news media uses different style of writing and positioning white-
space in a sentence
 Remove punctuation marks, white-spaces

14. Detect First Character (Consonant)
CAIPT2015
Input Text
Preprocessing
Detect Consonant
Data Dictionary
Cadidate
Extraction
Candidates.txt
Maximum Length
Comparison
Output Result

15. Detect First Character (Consonant)
CAIPT2015
Preprocessing:
1st Character Detection:
He joins the army.
He
Get Consonant:

16. Candidates Detection & Extraction
CAIPT2015
Input Text
Preprocessing
Detect Consonant
Data Dictionary
Cadidate
Extraction
Candidates.txt
Maximum Length
Comparison
Output Result

17. Candidates Detection & Extraction
CAIPT2015
Consonant:
Data
Dictionary
Candidates.txt
Let,
Length of word_#1 = 3;
Length of word_#2 = 5;
.
.
Length of word_#10= 20;
Truncate the input_sentence with the value of word_#n;
If (word_#n == truncate_word) {
mark_as_candidate;
} else
{
ignore();
}
1.
10.

18. Maximum-Length Comparison
CAIPT2015
Input Text
Preprocessing
Detect Consonant
Data Dictionary
Cadidate
Extraction
Candidates.txt
Maximum Length
Comparison
Output Result

19. Maximum-Length Comparison
CAIPT2015
1.
10.
Candidates.txt
If
Length of candidate_#1 = 3;
Length of candidate_#10= 20;
//Get the word with longest value among candidates
best_candidate = candidate_#10;
final_word = best_candidate;
Truncate the value of best_candidate from input;
Input:
He joins the army.
New input:

20. Maximum-Length Comparison Model
CAIPT2015
Input Text
Preprocessing
Detect Consonant
Data Dictionary
Cadidate
ExtractionCandidates.
txt
Maximum Length
Comparison
Output Result
While (length_input_sent <= 0)

21. Experimental Result & Future Study
CAIPT2015

22. Experimental Result
CAIPT2015

23. Future Study
CAIPT2015
 30147 sentences including a total of (23,454 words) have been tested
 21577 words out of 23,454 words are aright (92%)
 Error can be occurred according to the shortage of data dictionary, technical terms
and new derived words
 Increase the value of data dictionary
 Understand the meaning of segmented word semantically for further NLP tasks

24. Do You Have
any Questions?
CAIPT2015