Sujit Kumar Das
M.Tech 3rd sem,IT
Token Classification In Bengali
Language By Using Bangla
Under the Supervision Of
Mr. Sourish Dhar
Asst. Professor,Dept of IT
Future Works To Be Done
What is Token Classification?
Tokens classification means identification of each
tokens(words/terms) in a document and classify them
into some predefined categories.
Theses predefined categories can be name of a
person, symbols, punctuations, Abbreviations,
numbers, date etc.
Steps in Tokens Classification:
Tokenize the given input text.
Assign to each token the class (or tag) that it
What Is Tokenization?
Tokenization is the process of breaking a stream of
text up into words, phrases, symbols and other
meaningful elements called tokens.
Token: It’s a sequence of character that can be treated as
Typically TNoaktuernals L aanrgeu-ages Programming Languages
Symbols Special symbols
What Is Tokenizer?
The job of a Tokenizer is to break up a stream of text
It does very crucial task in pre-processing any
To handle semantic issues in the subsequent stages
in machine translation.
Produces a structural description on an input
For language modeling, the distribution of input text
into tokens is compulsory.
A Tokenizer is a component of parser . Parsing
natural language text is more difficult than the
computer languages such as compiler and word
processor because the grammars for natural
languages are complex, ambiguous and infinity
number of vocabulary.
Natural language applications namely Information
Extraction, Machine Translation, and Speech
Recognition, need to have an accurate parser.
A tokenizer plays its significant part in a parser, by
identifying the group or collection of words, existing
as a single and complex word in a sentence. Later
on, it breaks up the complex word into its
constituents in their appropriate forms.
Some Existing standard tokenizers-
Standford Tokenizer for English Language.
Shallow Tokenizer for Bengali Language.
Vaakkriti Tokenizer for Sanskrit Language.
These Tokenizers was developed for some
particular languages only i.e., all Tokenizers doesn’t
work for all languages.
Developed mainly for English Language and later
on for Arabic,Chinese and spanish languages also.
Java language was used for developing.
Shallow Bangla Tokenizer:
The shallow parser gives the analysis of a sentence in
Apart from the final output, intermediate output of
individual modules is also available.
A Rule-Based Stemmer for Bengali Language by
Sandipan Sarkar,IBM and Sivaji
A light weight stemmer for Bengali and which was
use in spelling checker by Md. Zahurul Islam, Md.
Nizam Uddin and Mumit Khan,CRBLP,BRAC
University,Dhaka in 2007.
Yet Another Suffix Stripper, which uses a clustering
based approach based on string distance
measures and requires no linguistic knowledge by
P.Majumdar, Gobinda Kole,ISI Pabitra Mitra,IIT and
Kalyankumar Dutta,Jadavpur University in
Comparison Of Three stemmers:
Stemmer Used Method Accuracy(%)
Light weight Longest Match
YASS String Distance
Supervised POS Tagging: Has pre-tagged
Corpora used for training to learn information
about the tagset, word-tag frequencies, rule sets
e.g., N-Gram,Maximum Entropy Model(ME),Hidden
Markov Model(HMM) etc.
Unsupervised POS Tagging: Do not require a
pre-tagged corpora. they use advanced
computational methods to automatically induce
e.g.,Brill, Baum-Welch algorithm etc.
Supervised POS Taggers Comparison:
Tagger Applied Method
Uni-Gram(N=1) Most likely approach
HMM One sentence at a
P (word | tag) * P (tag | previous n
Bi-Gram(N=2) Same as Unigram but consider just
previous word tag
UNI-GRAM BI-GRAM HMM
Tokens Accuracy(%) Accuracy(%) Accuracy(%)
87 1002 28.6 28.6 39.3
304 4003 42.4 41.9 49.7
532 8026 48.1 47.9 53.6
677 10001 49.8 49.5 54.3
Bangla - SPSAL Corpus and Tagset with Test data: 400
sentences, 5225 tokens from the SPSAL test corpus.
Bangla is very rich in inflections, vibhakties (suffix)
and karakas, and often they are ambiguous also.
It is not easy to provide necessary semantic and
world knowledge that we humans often use while
we parse and understand various Bangla
So, mainly due to grammatical vastness design of
bangla Toeknizer is not an easy task.
Bengali Grammar: Genders
There are four genders in Bengali grammar -
Bengali Grammar: Numbers
Like English language Bengali has also two
Singular: When we define a single object or
person its singular.
eg. a man, a girl etc.
When we consider more than one objects or
persons its plural numbers.
eg. Two man, mangoes etc.
We are going to develop such a system which can
be use for tokenize Bengali Text as well as the
system will be able to solve the problem of Tokens
Microsoft Excel Stylsheet
Flow Chart :
Input will be a Bengali Text.
Text will be split into words after removing all non-character
and white spaces and then store them into
Stop Words Removal(Done):
Stop words are the frequently occurring set of
words which do not aggregate relevant information to
the text classification task.
After pulling out prefixes and suffixes from any
word thus the origin form of a word is known as root
After finding the root word(stemming) each
elements will push into some particular classes
which is previously generated. Thus, Parts-Of-
Speech(POS) will be tagged with each word
Tokens classification means after finding
tokens from above tasks categories them into
some pre-defined classes.
Our consideration of classes will be mainly
Date, Unknown and foreign word.
Current Status Of Our Work:
27 Snapshot1: system Interface
Snapshot 2: After Loading Using Load
29 Snapshot 3: After getting tokens from
Snapshot4: Tokens after removing Stop-words
Snapshot3: After execution words are split and stored in excel file.
Future Works To Be Done:
Stemming i.e., Finding Root Words.
Although in Language processing tokenizing is
a Fundamental task, But due to richness of Bengali
grammar and structure of Bengali text it is not an
easy task in case of Bengali Language. Again
Stemming is also a difficult task to do. To make an
effective bangla Tokenizer one must have a vast
knowledge on Bengali Grammar. So, We hope that
we will able to develop such a system which will
overcome difficulties and the limitations of existing
bangla Tokenizer and give efficient Tokens and
finally we will able to classify the tokens.
 Aasish Pappu and Ratna Sanyal “Vaakkriti:
Sanskrit Tokenizer”Indian Institute of Information
Technology, Allahabad (U.P.), India.
 Firoj Alam, S. M. Murtoza Habib, Mumit Khan
“Text Normalization system for Bangla” Center for
research on Bangla Language Processing,
Department of Computer Science and Engineering,
BRAC University, Bangladesh.
 Goutam Kumar Saha, “Parsing Bengali Text - an
Intelligent Approach” Scientist-F, Centre for
Development of Advanced Computing, (CDAC),
 “Magic of ASP.Net with C#” by Kumar Sanjeeb and
 “Overview of Stemming Algorithms” Ilia Smirnov
 “Recognizing Bangla grammar using predictive
parser”, by K. M. Azharul Hasan, Al-Mahmud, Amit
Mondal, Amit Saha. Department of Computer Science
and Engineering (CSE) Khulna University of
Engineering and Technology (KUET) Khulna-9203,
 “Model for Sindhi Text Segmentation into Word
Tokens” J. A. MAHAR, H. SHAIKH*, G. Q. MEMON
Faculty of Engineering, Science and Technology,
Hamdard University, Karachi.
 “COMPARISON OF DIFFERENT POS TAGGING
TECHNIQUES FOR SOME SOUTH ASIAN
LANGUAGES” by Fahim Muhammad Hasan, BRAC
 “Design of a Rule-based Stemmer for Natural
Language Text in Bengali”by Sandipan Sarkar IBM
India and Sivaji Bandyopadhyay Computer Science
and Engineering Department Jadavpur University,
 “A Light Weight Stemmer for Bengali and Its Use in
Spelling Checker” by Md. Zahurul Islam, Md. Nizam
Uddin and Mumit Khan, Center for Research on
Bangla Language Processing, BRAC University,
 “Yet Another Suffix Stripper” by PRASENJIT
MAJUMDER, MANDAR MITRA, SWAPAN K. PARUI,
and GOBINDA KOLE Indian Statistical Institute.