The document presents a proposal for developing a Bangla tokenizer for part-of-speech tagging and token classification to address challenges in natural language processing for the Bengali language. It discusses the importance of tokenization and the complexities of Bangla grammar, emphasizing the need for comprehensive technology and standards to facilitate effective language processing. Future work aims to improve token classification and stemming to enhance the accuracy and efficiency of Bengali text processing.

1. Presented by-
Sujit Kumar Das
M.Tech 3rd sem,IT
Roll-021413 No-363202205
1
POS Tagging And Token
Classification By Using Bangla
TokenizerUnder the Supervision Of
Mr. Sourish Dhar
Asst. Professor,Dept of IT
Assam University

2. Contents…
2
 Introduction
 Literature Survey
 Our Proposal
 Future Works To Be Done
 Conclusions
 References

3. Introduction:
3
What is NLP?
 Field of computer science, artificial intelligence,
and linguistics concerned with the interactions
between computers and human (natural)
languages[1].
 NLP provides means of analyzing text .
 The goal of NLP is to make computers analyze
and understand the languages that humans use
naturally.

4. Cont…
4
Why Natural Language Processing?
Computers “see” text in English the same way we
use to see.
 People have no trouble understanding language
but computers have.
– No common sense knowledge.
– No reasoning capacity.

5. Cont…
5
What We Need In NLP Task?
 Knowledge about Language.
 Knowledge about world.
 A way to combine Knowledge sources.

6. Cont…
6
Mostly Solved Making Good
Progress
Still Really Hard
Spam Detection Sentiment Analysis Question
Answering
POS Tagging Word Sense
Disambiguation
Paraphrase
Named Entity
Recognition
Parsing Summarization
Machine
Translation
Dialog
Language Technology:

7. Cont…
7
POS Tagging:
Input: The grand jury commented on a number of
other topics.
Output: The/DT grand/JJ jury/NN commented/VBD
on/IN a/DT number/NN of/IN other/JJ topics/NNS ./.
NE Recognition:
Input: Dan went to London for attend a conference
on NLP in 2012.
Output: Dan went to London for attend a conference
on NLP in 2012.
Name Dan
Location London
Date 2010

8. Cont…
What Is Tokenization?
8
Tokenization is the process of breaking a stream of
text up into words, phrases, symbols and other
meaningful elements called tokens.
Token: It’s a sequence of character that can be
treated as a single
logical entity.
Typically Tokens are-Natural Languages Programming Languages
Words Identifiers
Numbers Keywords
Abbreviations Operators
Symbols Special symbols
Constants

9. Cont…
What Is Tokenizer?
9
The job of a Tokenizer is to break up a stream of text
into tokens.
Why Tokenizer?
 It does very crucial task in pre-processing any
natural language.
 To handle semantic issues in the subsequent stages
in machine translation.
 Produces a structural description on an input
sentence.
 For language modeling, the distribution of input text
into tokens is compulsory[9].

10. Cont…
10
What is Token Classification?
Tokens classification means identification of each
tokens(words/terms) in a document and classify them
into some predefined categories.
Theses predefined categories can be name of a
person, symbols, punctuations, Abbreviations,
numbers, date etc.

11. Cont…
Steps in Token Classification:
11
 Tokenize the given input text.
 Assign to each token the class (or tag) that it
belongs to.
For Example,
Token Class
মাইকেল Name
৪৫ Number
খবর Word

12. Cont…
12
Why Bengali Language Processing:
 One of the top ten spoken Language in the
world.
 Lack of research work till now.
Challenges In Bengali Language
Processing:
 Due to its Grammatical Vastness.
 Not well structured like Eastern Language(for
example English).

13. Cont…
13
Goals of Bengali Language Processing:
To develop technology and standards to make
computer usage Bangla enabled.
To establish standards for Bangla text processing to
ensure interoperability across platforms.
 To develop large standardized corpus for Bangla text
and speech.
 To create an ensemble of available Bangla software
and corpus in a standardized form and make them
easily available to all.
 To develop new software and modify or enhance the
existing software.
 To develop suitable speech Technology for Bangla.

14. Literature Survey:
14
 A Tokenizer is a component of parser . Parsing
natural language text is more difficult than the
computer languages such as compiler and word
processor because the grammars for natural
languages are complex, ambiguous and infinity
number of vocabulary[8].
 Natural language applications namely Information
Extraction, Machine Translation, and Speech
Recognition, need to have an accurate parser[8].
 A tokenizer plays its significant part in a parser, by
identifying the group or collection of words, existing
as a single and complex word in a sentence. Later
on, it breaks up the complex word into its

15. Cont…
Related Works:
15
Some Existing standard tokenizers-
 Standford Tokenizer for English Language[10].
 Shallow Tokenizer for Bengali Language.
 Vaakkriti Tokenizer for Sanskrit Language[2].
These Tokenizers was developed for some
particular languages only i.e., all Tokenizers doesn’t
work for all languages.

16. Cont…
Standford Tokenizer:
16
 Developed mainly for English Language and later
on for Arabic,Chinese and spanish languages also.
 Java language was used for developing.
Online Interface:

17. Cont…
Results after parsing:
17
S=sentence ,NP=Noun Phrase ,NNS=Noun Plural, VP=Verb Phrase,
VBZ=Verb,3rd present singular, VBN=Verb, past participle,
PP=Prepositional phrase ,TO=to, IN= Preposition or
subordinating conjunction.

18. Cont…
Shallow Bangla Tokenizer:
18
The shallow parser gives the analysis of a sentence in
terms of-
 Morphological Analysis.
 POS Tagging.
 Chunking.
Apart from the final output, intermediate output of
individual modules is also available.

19. Cont…
19
Online Interface:

20. Cont…
20
Result after submitting:

21. Cont…
21
Bengali Stemmers:
 A Rule-Based Stemmer for Bengali Language by
Sandipan Sarkar,IBM and Sivaji
Bandhopadhay,Jadavpur University[12].
 A light weight stemmer for Bengali and which was
use in spelling checker by Md. Zahurul Islam, Md.
Nizam Uddin and Mumit Khan,CRBLP,BRAC
University,Dhaka in 2007[13].
 Yet Another Suffix Stripper, which uses a clustering
based approach based on string distance
measures and requires no linguistic knowledge by
P.Majumdar, Gobinda Kole,ISI Pabitra Mitra,IIT and
Kalyankumar Dutta,Jadavpur University in

22. Cont…
22
Comparison Of Three stemmers:
Stemmer Used Method Accuracy(%)
Rule-Based Orthographic-
syllable
89.0
Light weight Longest Match
Basis
90.8
YASS String Distance
Measure
88.0

23. Cont…
23
POS Tagger:
 Supervised POS Tagging: Has pre-tagged
Corpora used for training to learn information
about the tagset, word-tag frequencies, rule sets
etc[11].
e.g., N-Gram,Maximum Entropy Model(ME),Hidden
Markov Model(HMM) etc.
 Unsupervised POS Tagging: Do not require a
pre-tagged corpora. they use advanced
computational methods to automatically induce
tagsets.
e.g.,Brill, Baum-Welch algorithm etc[11].

24. Cont…
24
Supervised Model POS Taggers
Comparison:Tagger Applied Method
Uni-Gram(N=1) Most likely approach
HMM One sentence at a
time. Formula-
P (word | tag) * P (tag | previous n
tags)
Bi-Gram(N=2) Same as Unigram but consider just
previous word tag

25. Cont…
25
UNI-GRAM BI-GRAM HMM
Sentences
Tokens Accuracy(%) Accuracy(%) Accuracy(%)
87 1002 28.6 28.6 39.3
304 4003 42.4 41.9 49.7
532 8026 48.1 47.9 53.6
677 10001 49.8 49.5 54.3
Bangla - SPSAL Corpus and Tagset with Test data: 400
sentences, 5225 tokens from the SPSAL test corpus[11].

26. Cont…
Problem Domain:
26
 Bangla is very rich in inflections, vibhakties (suffix)
and karakas, and often they are ambiguous also.
 It is not easy to provide necessary semantic and
world knowledge that we humans often use while
we parse and understand various Bangla
sentences.
So, mainly due to grammatical vastness design of
bangla Toeknizer is not an easy task.

27. Cont…
Bengali Grammar: POS
27

28. Cont…
Bengali Grammar: Genders
28
There are four genders in Bengali grammar -
1.Pung lingo(masculine)
2.Stree lingo(feminine)
3.Ubha lingo(common)
4.Klib lingo(material)

29. Cont…
Bengali Grammar: Numbers
29
Like English language Bengali has also two
numbers-
 Singular: When we define a single object or
person its singular.
eg. a man, a girl etc.
 When we consider more than one objects or
persons its plural numbers.
eg. Two man, mangoes etc.

30. Our Proposal:
30
We are going to develop such a system which can
be use for tokenize Bengali Text as well as the
system will be able to solve the problem of Tokens
Classification.
raw
(unstructured)
text
part-of-speech
tagging Token Classification
annotated
(structured)
text
Natural Language Processing
Fig: Our Model
Pre-
processing

31. Cont…
Flow Chart :
31
Input
Words
Stop Words
Removal
POS Tag
Classify
Text
Stemming

32. Cont…
32
Input:
Input will be a Bengali Text.
Words:(Completed)
Text will be split into words after removing all non-
character and white spaces and then store them into
excel file.
Stop Words Removal(Completed):
Stop words are the frequently occurring set of
words which do not aggregate relevant information to
the text classification task.
Root words:
After pulling out prefixes and suffixes from any
word thus the origin form of a word is known as root

33. Cont…
33
POS Tagging:
After finding the root word(stemming) each
elements will push into some particular classes
which is previously generated. Thus, Parts-Of-
Speech(POS) will be tagged with each word
here.
Tokens Classification:
Tokens classification means after finding
tokens from above tasks categories them into
some pre-defined classes.
Our consideration of classes will be mainly
Title,
Surname,Collocation,punctuation,Abbreviation,
Number,
Date, Unknown and foreign word.

34. Current Status Of Our Work:
34 Snapshot1: system Interface

35. Cont…
35
Snapshot 2: After Loading Using Load
Button

36. Cont…
36 Snapshot 3: After getting tokens from

37. Cont…
37
Snapshot4: Tokens after removing Stop-
words

38. Cont…
38
Snapshot3: After execution words are split and stored in excel file.

39. Future Works To Be Done:
39
 Stemming i.e., Finding Root Words.
 POS Tagging.
 Classification

40. Conclusions:
40
Although in Language processing tokenizing is
a Fundamental task, But due to richness of Bengali
grammar and structure of Bengali text it is not an
easy task in case of Bengali Language. Again
Stemming is also a difficult task to do. To make an
effective bangla Tokenizer one must have a vast
knowledge on Bengali Grammar. So, We hope that
we will able to develop such a system which will
overcome difficulties and the limitations of existing
bangla Tokenizer and give efficient Tokens and
finally we will able to classify the tokens.

41. References:
41
[1] Wikipedia
[2] Aasish Pappu and Ratna Sanyal “Vaakkriti:
Sanskrit Tokenizer”Indian Institute of Information
Technology, Allahabad (U.P.), India.
[3] Firoj Alam, S. M. Murtoza Habib, Mumit Khan
“Text Normalization system for Bangla” Center for
research on Bangla Language Processing,
Department of Computer Science and Engineering,
BRAC University, Bangladesh.
[4] Goutam Kumar Saha, “Parsing Bengali Text - an
Intelligent Approach” Scientist-F, Centre for
Development of Advanced Computing, (CDAC),
Kolkata.

42. Cont…
42
[5] “Magic of ASP.Net with C#” by Kumar Sanjeeb and
Shibi Panikkar.
[6] www.C-sharpcorner.com
[7] “Overview of Stemming Algorithms” Ilia Smirnov
http://the-smirnovs.org/info/stemming.pdf.
[8] “Recognizing Bangla grammar using predictive
parser”, by K. M. Azharul Hasan, Al-Mahmud, Amit
Mondal, Amit Saha. Department of Computer Science
and Engineering (CSE) Khulna University of
Engineering and Technology (KUET) Khulna-9203,
Bangladesh.
[9] “Model for Sindhi Text Segmentation into Word
Tokens” J. A. MAHAR, H. SHAIKH*, G. Q. MEMON
Faculty of Engineering, Science and Technology,

43. Cont…
43
[11] “COMPARISON OF DIFFERENT POS TAGGING
TECHNIQUES FOR SOME SOUTH ASIAN
LANGUAGES” by Fahim Muhammad Hasan, BRAC
University,Dhaka,Bangladesh.
[12] “Design of a Rule-based Stemmer for Natural
Language Text in Bengali”by Sandipan Sarkar IBM
India and Sivaji Bandyopadhyay Computer Science
and Engineering Department Jadavpur University,
Kolkata.
[13] “A Light Weight Stemmer for Bengali and Its Use in
Spelling Checker” by Md. Zahurul Islam, Md. Nizam
Uddin and Mumit Khan, Center for Research on
Bangla Language Processing, BRAC University,
Dhaka, Bangladesh.
[14] “Yet Another Suffix Stripper” by PRASENJIT
MAJUMDER, MANDAR MITRA, SWAPAN K. PARUI,

44. 44
Thank
You