The slides present the steps that have been followed in order to build a POS Tagger for Romanian language for a special kind of text: chats. We show the main differences that can be observed between chats and novel writings and present in a comparative way the results obtained using a chat trained model and a novel-trained one when tagging a corpus of chats.

1. Autor Conducător științific
Universitatea
Politehnica
București
Facultatea de
Automatică și
Calculatoare
Catedra de
Calculatoare
Chat-Adapted POS Tagger for Romanian
Language
Costin-Gabriel Chiru - costin.chiru@cs.pub.ro
Traian Rebedea
Mădălina Ioni ăț

2. Contents
• Introduction
• What is a POS tagger
• How does a POS tagger work
• Chat versus novel
• Chat model for POS tagger
• Results
• Conclusions and further developing
14.09.2010 1
The Second Workshop on Natural Language Processing in
Support of Learning: Metrics, Feedback and Connectivity

3. Introduction
• Purpose: to build a Part-of-speech (POS) Tagger for
Romanian that can be used for tagging the words
from a special type of texts: chats.
• Methodology: for this task, we used the Hidden
Markov Models paradigm to “learn” a model of the
POS of different words from the chat. The tagging
has been done using Viterbi – a dynamic
programming algorithm.
2
The Second Workshop on Natural Language Processing in
Support of Learning: Metrics, Feedback and Connectivity
14.09.2010

4. What is a POS tagger
• “The act of assigning each word in a sentence a
tag that describes how that word is used in the
sentence. Typically, these tags indicate syntactic
categories, such as noun or verb, and
occasionally include additional feature
information, such as number (singular or plural)
and verb tense.” (Thede & Parker, 1999)
• Difficult task - many words are polysemous and
can be associated with multiple POS. (Eg: book)
3
The Second Workshop on Natural Language Processing in
Support of Learning: Metrics, Feedback and Connectivity
14.09.2010

5. How does a POS tagger work
• The POS tagger „learns” a model –
represented by some probabilities from an
annotated corpus;
– Best solution is HMM: M = (π, A, B,), where π
-initial state probabilities, A - the transition matrix
and B - the confusion matrix.
• Depending on these, the POS tagger decides
the most suitable sequence of tags for a new
text that has to be tagged.
– Best solution is decoding using Viterbi algorithm.
4
The Second Workshop on Natural Language Processing in
Support of Learning: Metrics, Feedback and Connectivity
14.09.2010

6. How does a POS tagger work (2)
• 3 steps:
– Tokenization: pre-processing - splitting the string
into tokens;
– Ambiguity look-up – uses a lexicon and a tag set to
assign each word a list of possible POS tags. The
words that are missing from the lexicon are
evaluated based on the transition matrix of the
HMM;
– Disambiguation - “choose” only one tag from the
possible tag set found for every word in order to
maximize the probability of the whole sequence.
5
The Second Workshop on Natural Language Processing in
Support of Learning: Metrics, Feedback and Connectivity
14.09.2010

7. Chat versus Novels
• The POS taggers built until today (en: TnT tagger,
TreeTagger, Stanford Tagger, Qtag; ro:
http://www.cs.ubbcluj.ro/~dtatar/nlp/WebTagger/W
) obtained around 96-97% accuracy on regular
text.
• However, accuracy drops when applied to
different kind of texts – chats.
• For a good tagging, the model that is learnt has
to be similar to the one that is applied to.
6
The Second Workshop on Natural Language Processing in
Support of Learning: Metrics, Feedback and Connectivity
14.09.2010

8. Chat versus Novels (2)
• How chats differ from regular text:
– the massive use of abbreviations: brb, cu, lol, gtg, afk;
– the use of emoticons: , , :s, ;), :P;
– punctuation marks are often not respected: i told
john i go to him after school i’m sorry i wasn’t able to
do it;
– capitalized letter at the beginning of a sentence or
name are rarely used: i, john;
– much more misspelled words are encountered;
– diacritics are very rarely used;
– utterance ≠ sentence;
– usage of foreign language words – especially English.
7
The Second Workshop on Natural Language Processing in
Support of Learning: Metrics, Feedback and Connectivity
14.09.2010

9. Chat model for POS tagger
• The need to have an annotated corpus of chats
to be used as model for the chats.
• Start from an annotated corpus for novels – “1984”, by
George Orwell – having 154 tags taken from
http://www.racai.ro/books/awde/tufiscor.html to build
the new model in a semi-supervised manner:
• The model has been applied to tag a chat corpus (5 chats, ≈700
utterances, ≈ 15.000 words);
• The wrong tags have been manually corrected;
• The corpus of tagged chats has been used as the model for
chats annotation;
• The lexicon contained 2129 words.
8
The Second Workshop on Natural Language Processing in
Support of Learning: Metrics, Feedback and Connectivity
14.09.2010

10. 23.07.2010 10

11. Results
• Testing has been made using another corpus of
chats (4 chats, ≈500 utterances, ≈ 12.000 words).
• Both models built using the chat model and the
model built based on the “1984” novel have been
tested.
• The results have shown an increase of precision
of 10.6% and of recall of 6.7% when using the
chat model instead of the novel model.
10
The Second Workshop on Natural Language Processing in
Support of Learning: Metrics, Feedback and Connectivity
14.09.2010

12. Precision
11
Tag Novel Model
Precision (%)
Chat Model
Precision (%)
Improvement
(value and %)
Overall 66 73 7 (10.6 %)
Nouns 54 69 15 (27.7 %)
Numerals 66 69 3 (4.54 %)
Adjectives 59 53 -6 (-10.1 %)
Pronouns 66 67 1 (1.5 %)
Verbs 68 77 9 (13.2 %)
Abbreviations 35 81 46 (131.4 %)
Prepositions 85 82 -3 (-3.5 %)
Conjunctions 68 91 23 (33.8 %)
Prefixes 33 50 17 (51.5 %)
Articles 70 83 13 (18.5 %)
Adverbs 62 86 24 (38.7 %)
Interjections 56 89 33 (58.9 %)
Auxiliary 63 76 13 (20.6 %)
The Second Workshop on Natural Language Processing in
Support of Learning: Metrics, Feedback and Connectivity
14.09.2010

13. Recall
12
Tag Novel Model
Recall (%)
Chat Model
Recall (%)
Improvement
(value and %)
Overall 59 63 4 (6.7 %)
Nouns 48 54 6 (12.5 %)
Numerals 66 43 -23 (-34.8 %)
Adjectives 48 56 8 (16.6 %)
Pronouns 61 63 2 (3.2 %)
Verbs 60 61 1 (1.6 %)
Abbreviations 26 58 32 (123 %)
Prepositions 82 90 8 (9.7 %)
Conjunctions 68 84 16 (23.5 %)
Prefixes 100 100 0 (0 %)
Articles 63 73 10 (15.8 %)
Adverbs 80 77 -3 (-3.7 %)
Interjections 66 34 -32 (-48.4 %)
Auxiliary 54 52 -2 (-3.7 %)
The Second Workshop on Natural Language Processing in
Support of Learning: Metrics, Feedback and Connectivity
14.09.2010

14. Evolution of precision and recall
13
The evolution of precision and recall for each part
of speech
-100
-50
0
50
100
150
generally
nounnum
eral
adjectivepronoun
verb
abbreviation
preposition
conjunction
prefix
articleadverb
interjectionauxiliary
Precision
Recall
The Second Workshop on Natural Language Processing in
Support of Learning: Metrics, Feedback and Connectivity
14.09.2010

15. Conclusions
• Abbreviation tags received the most significant increase of precision and
recall due to their intensive use compared with the novels.
• Improvement for nouns and verbs, which shows the use of this POS in the
chats to ease the information exchange;
• A significant improvement is registered for conjunctions because there
are more contacts between utterances than in a novel.
• The interjections have known a big progress which shows that the
participants use them very often in order to transmit certain signals.
• The parentheses have encountered an improvement because users tend
to use emoticons for expressing themselves. The other punctuation marks
have not encountered a significant improvement because they are not
used so much in chats.
14
The Second Workshop on Natural Language Processing in
Support of Learning: Metrics, Feedback and Connectivity
14.09.2010

16. Possible Improvements
• Annotating more chats to be used as the chat
model.
• Using 2nd
order HMM instead of 1st
order like in
the actual form (trigrams instead of bigrams).
• Building a mapping from our tag set to the one
used by POS tagger built at the Babes-Bolyai
University in order to better evaluate the
performances.
15
The Second Workshop on Natural Language Processing in
Support of Learning: Metrics, Feedback and Connectivity
14.09.2010

17. Q&A
Thank you for your time!
The Second Workshop on Natural Language Processing in
Support of Learning: Metrics, Feedback and Connectivity
14.09.2010