Mikhail Korobov - Morphological Analyzer and Generator for Russian and Ukrainian Languages

1. Morphological Analyzer
and Generator for Russian
and Ukrainian Languages
Mikhail Korobov
AIST 2015

2. Morphological Analysis:
word -> possible grammatical tags
• стали: VERB,perf,intr plur,past,indc
(ГЛ,сов,неперех мн,прош,изъяв);
• стали: NOUN,inan,femn sing,[nomn;gent;datv;loct]
(СУЩ,неод,жр [ед,рд;ед,дт;eд,пр;мн,им;мн,вн])
• бутявка: NOUN,inan,femn sing,nomn
(СУЩ,неод,жр ед,им)

3. Moprhological Generation
• lemmatization: стали -> стать, ежом -> ёж
• inflection: стали -> (sing,3per,fut) -> станет
• inflection: ёж -> (datv) -> ежу

4. pymorphy2: features
• Morphological analysis of Russian words;
• morphological generation: lemmatization, inflection,
number agreement;
• P(tag | word) estimates;
• out-of-vocabulary words handling;
• experimental support for Ukrainian language.

5. pymorphy2: implementation
• Python library and a command line tool
• Permissive open-source license: MIT for code,
Creative Commons BY-SA for data
• 600+ unit tests; 90%+ test coverage
• Memory usage: 30MB = 15MB pymorphy2 + 15MB
Python interpreter
• Speed: 20-100K words per second with an optional
C++ extension

6. Analysis of Vocabulary
Words
• OpenCorpora dictionary for Russian (5M word
forms, 400K lemmas);
• a dictionary based on LanguageTool data (2.5M
word forms) by Andrey Rysin, Dmitry Chaplinsky,
Mariana Romanyshyn, Vladimir Sevastyanov &
others.

7. Analysis of Vocabulary
Words
Source dictionaries provide lexemes:
ёж NOUN,anim,masc sing,nomn
ежа NOUN,anim,masc sing,gent
ежу NOUN,anim,masc sing,datv
...
ежами NOUN,anim,masc plur,ablt
ежах NOUN,anim,masc plur,loct

8. Tasks
• Analyze: get a word from dictionary, return its tag
• Lemmatize: find a word in dictionary, get 1st word
from its lexeme
• Inflect: find a word in dictionary, get a compatible
word from its lexeme

9. Efficiency considerations
• OpenCorpora XML dictionary is 400MB on disk
• XML search lookup is O(N)
• When loaded to an in-memory hash table (Python
dict) dictionary takes several GB of RAM

10. Solution
• Extract paradigms from lexemes; encode words as
DAFSA.
• Also tried: succinct tries, two double-array tries
• 5M Russian word forms in DAFSA == 3MB RAM

11. Lexeme
word tag
хомяковый ADJF,Qual masc,sing,nomn
хомякового ADJF,Qual masc,sing,gent
...
хомяковы ADJS,Qual plur
хомяковее COMP,Qual
хомяковей COMP,Qual V-ej
похомяковее COMP,Qual Cmp2
похомяковей COMP,Qual Cmp2,V-ej

12. Lexeme
prefix stem suffix tag
хомяков ый ADJF,Qual masc,sing,nomn
хомяков ого ADJF,Qual masc,sing,gent
...
хомяков ы ADJS,Qual plur
хомяков ее COMP,Qual
хомяков ей COMP,Qual V-ej
по хомяков ее COMP,Qual Cmp2
по хомяков ей COMP,Qual Cmp2,V-ej

13. Paradigm
prefix suffix tag
ый ADJF,Qual masc,sing,nomn
ого ADJF,Qual masc,sing,gent
...
ы ADJS,Qual plur
ее COMP,Qual
ей COMP,Qual V-ej
по ее COMP,Qual Cmp2
по ей COMP,Qual Cmp2,V-ej

14. Paradigm, encoded
prefix_id suffix_id tag_id
0 66 78
0 67 79
...
0 37 94
0 82 95
0 121 96
1 82 97
1 121 98

15. DAFSA
10
14
0
2
3
1
16
4 6
32
И
sep
7
22
sep
8 9
sep
И
13
103
12103
102
2
2
0
17
104
2
(word, paradigm_id, form_index) triples:
(двор, 103, 0); (ёж, 104, 0);
(дворник, 101, 2); (дворник, 102, 2);
(ёжик, 101, 2); (ёжик, 102, 2)

16. Out of Vocabulary
Words

17. Common prefixes removal: language-
specific lists of common immutable
prefixes (e.g. "не", "псевдо")
• недопсевдоавиашоу == недо + псевдоавиашоу
• псевдоавиашоу == псевдо + авиашоу
• авиашоу == авиа + шоу
• шоу - a known word

18. Words Ending with Other Dictionary Words
Example: котопсина
• a word being analyzed has another word from a
dictionary as a suffix;
• the length of this "suffix" word is no less than 3;
• the length of the word without the "suffix" is no
greater than 5;
• "suffix" word is of an open class (noun, verb,
adjective, participle, gerund)

19. Endings Matching
Example: бурбуляторовый
• words with common endings often have the same
grammatical form
• pymorphy2 builds an index of all 1-5 char word
endings and their analyses
• (frequency, paradigm_id, form_index) triple is
stored for each ending

20. Words with a Hyphen
• adverbs with a hyphen: по-хорошему
• particles separated by a hyphen: смотри-ка
• compound words: интернет-магазин, человек-
паук

21. P(tag | word) estimation
• Based on partially disambiguated OpenCorpora
data;
• MLE with Laplace smoothing

22. Evaluation: bad ideas
• evaluate pymorphy2 on OpenCorpora data
• evaluate Mystem on ruscorpora.ru (НКРЯ) data

23. Evaluation Setup
• pymorphy2 and Mystem 3.0;
• 100 randomly selected sentences from OpenCorpora
("microcorpus");
• 100 randomly selected sentences from ruscorpora.ru;
• tagsets are different; evaluation requires complicated
tag matching and manual checking of all errors;
• available online (http://goo.gl/BNXQXf)

24. Evaluation: errors
(full grammatical tags, recall, errors in
hyphenated words are not considered errors)
0
7,5
15
22,5
30
pymorphy2 Mystem 3.0
8
9
15
10
microcorpus ruscorpora

25. Evaluation: errors
0
3,5
7
10,5
14
Abbreviations People Names Regular Words Other Hyphenated Words*
11
2
6
1
14
02
44
9
pymorphy2 Mystem 3.0

26. Evaluation: results
• Both pymorphy2 and mystem made less than 1%
errors (without disambiguation); most errors are in
special cases.
• Hard to draw a conclusion; interpretation of
evaluation results is important.
• 6 errors in ruscorpora.ru gold results are found by
parsing it with pymorphy2, 1 error in microcorpus
gold results is found by parsing it with mystem.

27. Future work
• Improve people names, abbreviations, hyphenated words
parsing;
• improve non-contextual P(tag|word) estimates;
• improve Ukrainian language support;
• add Belarusian language support;
• there is a room for speed improvements;
• nicer command-line utility;
• ideas?

28. You can help
https://github.com/kmike/pymorphy2