This document summarizes a research paper that proposes enhancing word embedding models with subword information by incorporating character n-grams. It finds that a skip-gram model augmented with character n-grams outperforms traditional skip-gram and CBOW models on word similarity and analogy tasks, especially for morphologically rich languages. Experiments show the character n-gram approach is more robust to the size of training data and helps learn better representations for rare and out-of-vocabulary words.

1. Enriching Word Vectors with
Subword Information
Piotr Bonjanowski, Edouard Grave, Armand Joulin,
Tomas Mikolov
Presented by Jaeyeun Yoon
On 2017-07-20

2. IDS Lab.
Word Embedding
 Word Embedding
• Word embedding is the collective name for a set of language modeling
and feature learning techniques where words or phrases from the
vocabulary are mapped to vectors of real numbers in a dimensional
space.
• Ex) Bag-of-words model, Latent semantic analysis(LSA), word2vec, GloVe
…
 Distributional Hypothesis
• “A word is characterized by the company it keeps”
[Firth, J.R. (1957)]
• Motivated by the idea that closely co-occuring words define the
semantics of the word
• Most of word embedding models are based on distributional hypothesis.
”The cat sat on the mat”
W("cat") = (0.2,−0.4,0.7,...)
W("mat") = (0.0,0.6,−0.1,...)

3. IDS Lab.
Characteristics of word embedding
• Once trained they can be used in various tasks
• Capture relative semantics of words
• Can be used to combat small dataset problems

4. IDS Lab.
Skip-Gram Model(general model)
• Mikolov, Tomas, et al. "Efficient estimation of
word representations in vector space." arXiv
preprint arXiv:1301.3781 (2013)
• Classifier for predicting words within
certain range of input word
• e.g.) … “It is time to finish”
• The objective of the skipgram model is
to maximize the following log-
likelihood:
𝒕=𝟏
𝑻
𝒄∈𝑪 𝒕
𝒍𝒐𝒈 𝒑 𝒘 𝒄 𝒘 𝒕)
𝑪𝒕 ∶ 𝑻𝒉𝒆 𝒔𝒆𝒕 𝒐𝒇 𝒊𝒏𝒅𝒊𝒄𝒆𝒔 𝒐𝒇 𝒘𝒐𝒓𝒅𝒔 𝒔𝒖𝒓𝒓𝒐𝒖𝒏𝒅𝒊𝒏𝒈 𝒘𝒐𝒓𝒅 𝒘 𝒕
𝒑 𝒘 𝒄 𝒘 𝒕) ∶ 𝑻𝒉𝒆 𝒑𝒓𝒐𝒃𝒂𝒃𝒊𝒍𝒊𝒕𝒚 𝒐𝒇 𝒐𝒃𝒔𝒆𝒓𝒗𝒊𝒏𝒈 𝒂 𝒄𝒐𝒏𝒕𝒆𝒙𝒕 𝒘𝒐𝒓𝒅 𝒘 𝒄 𝒈𝒊𝒗𝒆𝒏 𝒘 𝒕

5. IDS Lab.
Limitation of Skip-Gram
5
 It is difficult for good representations of rare words to
be learned with traditional word2vec.
 There could be words in the NLP task that were not
present in the word2vec training corpus
- This limitation is more pronounced in case of morphologically rich
languages
EX) In French or Spanish, most verbs have more than forty different
inflected froms. While the Finnish languages has fifteen cases for
nouns
-> It is possible to improve vector representations for
Morphologically rich languages by using character
level information.

6. IDS Lab.
Example
6
 German verb : ‘sein’ (English verb : ‘be’)

7. IDS Lab.
Limitation of Skip-Gram
7
 It is difficult for good representations of rare words to
be learned with traditional word2vec.
 There could be words in the NLP task that were not
present in the word2vec training corpus
- This limitation is more pronounced in case of morphologically rich
languages
EX) In French or Spanish, most verbs have more than forty different
inflected froms. While the Finnish languages has fifteen cases for
nouns
-> It is possible to improve vector representations for
Morphologically rich languages by using character
level information.

8. IDS Lab.
• The basic skip-gram model described above ignores the internal
structure of the word.
• However, character n-gram based model incorporates information
about the structure in terms of character n-gram embeddings.
• This paper suppose that each word 𝑤 is represented as a bag of
character n-gram.
EX) where / n = 3, it will be represented by the
character n-grams :
<wh / whe / her / ere / re>
And the special sequence
<where>
Character n-gram based model
8
word <her> is different from the
tri-gram her from the word where
where

9. IDS Lab.
• We represent a word by the sum of the vector representations of
its n-gram. We thus obtain the scoring function :
• We extract all the 𝑛 -grams. (3 ≤ 𝑛 ≤ 6)
Character n-gram based model (cont’d)
9
𝑠 𝑤, 𝑐 = g∈ς 𝑤
𝑧 𝑔
𝑇 𝑣𝑐
𝑤 ∶ 𝑔𝑖𝑣𝑒𝑛 𝑤𝑜𝑟𝑑
ς 𝑤 ∶ the set of n − grams appearing in word 𝑤
𝑧 𝑔 ∶ 𝑣𝑒𝑐𝑡𝑜𝑟 𝑟𝑒𝑝𝑟𝑒𝑠𝑒𝑛𝑡𝑎𝑡𝑖𝑜𝑛 𝑡𝑜 𝑒𝑎𝑐ℎ 𝑛 − 𝑔𝑟𝑎𝑚𝑠
𝑣𝑐 ∶ 𝑡ℎ𝑒 𝑤𝑜𝑟𝑑 𝑣𝑒𝑐𝑡𝑜𝑟 𝑜𝑓 𝑡ℎ𝑒 𝑐𝑒𝑛𝑡𝑒𝑟 𝑤𝑜𝑟𝑑 𝐶

10. IDS Lab.
Computing word vector representation
10

11. IDS Lab.
Experiments Settings
 Target Languages
• German / English / French / Spanish /
Arabic / Romanian / Russian / Czech
 Kind of tasks
1. Human similarity judgement
2. Word analogy tasks
3. Comparison with morphological representations
4. Effect of the size of the training data
5. Effect of the size of n-grams
11

12. IDS Lab.
1. Human similarity judgement
 Correlation between human judgement and similarity scores on word
similarity datasets.
12
sg : Skip-Gram
cbow : continuous bag of words
sisg- : Subword Information Skip-Gram
(Treat unseen words as a null vector)
sisg : Subword Information Skip-Gram
(Treat unseen words by summing the n-gram vectors)
RW : Rare Words dataset

13. IDS Lab.
2. Word analogy tasks
 Accuracy of our model and baselines on word analogy tasks for Czech,
German, English and Italian
13
* It is observed that morphological
information significantly improves the
syntactic tasks; our approach outperforms
the baselines. In contrast, it does not help for
semantic questions, and even degrades the
performance for German and Italian.

14. IDS Lab.
3. Comparison with morphological representations
 Spearman’s rank correlation coefficient between human judgement and
model scores for different methods using morphology to learn word
representations.
14

15. IDS Lab.
4. Effect of the size of the training data
 Influence of size of the training data on performance
(Data : full Wikipedia dump / Task : 1. similarity task)
 Sisg model is more robust to the size of the training data.
 However, the performance of the baseline cbow model gets better as more and more data is
available. Sisg model, on the other hand, seems to quickly saturate and adding more data
does not always lead to improved result.
 It is observed the performance sisg with very small dataset better than the performance
15

16. IDS Lab.
5. Effect of the size of n-grams
 The choice of n boundary ​​is observed to be language and task
dependent.
 Results are always improved by taking 𝑛 ≥ 3 rather than 𝑛 ≥ 2, which
shows that character 2-games arenot informative for that task
16
Minimum value of n
Maximum value of n

17. IDS Lab.
Conclusion
 General Skip-Gram model has some limitations. (Ex: OOV)
 But, This can be overcome by using subword information
(character n-grams).
 This model is simple. Because of simplicity, this model trains fast
and does not require any preprocessing or supervision.
 It works better for certain languages. (Ex: German)
17