1. Language Model
自然言語処理シリーズ 4 機械翻訳 pp.62-80
Koichi Akabe
MT study
NAIST
2014-05-08
2014-05-08 Koichi Akabe (NAIST MT) 1 / 20

2. Fluency of Machine Translation
Machine Translation: f −→ e
2014-05-08 Koichi Akabe (NAIST MT) 2 / 20

3. Fluency of Machine Translation
Machine Translation: f −→ e
Which translation e is correct?
▶ e1 = he is big
▶ e2 = is big he
▶ e3 = this is a purple dog
2014-05-08 Koichi Akabe (NAIST MT) 2 / 20

4. Fluency of Machine Translation
Machine Translation: f −→ e
Which translation e is correct?
▶ e1 = he is big
▶ e2 = is big he
▶ e3 = this is a purple dog
2014-05-08 Koichi Akabe (NAIST MT) 2 / 20

5. Fluency of Machine Translation
Machine Translation: f −→ e
Which translation e is correct?
▶ e1 = he is big
▶ e2 = is big he
▶ e3 = this is a purple dog
We can know the answer without f
2014-05-08 Koichi Akabe (NAIST MT) 2 / 20

6. Fluency of Machine Translation
Machine Translation: f −→ e
Which translation e is correct?
▶ e1 = he is big
▶ e2 = is big he −→ Syntax broken
▶ e3 = this is a purple dog −→ We have never seen
We can know the answer without f
2014-05-08 Koichi Akabe (NAIST MT) 2 / 20

7. Language model (LM)
Language model gives scores P(e) for each sentence without f
▶ P(e = he is big)
▶ P(e = is big he)
▶ P(e = this is a purple dog)
2014-05-08 Koichi Akabe (NAIST MT) 3 / 20

8. Language model (LM)
Language model gives scores P(e) for each sentence without f
▶ P(e = he is big)
▶ P(e = is big he)
▶ P(e = this is a purple dog)
Using this, we can compare sentences!
P(e = e1) > P(e = e3) > P(e = e2) ?
MT uses LM to increase translation accuracy
We call P(e) “language model probability”
2014-05-08 Koichi Akabe (NAIST MT) 3 / 20

9. How to calculate P(e)?
We want to calculate probability of a sentence:
P(e = he is big)
2014-05-08 Koichi Akabe (NAIST MT) 4 / 20

10. How to calculate P(e)?
We want to calculate probability of a sentence:
P(e = he is big)
Direct method: count frequency of sentences in the training data
PML(e) =
ctrain(e)
∑
e′ ctrain(e′)
2014-05-08 Koichi Akabe (NAIST MT) 4 / 20

11. How to calculate P(e)?
We want to calculate probability of a sentence:
P(e = he is big)
Direct method: count frequency of sentences in the training data
PML(e) =
ctrain(e)
∑
e′ ctrain(e′)
Almost possible sentences are not contained in the training data
(−→ PML(e) = 0 for almost sentences)
2014-05-08 Koichi Akabe (NAIST MT) 4 / 20

12. How to calculate P(e)?
We want to calculate probability of a sentence:
P(e = he is big)
Direct method: count frequency of sentences in the training data
PML(e) =
ctrain(e)
∑
e′ ctrain(e′)
Almost possible sentences are not contained in the training data
(−→ PML(e) = 0 for almost sentences)
Focus words to solve this problem
2014-05-08 Koichi Akabe (NAIST MT) 4 / 20

13. Rewrite P using words
P(e = he is big)
2014-05-08 Koichi Akabe (NAIST MT) 5 / 20

14. Rewrite P using words
P(e = he is big)
First, we split variable e into words and text length I
P(I = 3, e1 = he, e2 = is, e3 = big)
2014-05-08 Koichi Akabe (NAIST MT) 5 / 20

15. Rewrite P using words
P(e = he is big)
First, we split variable e into words and text length I
P(I = 3, e1 = he, e2 = is, e3 = big)
To use uniform variable type, we replace I to eI = ⟨/s⟩
P(e1 = he, e2 = is, e3 = big, e4 = ⟨/s⟩)
2014-05-08 Koichi Akabe (NAIST MT) 5 / 20

16. Rewrite P using words
P(e = he is big)
First, we split variable e into words and text length I
P(I = 3, e1 = he, e2 = is, e3 = big)
To use uniform variable type, we replace I to eI = ⟨/s⟩
P(e1 = he, e2 = is, e3 = big, e4 = ⟨/s⟩)
We also add a prefix symbol for contexts (described later)
P(e0 = ⟨s⟩, e1 = he, e2 = is, e3 = big, e4 = ⟨/s⟩)
2014-05-08 Koichi Akabe (NAIST MT) 5 / 20

17. Rewrite P using conditional probability P(word|context)
2014-05-08 Koichi Akabe (NAIST MT) 6 / 20

18. Rewrite P using conditional probability P(word|context)
Chain rule: P(e0 = ⟨s⟩, e1 = he, e2 = is, e3 = big, e4 = ⟨/s⟩)
= P(e4 = ⟨/s⟩|e0 = ⟨s⟩, e1 = he, e2 = is, e3 = big)
×P(e3 = big|e0 = ⟨s⟩, e1 = he, e2 = is)
×P(e2 = is|e0 = ⟨s⟩, e1 = he)
×P(e1 = he|e0 = ⟨s⟩)×P(e0 = ⟨s⟩)
2014-05-08 Koichi Akabe (NAIST MT) 6 / 20

19. Rewrite P using conditional probability P(word|context)
Chain rule: P(e0 = ⟨s⟩, e1 = he, e2 = is, e3 = big, e4 = ⟨/s⟩)
= P(e4 = ⟨/s⟩|e0 = ⟨s⟩, e1 = he, e2 = is, e3 = big)
×P(e3 = big|e0 = ⟨s⟩, e1 = he, e2 = is)
×P(e2 = is|e0 = ⟨s⟩, e1 = he)
×P(e1 = he|e0 = ⟨s⟩)×P(e0 = ⟨s⟩)
Generalize:
P(eI
1) =
I+1∏
i=1
PML(ei|ei−1
0 ) =
I+1∏
i=1
ctrain(ei
0)
ctrain(ei−1
0 )
ej
i = ei ei+1 · · · ej is a part of word sequence e0 e1 · · · eI+1
2014-05-08 Koichi Akabe (NAIST MT) 6 / 20

20. Rewrite P using conditional probability P(word|context)
Chain rule: P(e0 = ⟨s⟩, e1 = he, e2 = is, e3 = big, e4 = ⟨/s⟩)
= P(e4 = ⟨/s⟩|e0 = ⟨s⟩, e1 = he, e2 = is, e3 = big)
×P(e3 = big|e0 = ⟨s⟩, e1 = he, e2 = is)
×P(e2 = is|e0 = ⟨s⟩, e1 = he)
×P(e1 = he|e0 = ⟨s⟩)×P(e0 = ⟨s⟩)
Generalize:
P(eI
1) =
I+1∏
i=1
PML(ei|ei−1
0 ) =
I+1∏
i=1
ctrain(ei
0)
ctrain(ei−1
0 )
ej
i = ei ei+1 · · · ej is a part of word sequence e0 e1 · · · eI+1
However, ctrain(ei
0) becomes 0 for large i
2014-05-08 Koichi Akabe (NAIST MT) 6 / 20

21. n-gram language model
So, we do not use long word sequences!
2014-05-08 Koichi Akabe (NAIST MT) 7 / 20

22. n-gram language model
So, we do not use long word sequences!
n-gram model uses only n − 1 words as contexts:
P(eI
1) ≈
I+1∏
i=1
PML(ei|ei−1
i−n+1) =
I+1∏
i=1
ctrain(ei
i−n+1)
ctrain(ei−1
i−n+1)
n-gram model eases 0 probability problem
2014-05-08 Koichi Akabe (NAIST MT) 7 / 20

23. Example of strict / 2-gram probabilities
Strict probability:
P(e = he is big) = PML(⟨/s⟩|⟨s⟩ he is big)
×PML(big|⟨s⟩ he is)
×PML(is|⟨s⟩ he)
×PML(he|⟨s⟩)
2-gram probability:
P(e = he is big) ≈ PML(⟨/s⟩|big)
×PML(big|is)
×PML(is|he)
×PML(he|⟨s⟩)
2014-05-08 Koichi Akabe (NAIST MT) 8 / 20

24. Smoothing
Smoothing makes a robust LM for unknown linguistic phenomena
2014-05-08 Koichi Akabe (NAIST MT) 9 / 20

25. Smoothing
Smoothing makes a robust LM for unknown linguistic phenomena
Basically, we calculate n-gram LM probability with (n − 1)-gram or
shorter contexts
2014-05-08 Koichi Akabe (NAIST MT) 9 / 20

26. Linear interpolation
Interpolate probability with shorter n-grams:
P(ei|ei−1
i−n+1) = (1 − α)PML(ei|ei−1
i−n+1) + αP(ei|ei−1
i−n+2)
For large α: ■■■■
■■■
For small α: ■■■■
■■■
2014-05-08 Koichi Akabe (NAIST MT) 10 / 20

27. Linear interpolation
Interpolate probability with shorter n-grams:
P(ei|ei−1
i−n+1) = (1 − α)PML(ei|ei−1
i−n+1) + αP(ei|ei−1
i−n+2)
For large α: ■■■■
■■■
For small α: ■■■■
■■■
Give constant probability for unknown words:
P(ei) = (1 − α)PML(ei) + α
1
|V|
where |V| is the vocabulary size
2014-05-08 Koichi Akabe (NAIST MT) 10 / 20

28. Linear interpolation
Interpolate probability with shorter n-grams:
P(ei|ei−1
i−n+1) = (1 − α)PML(ei|ei−1
i−n+1) + αP(ei|ei−1
i−n+2)
For large α: ■■■■
■■■
For small α: ■■■■
■■■
Give constant probability for unknown words:
P(ei) = (1 − α)PML(ei) + α
1
|V|
where |V| is the vocabulary size
How to choose α?
2014-05-08 Koichi Akabe (NAIST MT) 10 / 20

29. Idea of Witten-Bell method
Table Comparison of two n-gram contexts
president was president ronald
elected 5 reagan 38
the 3 caza 1
in 3 venetiaan 1
first 3
· · ·
52 unique words, 110 times 3 unique words, 40 times
2014-05-08 Koichi Akabe (NAIST MT) 11 / 20

30. Idea of Witten-Bell method
Table Comparison of two n-gram contexts
president was president ronald
elected 5 reagan 38
the 3 caza 1
in 3 venetiaan 1
first 3
· · ·
52 unique words, 110 times 3 unique words, 40 times
▶ “president was” may follow unknown words
−→ We cannot trust P(·|president was)
▶ “president ronald” frequently follow “reagan”
−→ We can trust P(·|president ronald)
2014-05-08 Koichi Akabe (NAIST MT) 11 / 20

31. Witten-Bell method
α depends on each reliability of n-gram:
αei−1
i−n+1
=
u(ei−1
i−n+1, ·)
u(ei−1
i−n+1, ·) + c(ei−1
i−n+1)
P(ei|ei−1
i−n+1) = (1 − αei−1
i−n+1
)PML(ei|ei−1
i−n+1) + αei−1
i−n+1
P(ei|ei−1
i−n+2)
2014-05-08 Koichi Akabe (NAIST MT) 12 / 20

32. Witten-Bell method
α depends on each reliability of n-gram:
αei−1
i−n+1
=
u(ei−1
i−n+1, ·)
u(ei−1
i−n+1, ·) + c(ei−1
i−n+1)
P(ei|ei−1
i−n+1) = (1 − αei−1
i−n+1
)PML(ei|ei−1
i−n+1) + αei−1
i−n+1
P(ei|ei−1
i−n+2)
e.g.
αpresident was =
u(president was, ·)
u(president was, ·) + c(president was)
=
52
52 + 110
αpresident was = 0.32 −→ do not trust (see shorter contexts)
αpresident ronald = 0.07 −→ trust
2014-05-08 Koichi Akabe (NAIST MT) 12 / 20

33. Absolute discounting method
Discount constant parameter d from frequencies of n-grams
Pd(ei|ei−1
i−n+1) =
max(ctrain(ei
i−n+1)−d, 0)
ctrain(ei−1
i−n+1)
Ignore rare n-grams
0 1 2 3 4 5 6 7 8 9 10
0.0
0.5
1.0
d = 0.5
d = 0.1
d = 1.0
d = 2.0
Ratiotonormalcounting
2014-05-08 Koichi Akabe (NAIST MT) 13 / 20

34. Absolute discounting method
Give discounted quantity for shorter n-grams
αei−1
i−n+1
= 1 −
∑
ei
Pd(ei|ei−1
i−n+1)
P(ei|ei−1
i−n+1) = Pd(ei|ei−1
i−n+1) + αei−1
i−n+1
P(ei|ei−1
i−n+2)
2014-05-08 Koichi Akabe (NAIST MT) 14 / 20

35. Absolute discounting method
Give discounted quantity for shorter n-grams
αei−1
i−n+1
= 1 −
∑
ei
Pd(ei|ei−1
i−n+1)
P(ei|ei−1
i−n+1) = Pd(ei|ei−1
i−n+1) + αei−1
i−n+1
P(ei|ei−1
i−n+2)
e.g. α := 0.5 (normally decided to maximize likelihood of dev set)
Pd(reagan|president ronald) =
38 − 0.5
40
= 0.9375
Pd(caza|president ronald) =
1 − 0.5
40
= 0.0125
Pd(venetiaan|president ronald) =
1 − 0.5
40
= 0.0125
αpresident ronald = 1 −
∑
ei
Pd(ei|president ronald) = 0.0375
2014-05-08 Koichi Akabe (NAIST MT) 14 / 20

36. Kneser-Ney method
Idea
“ronald reagan” or “president reagan” is frequently contained in
corpora, and normal smoothing methods give large probability for
“ronald” and “reagan”. However, “reagan” is not used in other
contexts.
2014-05-08 Koichi Akabe (NAIST MT) 15 / 20

37. Kneser-Ney method
Idea
“ronald reagan” or “president reagan” is frequently contained in
corpora, and normal smoothing methods give large probability for
“ronald” and “reagan”. However, “reagan” is not used in other
contexts.
Kneser and Ney used the unique counter u in absolute discounting
Pkn(ei|ei−1
i−n+1) =
max(u(·, ei
i−n+1) − d, 0)
u(·, ei−1
i−n+1, ·)
αei−1
i−n+1
= 1 −
∑
ei
Pkn(ei|ei−1
i−n+1)
2014-05-08 Koichi Akabe (NAIST MT) 15 / 20

38. Kneser-Ney method (example)
u(·, reagan) = 2 u(·, ronald reagan) = 10
u(·, ronald smith) = 1 u(·, ronald, ·) = 11
u(·, ·) = 2000 d = 0.5
Pkn(reagan|ronald) =
max(u(·, ronald reagan) − d, 0)
u(·, ronald, ·)
= 0.864
Pkn(smith|ronald) =
max(u(·, ronald smith) − d, 0)
u(·, ronald, ·)
= 0.045
Pkn(reagan) =
max(u(·, reagan) − d, 0)
u(·, ·)
= 0.00075
αronald = 1 −
∑
ei
Pkn(ei|ronald) = 0.091
2014-05-08 Koichi Akabe (NAIST MT) 16 / 20

39. Other methods
Additive smoothing
Pd(ei|ei−1
i−n+1) =
ctrain(ei
i−n+1)+δ
ctrain(ei−1
i−n+1)+δ|W|
where |W| is a number of words (for normalization)
2014-05-08 Koichi Akabe (NAIST MT) 17 / 20

40. Other methods
Good-Turing (“Good” is a scientist)
Turing estimator uses revised values as a number of words:
r∗
= (r + 1)
Nr+1
Nr
where Nr is a number of words occurring r times
If Nr = 0, r∗ becomes indeterminate form
Good-Turing estimator uses linear regression with Zipf’s law to
solve this problem
Zri :=
2Nri
ri+1 − ri−1
where ri is ith non-zero number (r1 < r2 < r3 < · · · )
2014-05-08 Koichi Akabe (NAIST MT) 18 / 20

41. Other methods
Good-Turing (cont’d)
Estimate a and b:
log Zri ∼ a + b log ri
r∗
= (r + 1)
Zr+1
Zr
= r
(
1 +
1
r
)b+1
2014-05-08 Koichi Akabe (NAIST MT) 19 / 20

42. Other methods
Back-off
Use shorter n-grams only when longer n-gram is not contained
Absolute discounting with back-off:
P(ei|ei−1
i−n+1) =
{
Pd(ei|ei−1
i−n+1) (c(ei−1
i−n+1) > 0)
βei−1
i−n+1
P(ei|ei−1
i−n+2) (otherwise)
2014-05-08 Koichi Akabe (NAIST MT) 20 / 20