Presentation

Translation Memory Retrieval Methods
[Bloodgood and Strauss, 2014] in Proc of 14th EACL
Koichi Akabe and Philip Arthur
NAIST MT Study
2014-07-03
2014-07-03 Koichi Akabe and Philip Arthur (MT Study) 1 / 27

Introduction

Translation Memory (TM)

▶ Most widely used computer-assisted translation (CAT) tool
▶ Suggest translations using other translations

En The dog opened the door.
Ja 犬がドアを開けた。
En I saw a girl with a telescope.
Ja 僕は望遠鏡で少女を見た。
En John opened the door.
Ja

Ja
1. Find the nearest source sentence

Ja 犬がドアを開けた。 (fuzzy)
2. Suggest a translation

Ja ジョンがドアを開けた。
2. Suggest a translation
3. Post-editing

How to ﬁnd the nearest source sentence?
TM ﬁnds the nearest source sentence using similarity metrics

▶ Edit distance (Leven-shtein distance)
−→ Widely used metric

▶ MT evaluation metrics [Simard and Fujita, 2012]
−→ WER, BLEU, NIST, VMeteor, Meteor as TM metrics

▶ MT evaluation metrics [Simard and Fujita, 2012]
−→ WER, BLEU, NIST, VMeteor, Meteor as TM metrics
▶ This paper

Threshold of helpfulness
Matching algorithm always returns the nearest sentence
However, low score suggestions should not be shown

TM softwares set the threshold at 70% in practice

TM softwares set the threshold at 70% in practice −→ Why?

Translation Memory Similarity Metrics

Deﬁnitions
TM Similarity Metrics compare M and C.
M: workload sentence
C: source language side of a candidate pre-existing translation

Deﬁnitions
TM Similarity Metrics compare M and C.
M: workload sentence
C: source language side of a candidate pre-existing translation
En The dog opened the door .
En I saw a girl with a telescope .
En John opened the door .
Ja 犬がドアを開けた。 (fuzzy)
M =John opened the door .
C1 =The dog opened the door .
C2 =I saw a girl with a telescope .
...

Translation Memory Similarity Metrics
Compare the following metrics:
▶ Percent Match
▶ Weighted Percent Match
▶ Edit Distance
▶ N-gram Precision
▶ Weighted N-gram Precision
▶ Modiﬁed Weighted N-gram Precision

Percent Match (PM)

Percent Match (PM)
The simplest metric
PM(M, C) =
|Munigrams ∩ Cunigrams|
|Munigrams|

Percent Match (PM)
The simplest metric
PM(M, C) =
|Munigrams|
e.g.
C =The dog opened the door .

Percent Match (PM)
The simplest metric
PM(M, C) =
|Munigrams|
e.g.
PM(M, C) =
4
5
= 0.80

Weighted Percent Match (WPM)

We want to know translation of rare words

We want to know translation of rare words
PM with IDF weighting
WPM(M, C) =
∑
u∈{Munigrams∩Cunigrams}
idf(u, D)
∑
u∈Munigrams
idf(u, D)
where D is a set of all source sentences in the parallel corpus

Problem of PM and WPM
PM and WPM only consider coverage of words

−→ They cannnot see any context

−→ They cannnot see any context
We show methods that consider contexts in next slides

Edit Distance (ED)

Edit Distance (ED)
Widely used metric
ED = max
(
1 −
edit-dist(M, C)
|Munigrams|
, 0
)
where edit-dist(M, C) is the number of word insertions, deletions,
and substitutions required to transform M into C

Edit Distance (ED)
Widely used metric
ED = max
(
1 −
edit-dist(M, C)
|Munigrams|
, 0
)
e.g.

Edit Distance (ED)
Widely used metric
ED = max
(
1 −
edit-dist(M, C)
|Munigrams|
, 0
)
e.g.
substitution: 1

Edit Distance (ED)
Widely used metric
ED = max
(
1 −
edit-dist(M, C)
|Munigrams|
, 0
)
e.g.
substitution: 1
insertion: 1

Edit Distance (ED)
Widely used metric
ED = max
(
1 −
edit-dist(M, C)
|Munigrams|
, 0
)
e.g.
substitution: 1
insertion: 1
ED(M, C) = 1 −
2
5
= 0.60

N-gram Precision (NGP)

Mean of N-gram precision (like the BLEU metric)

However, BLEU → 0 when the precision of longer N-grams is 0

However, BLEU → 0 when the precision of longer N-grams is 0
This work uses arithmetic mean instead of geometric mean
NGP =
1
N
N∑
n=1
pn
pn =
|Mn-grams ∩ Cn-grams|
Z ∗ |Mn-grams| + (1 − Z) ∗ |Cn-grams|
where Z is a parameter to control normalization,
and N is the maximum length of N-gram
N = 4 and Z = 0.75 in main experiments (discuss later)

Weighted N-gram Precision (WNGP)

Weighted N-gram Precision (WNGP)
NGP with IDF weighting
WNGP =
N∑
n=1
1
N
wpn
wpn =
∑
i∈{Mn-grams∩Cn-grams}
w(i)
Z ∗


∑
i∈Mn-grams
w(i)

 + (1 − Z) ∗


∑
i∈Cn-grams
w(i)


w(i) =
∑
1-gram∈i
idf(1-gram, D)

Modiﬁed Weighted N-gram Precision (MWNGP)

Modiﬁed Weighted N-gram Precision (MWNGP)
Shorter N-grams may help translators more than longer N-grams
WNGP =
N∑
n=1
1
N
wpn
MWNGP =
2N
2N − 1
N∑
n=1
1
2n
wpn

Experiment

Experiment
Two diﬀerent technicals domains with Two diﬀerent language pairs
(Fr-En, Zn-En).

Experiment
(Fr-En, Zn-En).
▶ Zn-En: OpenOﬃce3
▶ Fr-En: EMEA

Experiment
(Fr-En, Zn-En).
▶ Fr-En: EMEA
Preprocessing is performed on both source sides to produce valid
segment.

Experiment
(Fr-En, Zn-En).
▶ Fr-En: EMEA
Preprocessing is performed on both source sides to produce valid
segment.
Some sentences are randomly sampled from corpus as M and C.
▶ Zn-En: 400 M and 10.000 C.
▶ Fr-En: 300 M and 10.000 C.

Evaluation

Evaluation
Evaluation is performed with Human Evaluation using Amazon
Mechanical Turk.

Evaluation
Mechanical Turk.
The Score is ranging from 1 to 5 (Not Helpful until Extremely
Helpful).

Evaluation
Mechanical Turk.
Helpful).
Each segment M is rated by 5 Turkers and we keep track which
metric performs best (ties is allowed).

Evaluation
Mechanical Turk.
Helpful).
Each segment M is rated by 5 Turkers and we keep track which
metric performs best (ties is allowed).
The scores of each M are averaged as Mean Opinion Score
(MOS).

Result and Analysis

Result: Which metric performs best?

Table OO3 Zn-En
Metric Found Best Total C
PM 178 400
WPM 200 400
ED 193 400
NGP 251 400
WNGP 271 400
MWNGP 282 400
Table EMEA Fr-En
PM 166 300
WPM 184 300
ED 148 300
NGP 188 300
WNGP 198 300
MWNGP 201 300

Table OO3 Zn-En
PM 178 400
WPM 200 400
ED 193 400
NGP 251 400
WNGP 271 400
MWNGP 282 400
Table EMEA Fr-En
PM 166 300
WPM 184 300
ED 148 300
NGP 188 300
WNGP 198 300
MWNGP 201 300
Modiﬁed Weighted N-Gram Precision (MWNGP) achieved the
best result compared to any other metrics.

Table OO3 Zn-En
PM 178 400
WPM 200 400
ED 193 400
NGP 251 400
WNGP 271 400
MWNGP 282 400
Table EMEA Fr-En
PM 166 300
WPM 184 300
ED 148 300
NGP 188 300
WNGP 198 300
MWNGP 201 300
Modified Weighted N-Gram Precision (MWNGP) achieved the
best result compared to any other metrics.
There are slight different between WNGP and Modified-WNGP.

Scatterplot: OO3 Percent Match
1.0 1.5 2.0 2.5 3.0 3.5 4.0 4.5 5.0
MOS
0.0
0.2
0.4
0.6
0.8
1.0
MetricValue

Scatterplot: OO3 Edit Distance
1.0 1.5 2.0 2.5 3.0 3.5 4.0 4.5 5.0
MOS
0.0
0.2
0.4
0.6
0.8
1.0
MetricValue

Scatterplot: OO3 Modiﬁed N-Gram Precision
1.0 1.5 2.0 2.5 3.0 3.5 4.0 4.5 5.0
MOS
0.0
0.2
0.4
0.6
0.8
1.0
MetricValue

The eﬀect of Z: Adjusting for length preferences

Many of the metrics are using Z as parameters.

Z parameter can be used to control for length preferences.
Table EMEA Fr-En
Z Value Avg Length
0.00 9.9298
0.25 13.204
0.50 16.0134
0.75 19.6355
1.00 27.8829
Table OO3 Zn-En
Z Value Avg Length
0.00 7.2475
0.25 9.5600
0.50 11.1250
0.75 14.1825
1.00 25.0875

Table EMEA Fr-En
Z Value Avg Length
0.00 9.9298
0.25 13.204
0.50 16.0134
0.75 19.6355
1.00 27.8829
Table OO3 Zn-En
Z Value Avg Length
0.00 7.2475
0.25 9.5600
0.50 11.1250
0.75 14.1825
1.00 25.0875
Smaller Z prefered shorter match that are more precise and
increased precision.

Table EMEA Fr-En
Z Value Avg Length
0.00 9.9298
0.25 13.204
0.50 16.0134
0.75 19.6355
1.00 27.8829
Table OO3 Zn-En
Z Value Avg Length
0.00 7.2475
0.25 9.5600
0.50 11.1250
0.75 14.1825
1.00 25.0875
Smaller Z prefered shorter match that are more precise and
increased precision.
Larger Z prefers longer match that contains many correct
translations and increased recall.

Conclusion

Conclusion
▶ This paper compares TM similarity metrics.

Conclusion
▶ The best method is Modiﬁed Weighted N-Gram Precision.

Conclusion
▶ All the discussed metrics only consider source sides in the
calculation.

Conclusion
▶ All the discussed metrics only consider source sides in the
calculation.
▶ Z parameter is used to adjust the length preferences of the
retrieved TM.

Thank you for your attention!

Presentation

Recommended

Recommended

More Related Content

Recently uploaded

Recently uploaded (20)

Featured

Featured (20)

Presentation