The document proposes using pre-translation with phrase-based machine translation (PBMT) to handle rare words before final translation with neural machine translation (NMT). Two methods are suggested: a pipeline method and a mix method that concatenates the PBMT output and source sentence as input to NMT. Experiments on English-German translation show the mix method outperforms pure NMT and PBMT, gaining up to 1.8 BLEU points by combining the advantages of both approaches.

1. Paper Reading
Pre-Transla0on for
Neural Machine Transla0on
author:Jan Niehues, Eunah Cho, Thanh-
Le Ha, Alex Waibel
presenta0on Sekizawa Yuuki
17/06/21 1

2. Pre-Transla0on for
Neural Machine Transla0on
• NMT produces completely different meaning
• especially rare words occur
• SMT: simplifying input in preprocess has significant gain
• commonly pre-reordering approach
• In this work:
• pre-translate input with PBMT to target language
• NMT system generate final hypothesis use pre-translate sentence
• use “only PBMT output” or
“combina0on of the PBMT output and the source sentence”
• evaluate on En-Ge transla0on task
• outperform PBMT and NMT baseline up to 2 BLEU points
17/06/21 2

3. SMT vs NMT
• NMT outputs fluent and accurate transla0on
• create gramma0cally correct sentence
• able to model the morphologically agreement
• but, some0mes lose the original meaning of
the source words due to limi0ng vocabulary
• for training model efficiency
• SMT don’t need to limit vocabulary
• oXen translate words except very rarely in training
17/06/21 3

4. NMT transla0on example
• English : the goalie parried
• Gold : Torwart
• NMT : der GoY
• NMT(gloss) : the god
• NMT, BPE can’t transport the meaning
17/06/21 4
BPE split goalie
àgo / al / ie

5. propose
• goal : combine the advantages of SMT and NMT
• use pre-transla0on : straigh]orward way
• proposed method
1. translate the input using a PBMT system
• handle the rare words well
2. generate the final transla0on using an NMT system
• more fluent and gramma0cally correct transla0on
• this approach naturally introduces a necessity to
handle the poten0al errors by the PBMT systems
17/06/21 5

6. method 1 : pipeline
1. translate the input language f
to the target language eʹ using PBMT
2. train a neural monolingual transla0on system
• that translates from the output of the
PBMT system eʹ to a beYer target sentence e
17/06/21 6

7. method 2 : mix
1. ensure a non-overlapping vocabulary of f and eʹ by marking
each token in f by a character and eʹ by different ones
2. concatenate input sequences to the input e∗ (NMT system)
• NMT can learn to focus on source word fj and words eʹiʹ
when generat- ing a word eʹj
• this method prevent PBMT error (NMT can’t recover)
17/06/21 7

8. training NMT
• train on the output of the PBMT system
and the target language data
• have to translate whole parallel training data using PBMT
17/06/21 8

9. problem of propose method
• PBMT normally performs significantly beYer on the training
data than on unseen test data
• ability to use very long phrases
• NMT will underes0mate the number of improvements it has
to perform on test data
• harm the performance of our approach
u don’t use whole phrase tables when transla0ng training data
remove all phrase pairs that occur only once for the transla0on of corpus
*Following (Niehues and Waibel, 2013)
17/06/21 9
to limit
this effect

10. experiment setup
• PBMT system : two different systems
• baseline system using 3 language models
• a word-based, a bilingual (Niehues et al., 2011),
a cluster based language model 100 automa0cally generated clusters
• a system with advanced models
• pre-reodering (Herrmann et al., 2013) and lexicalized reordering
using a discrimina0ve word lexicon (Niehues and Waibel, 2013) and
a language model trained on the large monolingual data
• NMT system : using Nematus
• use BPE with 40K opera0ons to limit vocabulary size
• ensemble system : took the last four models
17/06/21 10
MKCLS (Och, 1999)

11. experiment setup
• data English to German news transla0on task of WMT
• training : WMT2016 (En-Ge) 3.7M sentences and
around 90M words
• op0mize : tst2014 using Minimum error rate training (SMT)
• test : tst2015, tst2016
• PBMT, NMT selngs are default configura0on
17/06/21 11

12. result in English à German
• pipeline is worse than pure NMT
• can’t combine the strength of both system in pipeline
• can’t recover SMT errors
17/06/21 12
propose
method

13. result in English à German
• pipeline is worse than pure NMT
• can’t combine the strength of both system in pipeline
• can’t recover SMT errors
• advanced : rewrite test data with advanced PBMT
• beYer than non-advanced but worse than pure NMT
17/06/21 13
propose
method
: PreMT

14. result in English à German
• Mix is beYer than pure NMT and PBMT
• recover some of the errors done by PBMT system
• prevent SMT errors by transla0ng source sentence directly
17/06/21 14
propose
method
: PreMT

15. Compare BLEU score (quality) by word frequency
17/06/21 15
For all systems, replace all target words by the UNK token if words occur less than N 0mes in the training data

16. transla0on example
English : Then with a shot which the goalie parried
with his knee in the 35th minute.
PBMT : Dann mit einem Schuss, die der Torwart pariert
mit seinem Knie in der 35. Minute.
NMT : Dann mit einem Schuss, den der Go0
mit seinem Knie in der 35. Minute.
PreMT : Dann mit einem Schuss, das der Torwart
mit seinem Knie in der 35. Minute pariert.
PreMT(gloss) : Then with a shoot, that the goali
with his knee in the 35th minute parried.
17/06/21 16
have to be
at the end of
sentence
not perform reordering
infrequent words miss
• go/al/ie ! “Go0”
• parried ! droped

17. Pre-Transla0on for
Neural Machine Transla0on
• supposed a technique to combine SMT and NMT
• SMT : pre-transla0on, NMT : final transla0on
• simple concatena0on of SMT output and source sentence
• it is input of NMT
• generate more fluent transla0on and translate rare words
• experiment result
• single pre-translated system could even outperform
the ensemble NMT system
• propose method ensemble system could outperform
the NMT system by up to 1.8 BLEU points
17/06/21 17