The document proposes a novel linear associative unit (LAU) for neural machine translation. LAU reduces the gradient path inside recurrent units to address issues with optimizing deep neural networks. Experiments on Chinese-English, English-German, and English-French translation tasks show LAU outperforms GRU baselines, and increasing model depth is more effective than width. Analysis indicates LAU enables information transfer better than GRU, especially for longer sentences.

1. Deep Neural Machine
Translation
with Linear Associative Unit
Mingxuan Wang, Zhengdong Lu, Jie Zhou, Qun Liu
(ACL 2017)
首都大 B4 勝又智

2. Abstract
● NMT systems with deep architecture RNNs often suffer from severe gradient
diffusion.
○ due to the non-linear recurrent activations, which often make the optimization much more
difficult.
● we propose novel linear associative units (LAU).
○ reduce the gradient path inside the recurrent units
● experiment
○ NIST task: Chinese-English
○ WMT14: English-German
English-French
● analysis
○ LAU vs. GRU
○ Depth vs. Width
○ about Length 2

3. background: gate structure
● LSTM, GRU: capture long-term dependencies
● Residual Network (He et al., 2015)
● highway Network (Srivastava et al., 2015)
● Fast-Forward Network (Zhou et al., 2016) (F-F connections)
3
highway Network (H, T: non-linear function)

4. background: Gated Recurrent Unit
taking a linear sum between the existing state and the newly computed state
z_t: update gate
r_t: reset gate
4

5. Model: LAU (Linear Asocciative Unit)
LAU extends GRU by having an additional linear transformation of the input.
f_t and r_t express how much of the non-linear abstraction are preduced
by the input x_t and previous hidden state h_t.
g_t decides how much of the linear transformation of the input is carried
to the hidden state.
GRU
5

6. What is good using LAU?
LAU offers a direct way for input x_t to go to latter hidden state layers.
This mechanism is very useful for translation where the input should sometimes
be directly carried to the next stage of processing without any substantial
composition or nonlinear transformation.
ex. imagine we want to translate a rare entity name such as ‘Bahrain’ to Chinese.
→ LAU is able to retain the embedding of this word in its hidden state.
Otherwise, serious distortion occurs due to lack of training instances.
6

7. Model: encoder-decoder (DeepLAU)
● vertical stacking
○ only the output of the previous layer of RNN is
fed to the current layer as input.
● bidirectional encoder
○ φ is LAU.
○ the directions are marked by a direction term d.
d = -1 or +1
when d = -1, processing in forward diretion.
otherwise backward direction.
7

8. Model: Encoder side
● in order to learn more temporal dependencies, they choose unusual
bidirectional approach.
● encoding
○ an RNN layer processes the input sequence in forward direction.
○ the output of this layer is taken by an upper RNN layer as input, processed in reverse
direction.
○ Formally, following Equation (9), they set d = (-1)^ℓ
○ the final encoder consists of Lenc layers and produces the output
8

9. model: Attention side
α_t,j is caluclated by the first layer of decoder at step t - 1 (st-1),
the most-top layer of the encoder at step j (hj),
and context word yt-1.
σ(): tanh()
9

10. model: Decoder side
the decoder follows Equation (9) with fixed direction term d = -1.
At the first layer, they use the following input:
At inference stage, they only utilize the top-most hidden state s(Ldec)
to make the final predication with a softmax layer:
yt-1 is the target word embedding
10

11. Experiments: corpus
● NIST Chinese-English
○ training: LDC corpora 1.25M sents, 27.9M Chinese words and 34.5M English words
○ dev: NIST 2002 (MT02) dataset
○ test: NIST 2003 (MT03), NIST 2004 (MT04), NIST 2005 (MT05), NIST 2006 (MT06)
● WMT14 English-German
○ training: WMT14 training corpus 4.5M sents, 91M English words and 87M German words
○ dev: news-test 2012, 2013
○ test: news-test 2014
● WMT14 English-French
○ training: subset of WMT14 training corpus 12M sents, 304M English words and 348M French
words
○ dev: concatenation of news-test 2012 and news-test 2013
○ test: news-test 2014
11

12. Experiments: setup
● For all expariments
○ dimension: embedding, hidden states and ct are 512 size.
○ optimizer: Adadelta
○ batch size: 128
○ input length limit: 80 words
○ beam size: 10
○ dropout rate: 0.5
○ layer: both encoder and decoder have 4 layers
● settings of each experiment
○ in Chinese-English and English-French, use the soruce and target vocab frequent 30k
○ for English-German, use the source 120k and the target 80k in order of frequent
12

13. Result: Chinese-English, English-French
LAUs apply adaptive gate function conditioned on the input which it able to decide
how much linear information should be transferred to the next step.
13

14. Result: English-German
14

15. Analysis: LAU vs. GRU, Depth vs. Width
LAU vs. GRU
● row 3 to row 7
→ LAU bring imporovement.
● row 3,4 to row 7,8
→ GRU decrese BLEU,
but LAU bring improvement.
Depth vs. Width
● when increasing the model depth
but they failed to see further imporovements.
● hidden size (width)
row 2 to row 3
→ improvements is relative small
→ depth plays a more important role in incresing
the complexity of neural networks than width. in any analysis, use NIST Chinese-English task 15

16. Analysis: About Length
DeepLAU models yield higher BLEU score
than the DeepGRU model.
→ very deep RNN model is good at
modelling the nested latent structures
on relatively complicated sentences.
16

17. Conclusion
● propose a Linear Asociative Unit (LAU)
○ it makes a fusion of both linear and nonlinear transformation.
● LAU enable us to build a deep neural network for MT.
● My feeling
○ After all, is this model a non-recurrent deep or a recurrent deep?
maybe, both are likely to be good…
○ I was also interested in the weight of the model.
So, I wanted them to mention about it.
17

18. reference
● Srivastava et al. Training Very Deep Networks NIPS 2015
● Zhou et al. Deep Recurrent Models with Fast-Forward Connections for Neural
Machine Translation arXiv
● He et al. Deep residual learning for image recognition arXiv
● Wu et al. Google’s neural machine translation system: Bridging the gap
between human and machine translation arXiv
18