This document summarizes the key aspects of a neural machine translation model using sequence-to-sequence with attention. It describes the encoder-decoder architecture using stacked LSTMs, dot product attention, and how the model is trained on WMT'14 English-German data and evaluated on Newstest2013 data. It also provides information on the tokenizer, dataset creation, model configuration, and training utilities.

1. [IMPL] Neural Machine Translation
장재호
19. 09. 06 (금)

2. 목차
• Sequence to sequence with attention
• Data
• Tokenizer
• NMTDataset
• Model
• EncLayer
• DecCell
• DecLayer
• Utils

3. Sequence to sequence with
attention
참고자료
“Effective Approaches to Attention-based Neural Machine Translation”. Luong et al. 2015.
IBM/pytorch-seq2seq: https://github.com/IBM/pytorch-seq2seq

4. Sequence to sequence with attention
• Model: Sequence to sequence with attention (Luong et al.
2015)
• Tokenizer (SentencePiece):
• BPE (Byte Pair Encoding)
• WPM (WordPieceModel)
• Attention
• Dot Product Attention
• Multiplicative Attention
• Additive Attention

5. Sequence to sequence with attention
• Encoder-Decoder
• Encoder: Stacked Bidirectional
LSTM
• Decoder: Stacked LSTM

6. Sequence to sequence with attention
• Attention
• Dot Product Attention
• Multiplicative Attention
• Additive Attention

7. Data
Train: WMT’14 English-German Data
Test: Newstest2013 English-German Data
데이터 출처: https://nlp.stanford.edu/projects/nmt/
SentencePiece: https://github.com/google/sentencepiece

8. Data
• SentencePiece
• Subword Segmentation Library
• Pytorch 데이터 유틸리티
• torch.utils.data.Dataset
• Abstract class
• Iterable object
• __len__()
• __getitem__()
• torch.utils.data.Dataloader
• Generator object
• Collate (callback function)

9. Data
• https://github.com/om00839/Seq2Seq/blob/master/Script%20
-%20data.ipynb

10. Model
참고자료
“Effective Approaches to Attention-based Neural Machine Translation”. Luong et al. 2015.
IBM/pytorch-seq2seq: https://github.com/IBM/pytorch-seq2seq

11. Model
• Layers
• EncLayer
• DecCell
• DecLayer
• Attention
• BaseAttention
• DotProdAttention
• MulAttention
• AddAttention
• Model
• Seq2SeqWithAttn

12. Model
• EncLayer
𝒉
LSTM Layer
Embedding Layer
𝑤#
(%)
𝑤'
(%)
𝑤()
(%)
…
ℎ# ℎ' ℎ()…
Source Input
OutputTarget

13. Model
• DecCell
DecCell
Embedding
LSTM
Cell𝑤+
(+) Linear
Classifer 𝑙𝑜𝑔𝑖𝑡+Attention
𝑠+2#,
𝑐+2#
𝑠+, 𝑐+
𝒉
Source Input
OutputTarget

14. Model
• DecLayer
• train
𝐷𝑒𝑐
𝐶𝑒𝑙𝑙#
𝑤#
(+)
bos
𝑤'
(+) 𝑤(82#
(+)
…
𝑙𝑜𝑔𝑖𝑡# …𝑙𝑜𝑔𝑖𝑡'
𝑙𝑜𝑔𝑖𝑡(8
𝐷𝑒𝑐
𝐶𝑒𝑙𝑙'
𝐷𝑒𝑐
𝐶𝑒𝑙𝑙(8
𝑤'
(+)
𝑤9
(+) 𝑤(8
(+)
eos
…
Cross Entropy
…
Source Input
OutputTarget

15. Model
• DecLayer
• infer
𝐷𝑒𝑐
𝐶𝑒𝑙𝑙#
𝑤#
(+)
bos
:𝑤'
(+) :𝑤(82#
(+)
…
𝑙𝑜𝑔𝑖𝑡# …𝑙𝑜𝑔𝑖𝑡'
𝑙𝑜𝑔𝑖𝑡(8
𝐷𝑒𝑐
𝐶𝑒𝑙𝑙'
𝐷𝑒𝑐
𝐶𝑒𝑙𝑙(8
:𝑤'
(+)
:𝑤9
(+) :𝑤(8
(+)
eos
…
ℎ()
,
𝑐()
…
Source Input
OutputTarget

16. Model
• Configuration
• src_vocab_size=32000
• tar_vocab_size=32000
• attention=AddAttention
• embedding_dim=256
• hidden_dim=256
• enc_hidden_dim = 256
• dec_hidden_dim = 256*2 (enc_hidden_dim * n_directions)
• n_layers=2,
• bidirectional=True

17. Model
• https://github.com/om00839/Seq2Seq/blob/master/Script%20
-%20model.ipynb

18. Utils
참고자료
IBM/pytorch-seq2seq: https://github.com/IBM/pytorch-seq2seq

19. Utils
• Train & evaluate
• train_batches()
• evaluate_batches()
• generate_sentence()
• EarlyStopping
• Metrics (pytorch-nlp)
• bleu_score
• Accuracy

20. Utils
• https://github.com/om00839/Seq2Seq/blob/master/Script%20
-%20utils.ipynb

21. 감사합니다