The document describes a new approach for using deep neural networks to learn features for statistical machine translation. Specifically, it uses deep autoencoders to extract features from input data in an unsupervised manner, rather than manually engineering features. The approach feeds 16 input features into a deep belief network made of restricted Boltzmann machines. This network is then unrolled to form a deep autoencoder, which is fine-tuned using backpropagation. Stacking multiple trained autoencoders results in statistically significant improvements over baseline features on Chinese-English translation tasks.

1. Learning New Semi-Supervised Deep Auto-encoder
Features for Statistical Machine Translation
by Shixiang Lu, Zhenbiao Chen, Bo Xu
Presented By V B Wickramasinghe (148245F)

2. Overview
● Introduction
● Input features for DNN feature learning
● Semi-supervised deep auto-encoder
features learning for SMT
● Experiments and Results
● Conclusion

3. Introduction
● Paper describes a novel approach to statistical machine
translation(SMT).
● Uses two deep neural network architectures specifically,
○ Deep belief networks(DBN)
○ Deep auto encoders(DAE)
● The goal is to extract useful features of languages
automatically using DAEs instead of doing it manually.
● Achieves statistically significant improvements over
unsupervised DBN and baseline features.

4. Input features for DNN feature learning
● Uses a phrase-based translation model.
● Four phrase features are used as the baseline. With f as source and e as
target,
Other features,
● Bidirectional phrase pair similarity.
● Bidirectional Phrase generative probability.

5. Input features for DNN feature learning
● Phrase frequency.
● Phrase length.
In total there 16 input features which are represented by 16 input nodes in the
DAE.

6. Semi-supervised deep auto-encoder
features learning for SMT
● The introduced set of features(X) is then fed to a set of
RBMs.
● Combined together these form a DBN.
● These RBMs are layerwise pretrained to learn deep higher
order correlations between the input features.
● Then unrolling each performed on this DBN to form a DAE.
● Which is then finetuned using back propagation.
● Final step is to stack a number of these trained DAEs to
form a 16-32-32-32-16-16-8 architecture after tuning.

7. Semi-supervised deep auto-encoder
features learning for SMT

8. Experiments & Results
● Experimental Setup
IWSLT. The bilingual corpus is the Chinese English part of Basic Traveling
Expression corpus (BTEC) and China-Japan-Korea (CJK) corpus (0.38M
sentence pairs with 3.5/3.8M Chinese/English words).
NIST. The bilingual corpus is LDC4 (3.4M sentence pairs with 64/70M
Chinese/English words). The LM corpus is the English side of the parallel data
as well as the English Gigaword corpus (LDC2007T07) (11.3M sentences).

9. Experiments & Results

10. Thank you