This document discusses attention mechanisms in deep learning models. It covers attention in sequence models like recurrent neural networks (RNNs) and neural machine translation. It also discusses attention in convolutional neural network (CNN) based models, including spatial transformer networks which allow spatial transformations of feature maps. The document notes that spatial transformer networks have achieved state-of-the-art results on image classification tasks and fine-grained visual recognition. It provides an overview of the localisation network, parameterised sampling grid, and differentiable image sampling components of spatial transformer networks.

1. Attention in Deep Models
(Sept 28, 2016)
YANG Jiancheng
Attention

2. Outline
• I. Attention in Sequence Models
• II. Attention in CN based Models
• III. Spatial Transformer Network

3. • I. Attention in Sequence Models
Normal RNN
RNN and the unfolding Bidirectional RNN

4. • I. Attention in Sequence Models
Seq2Seq and Machine Translation
Neural Machine TranslationSeq2seq
Sequence to Sequence Learning with Neural Networks

5. Neural Machine Translation by Jointly Learning to
Align and Translate
• I. Attention in Sequence Models
Attention Based RNN in Neural Machine Translation

6. • I. Attention in Sequence Models
Attention Based Article Understanding
Teaching Machines to Read and Comprehend

7. • II. Attention in CN based Models
Attention with Reinforcement Learning
Recurrent Models of Visual Attention

8. • II. Attention in CN based Models
Attention with differentiable module - STN
Spatial Transformer Networks

9. • III. Spatial Transformer Network
State-of-the-art performance (GTSRB dataset)
IDSIA 99.46% top-1 accuracy.
Moodstocks 99.61% top-1 accuracy with a much simpler pipeline.

10. • III. Spatial Transformer Network
State-of-the-art performance (SVHN & CUB-200-2011)

11. • III. Spatial Transformer Network
My trial (12 animals classification)
Single ResNet-50: ~85%
STN + ResNet-50: ~90%

12. • III. Spatial Transformer Network
Unreasonable effectiveness

13. • III. Spatial Transformer Network
Fine-grained Visual Recognition
CUB-200-2011
Bilinear CNN Models for Fine-grained Visual Recognition

14. • III. Spatial Transformer Network
Layer Structure
1. Localisation Network
2. Parameterised Sampling Grid
3. Differentiable Image Sampling

15. • III. Spatial Transformer Network
Parameterised Sampling Grid

16. Bibliography
• [1] Attention and Memory in Deep Learning and NLP
• [2] Sequence to Sequence Learning with Neural Networks
• [3] Neural Machine Translation by Jointly Learning to Align
and Translate
• [4] Recurrent Models of Visual Attention
• [5] Spatial Transformer Networks

17. Thanks for your attention!