The document summarizes the DRAW paper which introduces an attention mechanism to a generative model called DRAW. It augments encoders and decoders with recurrent neural networks. The attention mechanism allows the model to focus on subsets of input data for reading and writing during generation. This allows DRAW to generate MNIST digits sequentially while focusing attention on different parts at each time step, producing higher quality images than without attention. It can also classify cluttered MNIST images by focusing attention on the digit.

1. DRAW
Deep Recurrent A-en.ve Writer
Karol Gregor, Ivo Danihelka, Alex Graves, Danilo Jimenez Rezende, Daan Wierstra
Google Deepmind – (ICML 2015)
Tran Quoc Hoan
Paper alert@2015-12-07

2. A-en.on Please!!!
(???)
Where to look?
Where to read?
Where to write?

3. Image genera.ng with a-en.on
Trained genera.ng sequen.al MNIST digit
without a)en,on
Trained genera.ng sequen.al MNIST digit
with a)en,on

4. Introduc.on
• Adds an a-en.on mechanism over the input
to define a sequen.al process
• Augments the encoder and decoder with
recurrent neural networks
• Inference and genera.on defined by a
sequen.al process, even for non-sequen,al
data
4/25

5. Varia.onal Autoencoder Recap
• Kingma and Welling, Auto-Encoding
Varia,onal Bayes, (ICLR) 2014
• Rezende, Mohamed and Wierstra, Stochas,c
back-propaga,on and varia,onal inference in
deep latent Gaussian models, (ICML) 2014
We’ll introduce VAE first!!!
5/25

6. VAE Approach
• Leverage neural networks to learn a latent variable
model
Image from: Ward, A. D., Hamarneh, G.: 3D Surface Parameteriza,on Using Manifold
Learning for Medial Shape Representa,on, Conference on Image Processing, Proc. of
SPIE Medical Imaging, 2007
Latent variable
6/25

7. The inference problem
• Where does z come from? -> The posterior p(z|x) is intractable
• VAE approach: find inference machine that learns to
approximate the posterior by op.mizing the
varia.onal lower bound
7/25
Encoder Decoder

8. Reparametriza.on trick
• Consider
• Parametrize z as where
• Parametrize x as where
8/25

9. Training with backpropaga.on
• Due to reparametriza.on trick, we can simultaneously
train both the genera.ve model and the inference
model by op.mizing the varia.onal bound using
gradient backpropaga.on
• Maximum
9/25

10. Building a simple VAE
10/25
From h-p://mul.threaded.s.tchfix.com/blog/
2015/09/17/deep-style/

11. Encoding
11/25

12. Encoding
12/25

13. Encoding
13/25

14. Varia.onal step
14/25

15. Varia.onal step
15/25

16. Decoding
16/25

17. Decoding
17/25

18. Update
18/25

19. Update
19/25

20. DRAW Model
Canvas at t
Could be a subset of x
(with a-en.on) 20/25

21. DRAW MNIST Genera.on
• Simplest instan.a.on of DRAW is without an a-en.on
mechanism
• En.re input is passed to the encoder at every .me-step
• Decoder writes to en.re canvas at every .me-step
No.ce how the network first
generates a very blurry
image that is subsequently
refined
21/25

22. DRAW A-en.on Mechanism
• DRAW uses a differen.able
a-en.on mechanism
• A-en.on uses recurrence (via
decoder) to select subsets of x
for reading and wri.ng
• A-en.on controls the
extracted patch
loca.on, scale and blur
22/25

23. DRAW A-en.on Mechanism
23/25

24. DRAW: Clu-ered MNIST Classifica.on
• Draw a-en.on being applied to
a classifica.on task: clu-ered
MNIST
• A-en.on learns to focus on the
digit in the scene
Time →
24/25

25. DRAW MNIST Genera.on with A-en.on
Figure 6. Generated MNIST images. All digits were generated by DRAW except those in the
rightmost column, which shows the training set images closest to those in the column
second to right. Note that the network was trained on the binary samples, while the
generated images are mean probabili.es. 25/25

26. DRAW MNIST Genera.on with A-en.on
h-ps://www.youtube.com/watch?
v=Zt-7MI9eKEo
26/25

27. DRAW
Deep Recurrent A-en.ve Writer
Karol Gregor, Ivo Danihelka, Alex Graves, Danilo Jimenez Rezende, Daan Wierstra
Google Deepmind – (ICML 2015)
Tran Quoc Hoan
Paper alert@2015-12-07
Many slides are borrowed from
h)p://videolectures.net/deeplearning2015_courville_autoencoder_extension/
h)p://mul,threaded.s,tchfix.com/blog/2015/09/17/deep-style/