ran

1. CS273B lecture 5: RNN and
autoencoder
James Zou
October 10, 2016

2. Recap

3. Recap: feedforward and convnets
Main take-aways:
• Composition. Units/layers of a NN are modular and can be
composed to form complex architecture.
• Weight-sharing. Enforcing that the weight be equal across a
set of units can dramatically decrease # of parameters.

4. What are limitations of convnets?

5. What are limitations of convnets?
• Fixed input length.
• Unclear how to adapt to time-series data.
• Convolution corresponds to strong prior—not appropriate
for many biological settings.
• Could require many labeled training examples (high sample
complexity).

6. What are limitations of convnets?
• Fixed input length.

7. What are limitations of convnets?
• Fixed input length.

8. Recurrent neural network
input
hidden units
output

9. Recurrent neural network
= ( · + )
= · +
input
hidden units
output

10. Recurrent neural network
input
hidden units
output

11. Recurrent neural network
+
+
+

12. Recurrent neural network
= ( · + · + )

13. Recurrent neural network
= · +

14. What does RNN remind you of?
+
+
+

15. Vanilla RNN: lacks long term memory
+
+
+

16. LSTM network

17. LSTM network
Hochreiter, Schmidhuber 1997

18. LSTM: inside the hood
Figure adapted from Olah blog.
memory

19. LSTM: inside the hood
= ( · [ , ] + )
= +
Figure adapted from Olah blog.

20. LSTM: inside the hood
= ( · [ , ] + )
= tanh( · [ , ] + )
= +
Figure adapted from Olah blog.

21. LSTM: inside the hood
= ( · [ , ] + )
= tanh( )
Figure adapted from Olah blog.
output

22. LSTM summary
• LSTM is a variant of RNN that makes it easier to retain long-
range interactions.
• Parameters of LSTM:
forget
new memory
weight of new memory (input)
output
,
,
,
,

23. LSTM application: enhancer/TF prediction
Input: 200bp sequence
Similar convolutional
architecture as before
Bi-directional LSTM
Output: 919 binary vector for
the presence of TF/chromatin
Quang and Xie. DanQ. 2016

24. Deep supervised learning
• Feedforward
• Convnets
• RNN, LSTM
Learning a nonlinear
mapping from inputs to
outputs.
Predicting:
TF binding,
gene expression,
disease status from images,
risk from SNPs,
protein structure
…

25. Deep unsupervised learning
• Nonlinear dimensional reduction and patterns mining.
• In many settings, have more unlabeled examples than labeled.
• Learn useful representations from unlabeled data.
• Better representation may improve prediction accuracy.

26. Low dimensional structure
What is the latent dimensionality of each row of images?
Urtasun and Zemel.

27. Autoencoder
ˆ
( )
encoding
decoding

28. Autoencoder
ˆ
( ) = ( · + )
ˆ = ( · + )
, = arg min
,
|| ˆ||
Train with backprop as before.

29. Autoencoder
ˆ
( )
, = arg min
,
|| ||
If encoding and decoding are linear
then
What does this remind you of?

30. Autoencoder
ˆ
( )
, = arg min
,
|| ||
If encoding and decoding are linear
then
Linear autoencoder is basically just
PCA!
General f and g corresponds to
nonlinear dimensional reduction.

31. What is wrong with this picture?
ˆ
( )

32. What is wrong with this picture?
ˆ
( )
h(X) can just copy X exactly!
Overcomplete. Need to impose
sparsity on h.

33. Denoising autoencoder
ˆ
( )
independent noise
0
0

34. Denoising autoencoder
ˆ
( )
independent noise
0
0

35. Illustration of denoising autoencoder
Figure from Hugo Larochelle

36. Filters from denoising autoencoder
Basis learned by
denoising autoencoder
Basis learned by weight-
decay autoencoder

37. Deep autoencoder
ˆ
ˆ

38. Deep autoencoder example
original
DAE
PCA
Hinton and Salakhutdinov. Science. 2016

39. Deep autoencoder example
Hinton and Salakhutdinov. Science. 2016
PCA
Deep autoencoder

40. Application: deep patient
Each patient = vector of 41k clinical descriptors
Stack of 3 denoising autoencoder
500 dim representation of each patient
Miotto et al. DeepPatient. 2016

41. Application: deep patient
500 dim representation of each patient
Random forest to predict future disease
Miotto et al. DeepPatient. 2016