The document provides an overview of deep learning topics focused on neural networks, including convolutional neural networks (CNN), recurrent neural networks (RNN), and long short-term memory (LSTM). It discusses architecture, training techniques, and applications such as image classification and sequence prediction, highlighting methods and best practices. Various concepts related to CNN and RNN are explored, emphasizing their roles in modern deep learning applications.

1. Deep Learning
ABDULRAZAK ZAKIEH

2. Content
Introduction
◦ Neural Networks (revision)
◦ Deep Learning
Convolutional Neural Networks (CNN)
Recurrent Neural Network (RNN)
◦ Long Short Term Memory (LSTM)
ABDULRAZAK ZAKIEH (ABDLARZAK.ZK@GMAIL.COM) 2

3. Content
Introduction
◦ Neural Networks (revision)
◦ Deep Learning
Convolutional Neural Networks (CNN)
Recurrent Neural Network (RNN)
◦ Long Short Term Memory (LSTM)
ABDULRAZAK ZAKIEH (ABDLARZAK.ZK@GMAIL.COM) 3

4. Neural Networks (revision)
Y
output
ABDULRAZAK ZAKIEH (ABDLARZAK.ZK@GMAIL.COM) 4

5. Neural Networks (revision)
ABDULRAZAK ZAKIEH (ABDLARZAK.ZK@GMAIL.COM) 5

6. Content
Introduction
◦ Neural Networks (revision)
◦ Deep Learning
Convolutional Neural Networks (CNN)
Recurrent Neural Network (RNN)
◦ Long Short Term Memory (LSTM)
ABDULRAZAK ZAKIEH (ABDLARZAK.ZK@GMAIL.COM) 6

7. Deep Learning
ABDULRAZAK ZAKIEH (ABDLARZAK.ZK@GMAIL.COM) 7

8. Deep Learning
ABDULRAZAK ZAKIEH (ABDLARZAK.ZK@GMAIL.COM) 8

9. Deep Learning
ABDULRAZAK ZAKIEH (ABDLARZAK.ZK@GMAIL.COM) 9

10. Deep Learning
AlexNet
ABDULRAZAK ZAKIEH (ABDLARZAK.ZK@GMAIL.COM) 10

11. Deep Learning
ABDULRAZAK ZAKIEH (ABDLARZAK.ZK@GMAIL.COM) 11

12. Deep Learning
ABDULRAZAK ZAKIEH (ABDLARZAK.ZK@GMAIL.COM) 12
256 * 256 RGB image
Overfit the data
does not capture the “natural” invariances we
expect in images (translation, scale)

13. Deep Learning
ABDULRAZAK ZAKIEH (ABDLARZAK.ZK@GMAIL.COM) 13

14. Content
Introduction
◦ Neural Networks (revision)
◦ Deep Learning
Convolutional Neural Networks (CNN)
Recurrent Neural Network (RNN)
Long Short Term Memory (LSTM)
ABDULRAZAK ZAKIEH (ABDLARZAK.ZK@GMAIL.COM) 14

15. Convolutional Neural Networks (CNN)
ABDULRAZAK ZAKIEH (ABDLARZAK.ZK@GMAIL.COM) 15

16. Convolutional Neural Networks (CNN)
ABDULRAZAK ZAKIEH (ABDLARZAK.ZK@GMAIL.COM) 16
y = z * w

17. Convolutional Neural Networks (CNN)
ABDULRAZAK ZAKIEH (ABDLARZAK.ZK@GMAIL.COM) 17

18. Convolutional Neural Networks (CNN)
ABDULRAZAK ZAKIEH (ABDLARZAK.ZK@GMAIL.COM) 18
“learn” the right filters for the specified task

19. Convolutional Neural Networks (CNN)
ABDULRAZAK ZAKIEH (ABDLARZAK.ZK@GMAIL.COM) 19
3D Convolution

20. Convolutional Neural Networks (CNN)
ABDULRAZAK ZAKIEH (ABDLARZAK.ZK@GMAIL.COM) 20
It’s correlation

21. Convolutional Neural Networks (CNN)
ABDULRAZAK ZAKIEH (ABDLARZAK.ZK@GMAIL.COM) 21
“zero pad” the input image

22. Convolutional Neural Networks (CNN)
ABDULRAZAK ZAKIEH (ABDLARZAK.ZK@GMAIL.COM) 22
max-pooling operation

23. Convolutional Neural Networks (CNN)
ABDULRAZAK ZAKIEH (ABDLARZAK.ZK@GMAIL.COM) 23
32 * 32 RGB
5 * 5 * 64 convolution
2 * 2 max pooling
3 * 3 * 128 convolution
2 * 2 max pooling
Fully-connected to 10-dimensional output
103
104
105
106

24. Convolutional Neural Networks (CNN)
ABDULRAZAK ZAKIEH (ABDLARZAK.ZK@GMAIL.COM) 24
Applications

25. Convolutional Neural Networks (CNN)
ABDULRAZAK ZAKIEH (ABDLARZAK.ZK@GMAIL.COM) 25
Applications
Using intermediate layers as features
Classify dogs/cats based upon 2000 images
(1000 of each class):
Approach 1: Convolution network from
scratch: 80%
Approach 2: Final-layer from VGG network ->
dense net: 90%

26. ABDULRAZAK ZAKIEH (ABDLARZAK.ZK@GMAIL.COM) 26

27. Content
Introduction
◦ Neural Networks (revision)
◦ Deep Learning
Convolutional Neural Networks (CNN)
Recurrent Neural Network (RNN)
◦ Long Short Term Memory (LSTM)
ABDULRAZAK ZAKIEH (ABDLARZAK.ZK@GMAIL.COM) 27

28. Recurrent Neural Network (RNN)
ABDULRAZAK ZAKIEH (ABDLARZAK.ZK@GMAIL.COM) 28
Predicting temporal data
Independent inputsPredict a sequence of outputs, given a sequence of inputs

29. Recurrent Neural Network (RNN)
ABDULRAZAK ZAKIEH (ABDLARZAK.ZK@GMAIL.COM) 29
Predicting temporal data

30. Recurrent Neural Network (RNN)
ABDULRAZAK ZAKIEH (ABDLARZAK.ZK@GMAIL.COM) 30

31. Recurrent Neural Network (RNN)
ABDULRAZAK ZAKIEH (ABDLARZAK.ZK@GMAIL.COM) 31
Training recurrent networks
“unroll” the RNN on some dataset, and
minimize the loss function

32. Recurrent Neural Network (RNN)
ABDULRAZAK ZAKIEH (ABDLARZAK.ZK@GMAIL.COM) 32

33. Recurrent Neural Network (RNN)
ABDULRAZAK ZAKIEH (ABDLARZAK.ZK@GMAIL.COM) 33
Initializing the first hidden layer?
The long of the sequence?

34. Recurrent Neural Network (RNN)
ABDULRAZAK ZAKIEH (ABDLARZAK.ZK@GMAIL.COM) 34
Trouble?
difficult to capture long-term dependencies

35. Content
Introduction
◦ Neural Networks (revision)
◦ Deep Learning
Convolutional Neural Networks (CNN)
Recurrent Neural Network (RNN)
◦ Long Short Term Memory (LSTM)
ABDULRAZAK ZAKIEH (ABDLARZAK.ZK@GMAIL.COM) 35

36. Long Short Term Memory (LSTM)
ABDULRAZAK ZAKIEH (ABDLARZAK.ZK@GMAIL.COM) 36
Figure from
(Jozefowicz et al., 2015)

37. Recurrent Neural Network (RNN)
ABDULRAZAK ZAKIEH (ABDLARZAK.ZK@GMAIL.COM) 37
Applications
Char-RNN (RNN (using stacked LSTMs))

38. Recurrent Neural Network (RNN)
ABDULRAZAK ZAKIEH (ABDLARZAK.ZK@GMAIL.COM) 38
Applications
Char-RNN (RNN (using stacked LSTMs))

39. Recurrent Neural Network (RNN)
ABDULRAZAK ZAKIEH (ABDLARZAK.ZK@GMAIL.COM) 39
Applications
Char-RNN (RNN (using stacked LSTMs))

40. Recurrent Neural Network (RNN)
ABDULRAZAK ZAKIEH (ABDLARZAK.ZK@GMAIL.COM) 40
Sequence to sequence models
LSTM without outputs
on “input” sequence
autoregressive
LSTM on output sequence

41. Recurrent Neural Network (RNN)
ABDULRAZAK ZAKIEH (ABDLARZAK.ZK@GMAIL.COM) 41
Sequence to sequence models - application
Google’s
machine
translation
methods

42. Combining RNNs and CNNs
ABDULRAZAK ZAKIEH (ABDLARZAK.ZK@GMAIL.COM) 42

43. Combining RNNs and CNNs
ABDULRAZAK ZAKIEH (ABDLARZAK.ZK@GMAIL.COM) 43

44. Thanks for listening
ABDULRAZAK ZAKIEH (ABDLARZAK.ZK@GMAIL.COM) 44