In these slides, I made a review of Neural Networks, single layer, and multi-layer. Then, I talked about Deep Neural Networks (DNN) in general and some of the most famous DNNs where I introduce Convolutional Neural Networks (CNN) and its applications on images. Later, I talk briefly about Recurrent Neural Network, its benefits, specifications, how to train it, and its problem. In addition, I explained a bit about Long-short Term Memory (LSTM) and its feature and applications. Finally, I presented about some of the benefits of combining CNNs and RNNs.

1. Deep Learning
ABDULRAZAK ZAKIEH

2. Content
Introduction
◦ Neural Networks (revision)
◦ Deep Learning
Convolutional Neural Networks (CNN)
Recurrent Neural Network (RNN)
◦ Long Short Term Memory (LSTM)
ABDULRAZAK ZAKIEH (ABDLARZAK.ZK@GMAIL.COM) 2

3. Content
Introduction
◦ Neural Networks (revision)
◦ Deep Learning
Convolutional Neural Networks (CNN)
Recurrent Neural Network (RNN)
◦ Long Short Term Memory (LSTM)
ABDULRAZAK ZAKIEH (ABDLARZAK.ZK@GMAIL.COM) 3

4. Neural Networks (revision)
Y
output
ABDULRAZAK ZAKIEH (ABDLARZAK.ZK@GMAIL.COM) 4

5. Neural Networks (revision)
ABDULRAZAK ZAKIEH (ABDLARZAK.ZK@GMAIL.COM) 5

6. Content
Introduction
◦ Neural Networks (revision)
◦ Deep Learning
Convolutional Neural Networks (CNN)
Recurrent Neural Network (RNN)
◦ Long Short Term Memory (LSTM)
ABDULRAZAK ZAKIEH (ABDLARZAK.ZK@GMAIL.COM) 6

7. Deep Learning
ABDULRAZAK ZAKIEH (ABDLARZAK.ZK@GMAIL.COM) 7

8. Deep Learning
ABDULRAZAK ZAKIEH (ABDLARZAK.ZK@GMAIL.COM) 8

9. Deep Learning
ABDULRAZAK ZAKIEH (ABDLARZAK.ZK@GMAIL.COM) 9

10. Deep Learning
AlexNet
ABDULRAZAK ZAKIEH (ABDLARZAK.ZK@GMAIL.COM) 10

11. Deep Learning
ABDULRAZAK ZAKIEH (ABDLARZAK.ZK@GMAIL.COM) 11

12. Deep Learning
ABDULRAZAK ZAKIEH (ABDLARZAK.ZK@GMAIL.COM) 12
256 * 256 RGB image
Overfit the data
does not capture the “natural” invariances we
expect in images (translation, scale)

13. Deep Learning
ABDULRAZAK ZAKIEH (ABDLARZAK.ZK@GMAIL.COM) 13

14. Content
Introduction
◦ Neural Networks (revision)
◦ Deep Learning
Convolutional Neural Networks (CNN)
Recurrent Neural Network (RNN)
Long Short Term Memory (LSTM)
ABDULRAZAK ZAKIEH (ABDLARZAK.ZK@GMAIL.COM) 14

15. Convolutional Neural Networks (CNN)
ABDULRAZAK ZAKIEH (ABDLARZAK.ZK@GMAIL.COM) 15

16. Convolutional Neural Networks (CNN)
ABDULRAZAK ZAKIEH (ABDLARZAK.ZK@GMAIL.COM) 16
y = z * w

17. Convolutional Neural Networks (CNN)
ABDULRAZAK ZAKIEH (ABDLARZAK.ZK@GMAIL.COM) 17

18. Convolutional Neural Networks (CNN)
ABDULRAZAK ZAKIEH (ABDLARZAK.ZK@GMAIL.COM) 18
“learn” the right filters for the specified task

19. Convolutional Neural Networks (CNN)
ABDULRAZAK ZAKIEH (ABDLARZAK.ZK@GMAIL.COM) 19
3D Convolution

20. Convolutional Neural Networks (CNN)
ABDULRAZAK ZAKIEH (ABDLARZAK.ZK@GMAIL.COM) 20
It’s correlation

21. Convolutional Neural Networks (CNN)
ABDULRAZAK ZAKIEH (ABDLARZAK.ZK@GMAIL.COM) 21
“zero pad” the input image

22. Convolutional Neural Networks (CNN)
ABDULRAZAK ZAKIEH (ABDLARZAK.ZK@GMAIL.COM) 22
max-pooling operation

23. Convolutional Neural Networks (CNN)
ABDULRAZAK ZAKIEH (ABDLARZAK.ZK@GMAIL.COM) 23
32 * 32 RGB
5 * 5 * 64 convolution
2 * 2 max pooling
3 * 3 * 128 convolution
2 * 2 max pooling
Fully-connected to 10-dimensional output
103
104
105
106

24. Convolutional Neural Networks (CNN)
ABDULRAZAK ZAKIEH (ABDLARZAK.ZK@GMAIL.COM) 24
Applications

25. Convolutional Neural Networks (CNN)
ABDULRAZAK ZAKIEH (ABDLARZAK.ZK@GMAIL.COM) 25
Applications
Using intermediate layers as features
Classify dogs/cats based upon 2000 images
(1000 of each class):
Approach 1: Convolution network from
scratch: 80%
Approach 2: Final-layer from VGG network ->
dense net: 90%

26. ABDULRAZAK ZAKIEH (ABDLARZAK.ZK@GMAIL.COM) 26

27. Content
Introduction
◦ Neural Networks (revision)
◦ Deep Learning
Convolutional Neural Networks (CNN)
Recurrent Neural Network (RNN)
◦ Long Short Term Memory (LSTM)
ABDULRAZAK ZAKIEH (ABDLARZAK.ZK@GMAIL.COM) 27

28. Recurrent Neural Network (RNN)
ABDULRAZAK ZAKIEH (ABDLARZAK.ZK@GMAIL.COM) 28
Predicting temporal data
Independent inputsPredict a sequence of outputs, given a sequence of inputs

29. Recurrent Neural Network (RNN)
ABDULRAZAK ZAKIEH (ABDLARZAK.ZK@GMAIL.COM) 29
Predicting temporal data

30. Recurrent Neural Network (RNN)
ABDULRAZAK ZAKIEH (ABDLARZAK.ZK@GMAIL.COM) 30

31. Recurrent Neural Network (RNN)
ABDULRAZAK ZAKIEH (ABDLARZAK.ZK@GMAIL.COM) 31
Training recurrent networks
“unroll” the RNN on some dataset, and
minimize the loss function

32. Recurrent Neural Network (RNN)
ABDULRAZAK ZAKIEH (ABDLARZAK.ZK@GMAIL.COM) 32

33. Recurrent Neural Network (RNN)
ABDULRAZAK ZAKIEH (ABDLARZAK.ZK@GMAIL.COM) 33
Initializing the first hidden layer?
The long of the sequence?

34. Recurrent Neural Network (RNN)
ABDULRAZAK ZAKIEH (ABDLARZAK.ZK@GMAIL.COM) 34
Trouble?
difficult to capture long-term dependencies

35. Content
Introduction
◦ Neural Networks (revision)
◦ Deep Learning
Convolutional Neural Networks (CNN)
Recurrent Neural Network (RNN)
◦ Long Short Term Memory (LSTM)
ABDULRAZAK ZAKIEH (ABDLARZAK.ZK@GMAIL.COM) 35

36. Long Short Term Memory (LSTM)
ABDULRAZAK ZAKIEH (ABDLARZAK.ZK@GMAIL.COM) 36
Figure from
(Jozefowicz et al., 2015)

37. Recurrent Neural Network (RNN)
ABDULRAZAK ZAKIEH (ABDLARZAK.ZK@GMAIL.COM) 37
Applications
Char-RNN (RNN (using stacked LSTMs))

38. Recurrent Neural Network (RNN)
ABDULRAZAK ZAKIEH (ABDLARZAK.ZK@GMAIL.COM) 38
Applications
Char-RNN (RNN (using stacked LSTMs))

39. Recurrent Neural Network (RNN)
ABDULRAZAK ZAKIEH (ABDLARZAK.ZK@GMAIL.COM) 39
Applications
Char-RNN (RNN (using stacked LSTMs))

40. Recurrent Neural Network (RNN)
ABDULRAZAK ZAKIEH (ABDLARZAK.ZK@GMAIL.COM) 40
Sequence to sequence models
LSTM without outputs
on “input” sequence
autoregressive
LSTM on output sequence

41. Recurrent Neural Network (RNN)
ABDULRAZAK ZAKIEH (ABDLARZAK.ZK@GMAIL.COM) 41
Sequence to sequence models - application
Google’s
machine
translation
methods

42. Combining RNNs and CNNs
ABDULRAZAK ZAKIEH (ABDLARZAK.ZK@GMAIL.COM) 42

43. Combining RNNs and CNNs
ABDULRAZAK ZAKIEH (ABDLARZAK.ZK@GMAIL.COM) 43

44. Thanks for listening
ABDULRAZAK ZAKIEH (ABDLARZAK.ZK@GMAIL.COM) 44