This document provides an overview of recurrent neural networks and long short-term memory (LSTM) networks. It discusses the vanishing gradient problem with traditional RNNs and how LSTMs address this issue using gating mechanisms. Several LSTM variations and practical intuition on LSTMs are also covered.

1. © SuperDataScienceDeep Learning A-Z

2. © SuperDataScienceDeep Learning A-Z

3. © SuperDataScienceDeep Learning A-Z
What we will learn in this section:
• The idea behind Recurrent Neural Networks
• The Vanishing Gradient Problem
• Long Short-Term Memory (LSTM)
• Practical Intuition
• EXTRA: LSTM Variations

4. © SuperDataScienceDeep Learning A-Z
Used for Regression & ClassificationArtificial Neural Networks
Used for Computer VisionConvolutional Neural Networks
Used for Time Series AnalysisRecurrent Neural Networks
Used for Feature DetectionSelf-Organizing Maps
Used for Recommendation SystemsDeep Boltzmann Machines
Used for Recommendation SystemsAutoEncoders
SupervisedUnsupervised

5. © SuperDataScienceDeep Learning A-Z
Image Source: Wikipedia
Cerebellum
Brainstem
Cerebrum
Frontal Lobe
Temporal Lobe
Parietal Lobe
Occipital Lobe

6. © SuperDataScienceDeep Learning A-Z
Input value 1
Input value 2
Input value 3
Output values
Input
Layer
Hidden
Layer
Output
Layer
4
5
6
7
5
6
2
1
1

7. © SuperDataScienceDeep Learning A-Z
456
Input value 1
Input value 2
Input value 3
Output values
Input
Layer
Hidden
Layer
Output
Layer
7 56211

8. © SuperDataScienceDeep Learning A-Z
456Input Vector
Output Vector
Input
Layer
Hidden
Layer
Output
Layer
7 56211

9. © SuperDataScienceDeep Learning A-Z
Input Vector
Output Vector
Input
Layer
Hidden
Layer
Output
Layer
7 51

10. © SuperDataScienceDeep Learning A-Z
751

11. © SuperDataScienceDeep Learning A-Z
517

12. © SuperDataScienceDeep Learning A-Z
517
Time

13. © SuperDataScienceDeep Learning A-Z
517
Time
517
517
517

14. © SuperDataScienceDeep Learning A-Z
517
Time
517
517
517
517
517
517
517
517
517
517
517
517
517
517
517
517
517
517
517
517
517
517
517

15. © SuperDataScienceDeep Learning A-Z
517
Time
517
517
517

16. © SuperDataScienceDeep Learning A-Z
Time
Many to Many
Time
One to Many
Time
Many to One
Time
Many to Many
Reference: karpathy.github.io
karpathy.github.io
dev.havenondemand.com

17. © SuperDataScienceDeep Learning A-Z
Sunspring (movie, 2016)
• Directed By Oscar Sharp
• Written by Benjamin
Link:
https://arstechnica.com/the-multiverse/2016/06/an-ai-wrote-this-movie-and-
its-strangely-moving/
Additional Reading:

18. © SuperDataScienceDeep Learning A-Z

19. © SuperDataScienceDeep Learning A-Z
Image Source: people.idsia.ch

20. © SuperDataScienceDeep Learning A-Z
Image Source: Université Montréal

21. © SuperDataScienceDeep Learning A-Z
Image Source: recode.net

22. © SuperDataScienceDeep Learning A-Z
C
ŷ
C = ½(ŷ- y)2

23. © SuperDataScienceDeep Learning A-Z
y
X2
X1
W11,1
W11,2
W11,3
W12,1
W12,2
W12,3
W21,1
W22,1
W23,1

24. © SuperDataScienceDeep Learning A-Z
517
Time
517
517
517
517
Win
Wout
Wrec
Win
Wout
Wrec
Win
Wout
Wrec
Win
Wout
Wrec
Win
Wout
yt
εt
xtxt-1xt-2xt-3
Formula Source: Razvan Pascanu et al. (2013)
εt+1εt-1εt-2εt-3
Wrec ~ small
Wrec ~ large
Vanishing
Exploding
Wrec Wrec Wrec

25. © SuperDataScienceDeep Learning A-Z
Solutions:
1. Exploding Gradient
• Weight Initialization
• Echo State Networks
• Long Short-Term Memory Networks (LSTMs)
• Truncated Backpropagation
• Penalties
• Gradient Clipping
2. Vanishing Gradient

26. © SuperDataScienceDeep Learning A-Z
Untersuchungen zu dynamischen
neuronalen Netzen
By Sepp (Josef) Hochreiter (1991)
Link:
http://people.idsia.ch/~juergen/SeppHochreiter1991ThesisAdvisorSchmidh
uber.pdf
Additional Reading:

27. © SuperDataScienceDeep Learning A-Z
Learning Long-Term
Dependencies with Gradient
Descent is Difficult
By Yoshua Bengio et al. (1994)
Link:
http://www-dsi.ing.unifi.it/~paolo/ps/tnn-94-gradient.pdf
Additional Reading:

28. © SuperDataScienceDeep Learning A-Z
On the difficulty of training recurrent
neural networks
By Razvan Pascanu et al. (2013)
Link:
http://www.jmlr.org/proceedings/papers/v28/pascanu13.pdf
Additional Reading:

29. © SuperDataScienceDeep Learning A-Z

30. © SuperDataScienceDeep Learning A-Z
Today:
• A bit of history
• LSTM Architecture
• Example walkthrough

31. © SuperDataScienceDeep Learning A-Z
517
Time
517
517
517
517
Win
Wout
Wrec
Win
Wout
Wrec
Win
Wout
Wrec
Win
Wout
Wrec
Win
Wout
yt
εt
xtxt-1xt-2xt-3
Formula Source: Razvan Pascanu et al. (2013)
εt+1εt-1εt-2εt-3
Wrec ~ small
Wrec ~ large
Vanishing
Exploding
Wrec Wrec Wrec
Wrec = 1
< 1
> 1

32. © SuperDataScienceDeep Learning A-Z
Image Sources: people.idsia.ch, ics.usi.ch

33. © SuperDataScienceDeep Learning A-Z
517
Time
517
517
517

34. © SuperDataScienceDeep Learning A-Z
Image Source: colah.github.io

35. © SuperDataScienceDeep Learning A-Z
Image Source: colah.github.io

36. © SuperDataScienceDeep Learning A-Z
Image Source: arxiv.org/pdf/1503.04069.pdf

37. © SuperDataScienceDeep Learning A-Z
Image Source: colah.github.io

38. © SuperDataScienceDeep Learning A-Z
Ct-1
Image Source: colah.github.io
Ct

39. © SuperDataScienceDeep Learning A-Z

40. © SuperDataScienceDeep Learning A-Z
Ct-1
Image Source: colah.github.io
Ct

41. © SuperDataScienceDeep Learning A-Z

42. © SuperDataScienceDeep Learning A-Z
Ct-1
Image Source: colah.github.io
Ct

43. © SuperDataScienceDeep Learning A-Z
Image Source: colah.github.io
Ct-1 Ct

44. © SuperDataScienceDeep Learning A-Z
Image Source: colah.github.io
Ct-1 Ct

45. © SuperDataScienceDeep Learning A-Z
Image Source: colah.github.io
Ct-1 Ct

46. © SuperDataScienceDeep Learning A-Z
Image Source: colah.github.io
Ct-1 Ct

47. © SuperDataScienceDeep Learning A-Z

48. © SuperDataScienceDeep Learning A-Z
CtCt-1

49. © SuperDataScienceDeep Learning A-Z
Long Short-Term Memory
By Sepp Hochreiter &
Jurgen Schmidhuber (1997)
Link:
http://www.bioinf.jku.at/publications/older/2604.pdf
Additional Reading:

50. © SuperDataScienceDeep Learning A-Z
Understanding LSTM Networks
By Christopher Olah (2015)
Link:
http://colah.github.io/posts/2015-08-Understanding-LSTMs/
Additional Reading:

51. © SuperDataScienceDeep Learning A-Z
Understanding LSTM and its
diagrams
By Shi Yan (2016)
Link:
https://medium.com/@shiyan/understanding-lstm-and-its-diagrams-
37e2f46f1714
Additional Reading:

52. © SuperDataScienceDeep Learning A-Z

53. © SuperDataScienceDeep Learning A-Z
Image Source: colah.github.io

54. © SuperDataScienceDeep Learning A-Z
Image Source: karpathy.github.io

55. © SuperDataScienceDeep Learning A-Z
Image Source: karpathy.github.io

56. © SuperDataScienceDeep Learning A-Z
Image Source: karpathy.github.io

57. © SuperDataScienceDeep Learning A-Z
Image Source: colah.github.io

58. © SuperDataScienceDeep Learning A-Z
Image Source: karpathy.github.io

59. © SuperDataScienceDeep Learning A-Z
The Unreasonable Effectiveness of
Recurrent Neural Networks
By Andrej Karpathy (2015)
Link:
http://karpathy.github.io/2015/05/21/rnn-effectiveness/
Additional Reading:

60. © SuperDataScienceDeep Learning A-Z
Visualizing and Understanding
Recurrent Networks
By Andrej Karpathy et al. (2015)
Link:
https://arxiv.org/pdf/1506.02078.pdf
Additional Reading:

61. © SuperDataScienceDeep Learning A-Z

62. © SuperDataScienceDeep Learning A-Z
Ct-1
Image Source: colah.github.io
Ct

63. © SuperDataScienceDeep Learning A-Z
Image Source: colah.github.io
Ct-1 Ct

64. © SuperDataScienceDeep Learning A-Z
Image Source: colah.github.io
Ct-1 Ct

65. © SuperDataScienceDeep Learning A-Z

66. © SuperDataScienceDeep Learning A-Z
LSTM: A Search Space Odyssey
By Klaus Greff et al. (2015)
Link:
https://arxiv.org/pdf/1503.04069.pdf
Additional Reading: