This slides gives an introduction to the deep learning including cnn and rnn and lstm. To lead the readers to the deep learning field, it has started from the very basics of aritifical neural networks and then proceed to the convolutional neural networks, and then recurrent neural networks and lstm. I tried to my best to give credits to those who made the many slides I borrowed...But sometimes I couldn't remeber their exact source I borrowed...I am very sorry for this..and please let me know if you know their original source that I couldn't find....

2. 


3. 


•

5. <google trends on deep learning>

6. 제프리 힌튼
캐나다 토론토 대학
교수
얀 레쿤
뉴욕대학교 교수
앤드류
응
스탠포드
대학교 교수
요슈아
벤지오
캐나다 몬트리올
대학교 교수

17.  

23. W1
W2
W3
f(x)
1.4
-2.5
-0.06

24. 2.7
-8.6
0.002
f(x)
1.4
-2.5
-0.06
x = -0.06×2.7 + 2.5×8.6 + 1.4×0.002 = 21.34

25. Training data
Fields class
1.4 2.7 1.9 0
3.8 3.4 3.2 0
6.4 2.8 1.7 1
4.1 0.1 0.2 0
etc …
Initialise with random weights

26. Present a training pattern
Training data
Fields class
1.4 2.7 1.9 0
3.8 3.4 3.2 0
6.4 2.8 1.7 1
4.1 0.1 0.2 0
etc …
1.4
2.7
1.9

27. Training data
Fields class
1.4 2.7 1.9 0
3.8 3.4 3.2 0
6.4 2.8 1.7 1
4.1 0.1 0.2 0
etc …
Feed it through to get output
1.4
2.7 0.8
1.9

28. Compare with target output
Training data
Fields class
1.4 2.7 1.9 0
3.8 3.4 3.2 0
6.4 2.8 1.7 1
4.1 0.1 0.2 0
etc …
1.4
2.7 0.8
0
1.9 error 0.8

29. Training data
Fields class
1.4 2.7 1.9 0
3.8 3.4 3.2 0
6.4 2.8 1.7 1
4.1 0.1 0.2 0
etc …
Adjust weights based on error
1.4
2.7 0.8
0
1.9 error 0.8

30. Training data
Fields class
1.4 2.7 1.9 0
3.8 3.4 3.2 0
6.4 2.8 1.7 1
4.1 0.1 0.2 0
etc …
6.4
2.8
1.7
Present a training pattern

31. Training data
Fields class
1.4 2.7 1.9 0
3.8 3.4 3.2 0
6.4 2.8 1.7 1
4.1 0.1 0.2 0
etc …
Feed it through to get output
6.4
2.8 0.9
1.7

32. Training data
Fields class
1.4 2.7 1.9 0
3.8 3.4 3.2 0
6.4 2.8 1.7 1
4.1 0.1 0.2 0
etc …
Compare with target output
6.4
2.8 0.9
1
1.7 error -0.1

33. Adjust weights based on error
Training data
Fields class
1.4 2.7 1.9 0
3.8 3.4 3.2 0
6.4 2.8 1.7 1
4.1 0.1 0.2 0
etc …
6.4
2.8 0.9
1
1.7 error -0.1

34. And so on ….
Training data
Fields class
1.4 2.7 1.9 0
3.8 3.4 3.2 0
6.4 2.8 1.7 1
4.1 0.1 0.2 0
etc …
6.4
2.8 0.9
1
1.7 error -0.1
Repeat this thousands, maybe millions of times – each time
taking a random training instance, and making slight
weight adjustments
Algorithms for weight adjustment are designed to make
changes that will reduce the error

35. Initial random weights

36. Present a training instance / adjust the weights

37. Eventually ….

39. 1
63
…
1 5 10 15 20 25 …
strong +ve weight
low/zero weight

40. …
1 5 10 15 20 25 …
strong +ve weight
low/zero weight
1
63
it will send strong signal for a horizontal
line in the top row, ignoring everywhere else

41. …
1 5 10 15 20 25 …
strong +ve weight
low/zero weight
1
63

42. …
1 5 10 15 20 25 …
strong +ve weight
low/zero weight
1
63
Strong signal for a dark area in the top left
corner

43. etc …detect lines in
Specific positions
Higher level detetors
( horizontal line,
“RHS vertical lune”
“upper loop”, etc…
etc …

44. etc …detect lines in
Specific positions
Higher level detetors
( horizontal line,
“RHS vertical lune”
“upper loop”, etc…
etc …
What does this unit detect?

47. Training data
Fields class
1.4 2.7 1.9 0
3.8 3.4 3.2 0
6.4 2.8 1.7 1
4.1 0.1 0.2 0
etc …
Adjust weights based on error
1.4
2.7 0.8
0
1.9 error 0.8

48. Training data
Fields class
1.4 2.7 1.9 0
3.8 3.4 3.2 0
6.4 2.8 1.7 1
4.1 0.1 0.2 0
etc …
Adjust weights based on error
1.4
2.7 0.8
0
1.9 error 0.8
𝑐𝑜𝑠𝑡 = 0.8 − 0 2

49. Training data
Fields class
1.4 2.7 1.9 0
3.8 3.4 3.2 0
6.4 2.8 1.7 1
4.1 0.1 0.2 0
etc …
1.4
2.7 0.8
0
1.9 error 0.8
Adjust weights based on how much they contributed to the error

51. 

Neural network: w, b

52. Training data
Fields class
1.4 2.7 1.9 0
3.8 3.4 3.2 0
6.4 2.8 1.7 1
4.1 0.1 0.2 0
etc …
1.4
2.7 0.8
0
1.9 error 0.8
Adjust weights based on how much they
contributed to the error
𝜕𝐶
𝜕𝑤

57. What’s good can we have when we use logistic sigmoid function as our activation function?

64. 𝑤5

65. 𝑤5

66. 𝑤5

68. 𝑤5

70. 𝑤5

84. 


85. input
features
Abstracted features
More abstracted features
output

89. 

90. 




91. 

93. 



96. 




98. 



99. 



102.    



103.    
 

122. 

133. 

136. 


138. 

139. 

140. 

141. 

142. 

143. 

144. 

145. 

146. 

150. 

153. 강연



154. 



157. Source:http://lstm.iupr.com/

159. 
•

161. 강연




162. 강연


163. 강연

164. 

166. 강연




167. 강연


168. 강연

169. 강연

𝛿

170. 강연
 𝛿
𝛿𝑖 =

171. 강연

172. 강연


173. 강연


174. 강연




175. 강연

176. 강연


177. 강연

178. 강연


179. 강연


182. 강연 Source:http://lstm.iupr.com/

183. 강연 Source:http://lstm.iupr.com/

184. 강연 Source:http://lstm.iupr.com/

185. 강연

186. 강연



187. 강연

188. 강연

189. 강연

190. 강연

191. 강연
Peepholes are variations to the original LSTM

192. 강연
Peepholes are variations to the original LSTM

193. 강연

194. 강연
(1)
(2)
(3)
(4)
(5) (6)
In this case we have one cell in this memory block, thus C=1
For peepholes
(1)
(2)
(3)
(4)
(5)
(6)

195. 강연
(1)
(2)
(3)
(4)
(5)
(6)

196. 강연
(1)
(2)
(3)
(4)
(5)
(6)