Recommended
More Related Content
What's hot
Similar to 20190611 Study Neural Network
Similar to 20190611 Study Neural Network (20)
Recently uploaded
Recently uploaded (20)
20190611 Study Neural Network
- 2. AI/ 2006 DL:Deep Learning( ) Internet/Cloud GPU 2012 AlexNet 2016 AlphaGo
- 3. LSVRC(Large Scale Visual Recognition Challenge) ImageNet(http://www.image-net.org/) 1000 CNN AlexNet VGGNet ResNet GoogLeNet
- 6. RNN Seq2Seq(Sequence to Sequence) Encoder Decoder My name is Yamada
- 9. 1
- 10. 1 Run!!!
- 12. (Node)
- 14. y1 = w11x1 + w21x2 + w31x3 x1 x2 x3 w11 w21 w31 y1
- 15. y1 = w11x1 + w21x2 + w31x3 x1 x2 x3 w11 w21 w31 y1 W
- 16. x1 x2 x3 w11 w21 w31 y1 W y1 = w11x1 + w21x2 + w31x3
- 17. x1 x2 x3 w11 w21 w31 y1 W y1 = w11x1 + w21x2 + w31x3
- 18. x1 x2 x3 y1 = w11x1 + w21x2 + w31x3 y1
- 19. x1 x2 x3 y2 = w12x1 + w22x2 + w32x3 y1 y2
- 20. x1 x2 x3 y3 = w13x1 + w23x2 + w33x3 y1 y2 y3
- 21. x1 x2 x3 y4 = w14x1 + w24x2 + w34x3 y1 y2 y3 y4
- 22. x1 x2 x3 z1 = w11y1 + w21y2 + w31y3 + w41y4 y1 y2 y3 y4 z1
- 23. x1 x2 x3 z2 = w12y1 + w22y2 + w32y3 + w42y4 y1 y2 y3 y4 z1 z2
- 24. x1 x2 x3 z2 = w12y1 + w22y2 + w32y3 + w42y4 y1 y2 y3 y4 z1 z2 SoftMax
- 25. (Node) ( ) Deep Neural Network(DNN) →Deep Learning( ) (Weight)
- 27. or ( )
- 28. DATA
- 29. ( )
- 30. :(Layer )
- 31. :(Layer ) :( )
- 32. ( )
- 33. Click
- 36. Activation
- 37. y1 = w11x1 + w21x2 + w31x3 x1 x2 x3 w11 w21 w31 y1
- 38. y1 = f(w11x1 + w21x2 + w31x3) x1 x2 x3 w11 w21 w31 u f(u)
- 39. Linear( )
- 40. ReLU https://dashee87.github.io/deep%20learning/visualising-activation-functions-in-neural-networks/
- 41. Tanh https://dashee87.github.io/deep%20learning/visualising-activation-functions-in-neural-networks/
- 43. (Forward) → (Weight ) ✔ (Backward) ✔ (Epoch, mini Batch) ✔ (Learning Rate)
- 44. (Label)
- 45. (Forward)
- 46. (Forward)
- 47. (Forward)
- 48. (Forward)
- 50. (loss)
- 54. ( ) 3 train, (eval), test (over fitting)
- 55. Iteration weight (Forward/Backward:1 ) (mini)Batch size 1iteration Epoch 1Epoch =
- 56. MNIST( ) 60,000 10,000 5Epoch, Batch size=32 60000/32 = 1,875 iteration 5
- 57. MNIST( ) 60,000 10,000 5Epoch, Batch size=32 60000/32 = 1,875 iteration 5
- 58. (Learning Rate) weight = = Epoch
- 59. (train vs test)
- 60. Epoch Batch size
- 61. Loss
- 62. Learning rate
- 63. Training( Show test data test ) batch size=10, 30 training loss=0.001 Epoch Learning rate=1 Training Learning rate=0.001 Training
- 66. / df(x) dx = lim h→0 f(x + h) − f(x) h df(x) dx = lim h→0 f(x + h) − f(x − h) 2h
- 67. f(x0, x1) = x2 0 + x2 1 ( ∂f ∂x0 , ∂f ∂x1 )
- 68. (gradient) f(x0, x1) = x2 0 + x2 1 f(x0, x1) = sin(x0) + cos(x1)
- 72. < >
- 73. < >
- 74. x0 = x0 − ∂f ∂x0 x0 = x0−η ∂f ∂x0 η: (Learning Rate)
- 75. η = 0.01f(x0, x1) = x2 0 + x2 1
- 76. η = 1.1f(x0, x1) = x2 0 + x2 1
- 77. η = 0.001f(x0, x1) = x2 0 + x2 1
- 78. Optimizer Y = WX W = w11 w12 w13 w14 w21 w22 w23 w24 w31 w32 w33 w34 x1 x2 x3 y1 y2 y3 y4
- 79. SGD Stochastic Gradient Descent W ← W − η ∂L ∂W
- 80. SGD Stochastic Gradient Descent W ← W − η ∂L ∂W
- 81. Momentum v ← av − η ∂L ∂W W ← W + v
- 82. AdaGrad h ← h + ∂L ∂W ⊙ ∂L ∂W W ← W − η 1 h ∂L ∂W
- 83. Adam Momentum+AdaGrad mt ← β1mt−1 + (1 − β1)gt θt+1 ← θt − α ̂mt ̂vt + ϵ vt ← β2vt−1 + (1 − β2)g2 t ̂mt ← mt 1 − βt 1 ̂vt ← vt 1 − βt 2 β1 = 0.9,β2 = 0.999,ϵ = 10−8 gt = ∂L ∂W θt = W α = η αt = α (1 − βt 2) (1 − βt 1) θt+1 ← θt − αt mt vt + ̂ϵ
- 84. Adam Momentum+AdaGrad mt ← β1mt−1 + (1 − β1) ∂L ∂W vt ← β2vt−1 + (1 − β2)( ∂L ∂W ⊙ ∂L ∂W ) ηt = η (1 − βt 2) (1 − βt 1) W ← W − ηt mt vt + ̂ϵ
- 86. Optimizer - RMSprop, Adadelta, AMSGrad - Adabound, AMSbound
- 88. Adabound ←β α( )
- 89. Adabound
- 90. Weight Decay (overfitting) (Weight) SGD W ← W − η ∂L ∂W − ηλW λ :Weight Decay
- 91. Loss L2 W ← W − η ∂L ∂W − ηλW ̂L(W) = L(W) + λ 2 ∥W∥2 L2 (Ridge): L1 (LASSO): w2 1 + w2 2 + ⋯ + w2 n |w1 | + |w2 | + ⋯ + |wn |
- 92. Tips:Adam ? Adam Optimizer Weight Decay !? Decoupled Weight Decay Regularization https://arxiv.org/abs/1711.05101
- 93. ( ) Y = (WX + b)2 s = z + b Y = s2 z = WX
- 94. ( ) ✖ ^2X Y W b s = z + b Y = s2z = WX z s
- 95. ( ) ✖ ^2X Y W b s = z + b Y = s2z = WX z s ∂L ∂Y
- 96. ( ) ✖ ^2X Y W b s = z + b Y = s2z = WX z s ∂L ∂Y ∂Y ∂s = 2s 2s ∂L ∂Y
- 97. ( ) ✖ ^2X Y W b s = z + b Y = s2z = WX z s ∂L ∂Y 2s ∂L ∂Y 2s ∂L ∂Y 2s ∂L ∂Y
- 98. ( ) ✖ ^2X Y W b s = z + b Y = s2z = WX z s ∂L ∂Y 2s ∂L ∂Y 2s ∂L ∂Y 2s ∂L ∂Y ∂z ∂X 2s ∂L ∂Y
- 99. ( ) ✖ ^2X Y W b s = z + b Y = s2z = WX z s ∂L ∂Y 2s ∂L ∂Y 2s ∂L ∂Y 2s ∂L ∂Y ∂z ∂X 2s ∂L ∂Y ∂z ∂W 2s ∂L ∂Y
- 100. (/ ) ✖ ^2X Y W b z s ∂L ∂Y 2s ∂L ∂Y 2s ∂L ∂Y 2s ∂L ∂Y ∂z ∂X 2s ∂L ∂Y ∂z ∂W 2s ∂L ∂Y ( ) ZERO