The document discusses optimization techniques in machine learning, including gradient descent and its variants. It notes that machine learning aims to optimize an objective function by minimizing a loss function using gradient descent. Batch gradient descent and stochastic gradient descent are introduced as algorithms that use gradients to iteratively update model parameters toward minimizing the loss. Adam is presented as a stochastic optimization method. Deep learning models can now automatically learn their own optimization procedures rather than relying on hand-crafted optimization techniques.

1. Optimization
Via deep learning lens

2. Optimization

3. NIPS 2016 – LOCAL MINIMA?
https://www.youtube.com/watch?v=MRjwGZGicYA

4. Optimization…
https://www.youtube.com/watch?v=yxGyNv0Kjcs

5. Optimization…
https://www.youtube.com/watch?v=yxGyNv0Kjcs

6. Machine Learning이라는 것은…
• 목적 함수 f 를 최적화 ( minimize ) 시키는 것
• f(y’) = (y – y’) ^ 2 : loss function
• 미분 가능한 함수들에 대한 최적화는 Gradient
Descent를 많이 사용
https://www.youtube.com/watch?v=yxGyNv0Kjcs

7. NIPS 2016 – Gradient Descent
• Gradient Descent 알고리즘
1. 시작 포인트 x_0
2. 반복
-
https://www.youtube.com/watch?v=yxGyNv0Kjcs

8. NIPS 2016 – Gradient Descent
• Gradient Descent 알고리즘
https://www.youtube.com/watch?v=yxGyNv0Kjcs

9. NIPS 2016 – Gradient Descent
• Batch GD (BGD)
https://www.coursera.org/learn/machine-
learning/lecture/DoRHJ/stochastic-gradient-descent

10. NIPS 2016 – Gradient Descent
• Stochastic GD (SGD)
https://www.coursera.org/learn/machine-
learning/lecture/DoRHJ/stochastic-gradient-descent

11. NIPS 2016 – Gradient Descent
• Adam : a method for stochastic optimization
https://arxiv.org/pdf/1412.6980.pdf

12. Compare…

13. No more Hand-crafted
Optimization
https://arxiv.org/pdf/1606.04474.pdf

14. No more Hand Optimization
https://arxiv.org/pdf/1606.04474.pdf

15. No more Hand Optimization
https://arxiv.org/pdf/1606.04474.pdf
optimizer
optimizee

16. No more Hand Optimization
https://arxiv.org/pdf/1606.04474.pdf
RNN 모델
1
2
3
Quadratic : square loss

17. No more Hand Optimization
https://arxiv.org/pdf/1606.04474.pdf
RNN 모델
3

18. No more Hand Optimization

19. No more Hand Optimization

20. No more Hand Optimization