The document discusses optimization techniques for deep learning, specifically focusing on gradient descent and its variants such as stochastic gradient descent and mini-batch gradient descent. It emphasizes the importance of cost functions and learning rates in adjusting parameters to minimize costs efficiently. The comparison of gradient descent and its alternatives aims to highlight strategies for achieving better local minima more effectively.

1. Ali Madani
https://www.linkedin.com/in/amlearning/
Introduction to optimization
for deep learning

2. gradient descent
2
Obtaining the parameters in the direction of
maximum variation:
Cost function
Learning rate

3. gradient descent
3
Obtaining the parameters in the direction of
maximum variation:
Cost function
Learning rate
Changing the
parameters to minimize
the cost

4. gradient descent
4
Obtaining the parameters in the direction of
maximum variation:
Cost function
Learning rate
Summation over
all data points
● slow
● intractable

5. Stochastic gradient descent
5
Obtaining the parameters in the direction of
maximum variation:
Cost function
Learning rate

6. Stochastic gradient descent
6
Obtaining the parameters in the direction of
maximum variation:
Issue solved: parameter update for each training example
● Objective function fluctuates
○ Maybe get to a better local minima faster (by jumps)
Cost function
Learning rate

7. Comparison of gradient descent and stochastic
gradient descent
7

8. Mini-batch gradient descent
8
Let’s get the middle
Cost function
Learning rate

9. Mini-batch gradient descent
9
Let’s get the middle
Cost function
Learning rate
Update for mini-batches
of n training examples

10. Please share what you
learned with others