Although stochastic gradient descent (SGD) resolves some issues of gradient descent (GD), as a slow and costly process, it results in fluctuations around the local minima. Mini-batch gradient descent was suggested to get the middle point reducing fluctuations observed in SGD and cost of GD. Benefit of mini-batch gradient descent is usually reported in deep learning (DL) setting as DL models usually are built training on large datasets.

1. Ali Madani
https://www.linkedin.com/in/amlearning/
Introduction to optimization
for deep learning

2. gradient descent
2
Obtaining the parameters in the direction of
maximum variation:
Cost function
Learning rate

3. gradient descent
3
Obtaining the parameters in the direction of
maximum variation:
Cost function
Learning rate
Changing the
parameters to minimize
the cost

4. gradient descent
4
Obtaining the parameters in the direction of
maximum variation:
Cost function
Learning rate
Summation over
all data points
● slow
● intractable

5. Stochastic gradient descent
5
Obtaining the parameters in the direction of
maximum variation:
Cost function
Learning rate

6. Stochastic gradient descent
6
Obtaining the parameters in the direction of
maximum variation:
Issue solved: parameter update for each training example
● Objective function fluctuates
○ Maybe get to a better local minima faster (by jumps)
Cost function
Learning rate

7. Comparison of gradient descent and stochastic
gradient descent
7

8. Mini-batch gradient descent
8
Let’s get the middle
Cost function
Learning rate

9. Mini-batch gradient descent
9
Let’s get the middle
Cost function
Learning rate
Update for mini-batches
of n training examples

10. Please share what you
learned with others