Basic introduction to first order optimization techniques in modern ML, Spring School on Algorithms for Big Data IIt Mumbai, Feb. 18-22.

1. Basic Concepts of Large Scale
Optimization for Machine Learning
Devdatt Dubhashi
AI and Data Science
Computer Science and Engineering
Chalmers
Machine Intelligence Sweden AB

2. Behind the Cat Pictures …
• Amazing successes of ML in
computer vision, natural
language processing …
• Underneath the hood is
optimization
• Large scale machine learning:
– large n (data points)
– large d (dimension)

3. Minimization of Finite Sums
• Assumptions on component functions: convex, smooth, …

4. Empirical Risk Minimization (ERM)
• Labelled training data:
• Parametrized class of prediction
functions:
• Empirical Loss:
min

5. Data Driven Clustering
• Given data points, cluster
• Classic K-means algorithm
• Needs to know k, number of
clusters
• Data driven clustering: find the
right number of clusters driven
by data. (Panahi, D: ICML 2017)

6. Minimization of Finite Sums
• Assumptions on component functions: convex, smooth, …

7. Mother of all First Order Methods:Gradient Descent

8. Gradient Descent Convergence
However, GD is not viable for large scale ML because each iteration has cost nd

9. Stochastic Gradient Descent (SGD)
Robbins and Munro 1950
• Index sampled uniformly at random with
replacement from [n]
• Cost per iteration is d
• Hugely successful in machine learning!

10. Stochastic, Batch and Full Gradient Descent
• Full GD:
• Minibatch GD:
• Stochastic GD:

11. The Unreasonable Effectiveness of SGD
• Very fast initial convergence
• Cheap O(d) per iteration as
opposed to O(nd) for full GD
• Very slow at the end ...
Convergence is only O(1/ 𝑘) for
smooth and O(1/k) for smooth
strongly convex functions.
• … but we do not need to run the
iterations to optimum, better to
stop early (Bottou and Bosquet)

12. SGD: Have the Cake and Eat it Too!
(Bottou and Bosquet 2008)

13. Variance/Noise Reduction
Can we improve convergence of SGD?

14. Variance/Noise Reduction

16. Variance Reduction: Three Takes

18. Nesterov Momentum for GD
Y. Nesterov, Doklady 1983.

19. Katyusha Momentum for SGD
Z. Allen-Zhu, STOC 2017, JMLR 2018

20. Non-smooth Objectives
• What if the objective is non-smooth?
• Need a proxy for gradients.
LASSO
SON Clustering

21. Proximal Operator
• Proximal operator:
• Special case: projection:
• Like a gradient step:
• Fixed points are minimizers:

22. Proximal Gradient Algorithm
• Objective split
• Iteration
• Special cases:
– g=0: usual gradient descent
– f=0: proximal algorithm
– g= indicator of convex set: projected gradient descent (constrained
optimization)
• Only works if the proximal operator can be evaluated efficiently!

23. PointSAGA: Stochastic Prox with Variance Reduction
Defazio 2016

24. MP-SAGA: Stochastic Prox with Variance Reduction
Panahi, Dubhashi, ICML 2017, (2019 under review: proximal operator in closed form!

25. SGD for Deep Learning
• SGD variants (Adagrad, RMSprop, Adam …) used to train
neural networks.
• Use aggressive adaptation with different learning rates for
different parameters.
• Theory says it shouldn’t work for highly nonconvex problems!
• But Adagrad greatly improved the robustness of SGD and
Google used it for training large-scale neural nets to recognize
cats

26. Variance Reduction for Deep Learning
Defazio, Bottou, 2019

27. References
Frances Bach Tutorials/Short Courses: https://www.di.ens.fr/~fbach/