Slides for the ICLR 2022 paper "Minibatch vs Local SGD with Shuffling: Tight Convergence Bounds and Beyond" by Chulhee Yun (KAIST AI), Shashank Rajput (University of Wisconsin-Madison), Suvrit Sra (MIT). The slides were used for an oral presentation at ICLR 2022.

1. Minibatch vs Local SGD with Shuffling:
Tight Convergence Bounds and Beyond
Chulhee "Charlie" Yun (KAIST AI)
Shashank Rajput
(UW-Madison CS)
Suvrit Sra
(MIT LIDS/EECS)

2. Distributed/Federated Learning
2
…
Central
Server
Devices

3. Distributed/Federated Learning
2
…
Central
Server
Devices
F1
(x) :=
1
N
N
∑
i=1
f1
i (x)
F2
(x)
F3
(x)
FM−1
(x)
FM
(x)

4. Distributed/Federated Learning
2
…
Central
Server
Devices
F1
(x) :=
1
N
N
∑
i=1
f1
i (x)
F2
(x)
F3
(x)
FM−1
(x)
FM
(x)
minx F(x) :=
1
M
M
∑
m=1
Fm
(x)

5. Local SGD (Federated Averaging)
3

6. Local SGD (Federated Averaging)
Local SGD (aka federated averaging): run SGD locally, sync once in a while
3

7. Local SGD (Federated Averaging)
Local SGD (aka federated averaging): run SGD locally, sync once in a while
Analysis of local SGD under unbiased independent stochastic gradient estimates
[Dieuleveut & Patel, 2019; Haddadpour, Kamani, Mahdavi, Cadambe, 2019; Haddadpour & Mahdavi,
2019; Stich, 2019; Yu, Yang, Zhu, 2019; Li, Sahu, Zaheer, Sanjabi, Talwalkar, Smith, 2020; Li, Huang,
Yang, Wang, Zhang, 2020; Koloskova, Loizou, Boreiri, Jaggi, Stich, 2020; Khaled, Mishchenko, Richtarik,
2020; Spiridonoff, Olshevsky, Paschalidis, 2020; Karimireddy, Kale, Mohri, Reddi, Stich, Suresh, 2020;
Stich & Karimireddy, 2020; Qu, Lin, Kalagnanam, Li, Zhou, Zhou, 2020; Gorbunov, Hansel, Richtarik,
2021; Glasgow, Yuan, Ma, 2021; Qin, Etesami, Uribe, 2022; …]
3

8. Local SGD (Federated Averaging)
Local SGD (aka federated averaging): run SGD locally, sync once in a while
Analysis of local SGD under unbiased independent stochastic gradient estimates
[Dieuleveut & Patel, 2019; Haddadpour, Kamani, Mahdavi, Cadambe, 2019; Haddadpour & Mahdavi,
2019; Stich, 2019; Yu, Yang, Zhu, 2019; Li, Sahu, Zaheer, Sanjabi, Talwalkar, Smith, 2020; Li, Huang,
Yang, Wang, Zhang, 2020; Koloskova, Loizou, Boreiri, Jaggi, Stich, 2020; Khaled, Mishchenko, Richtarik,
2020; Spiridonoff, Olshevsky, Paschalidis, 2020; Karimireddy, Kale, Mohri, Reddi, Stich, Suresh, 2020;
Stich & Karimireddy, 2020; Qu, Lin, Kalagnanam, Li, Zhou, Zhou, 2020; Gorbunov, Hansel, Richtarik,
2021; Glasgow, Yuan, Ma, 2021; Qin, Etesami, Uribe, 2022; …]
Local SGD hard to beat minibatch SGD, though possible in some convex cases
[Woodworth, Patel, Stich, Dai, Bullins, Mcmahan, Shamir, Srebro, 2020; Woodworth, Patel, Srebro, 2020;
Woodworth, Bullins, Shamir, Srebro, 2021]
3

9. Random Reshuffling
Unbiased & independent noisy grad with-replacement sampling
, without-replacement sampling (shuffling)
=
≠
4

10. Random Reshuffling
Unbiased & independent noisy grad with-replacement sampling
, without-replacement sampling (shuffling)
=
≠
Random Reshuffling (shuffling-based SGD) converges faster than with-replacement
SGD if # epochs is large enough
[Gurbuzbalaban, Ozdaglar, Parrilo, 2019; Haochen & Sra, 2019; Nagaraj, Jain, Netrapalli, 2019;
Nguyen, Tran-Dinh, Phan, Nguyen, van Dijk, 2020; Safran & Shamir, 2020; 2021; Rajput, Gupta,
Papailiopoulos, 2020; Rajput, Lee, Papailiopoulos, 2021; Ahn, Yun, Sra, 2020; Yun, Sra, Jadbabaie,
2021; Mishchenko, Khaled, Richtarik, 2020; Tran, Nguyen, Tran-Dinh, 2021; …]
4

11. Random Reshuffling
Unbiased & independent noisy grad with-replacement sampling
, without-replacement sampling (shuffling)
=
≠
Random Reshuffling (shuffling-based SGD) converges faster than with-replacement
SGD if # epochs is large enough
[Gurbuzbalaban, Ozdaglar, Parrilo, 2019; Haochen & Sra, 2019; Nagaraj, Jain, Netrapalli, 2019;
Nguyen, Tran-Dinh, Phan, Nguyen, van Dijk, 2020; Safran & Shamir, 2020; 2021; Rajput, Gupta,
Papailiopoulos, 2020; Rajput, Lee, Papailiopoulos, 2021; Ahn, Yun, Sra, 2020; Yun, Sra, Jadbabaie,
2021; Mishchenko, Khaled, Richtarik, 2020; Tran, Nguyen, Tran-Dinh, 2021; …]
Only a couple of concurrent papers on Random Reshuffling in federated setup
[Mishchenko, Khaled, Richtarik, 2021; Malinovsky, Mishchenko, Richtarik, 2022; …]
4

12. Where This Paper Stands
5
Distributed SGD
with replacement
Single-machine SGD
without replacement
(aka Random Reshuffling)

13. Where This Paper Stands
5
Distributed SGD
with replacement
Single-machine SGD
without replacement
(aka Random Reshuffling)
Tight analysis of
Minibatch & Local
Random Reshuffling (RR)

14. Where This Paper Stands
5
Distributed SGD
with replacement
Single-machine SGD
without replacement
(aka Random Reshuffling)
Tight analysis of
Minibatch & Local
Random Reshuffling (RR)
·Upper bounds

40.
[Khaled et al., 2020; Spiridonoff et al., 2020; Qu et al., 2020]
Õ (
Lν2
μ2MNK
+
L2
ν2
B
μ3N2K2
+
L2
τ2
B2
μ3N2K2 ) K ≳ κ
# machines , # local components/machine , # epochs , # gradients/machine/comm

41.
[Khaled et al., 2020; Spiridonoff et al., 2020; Qu et al., 2020]
Õ (
Lν2
μ2MNK
+
L2
ν2
B
μ3N2K2
+
L2
τ2
B2
μ3N2K2 ) K ≳ κ
·Too large doesn't help : if , even when rate becomes
B B = Θ(N) τ = 0 Õ (
1
NK2 )
# machines , # local components/machine , # epochs , # gradients/machine/comm

42.
[Khaled et al., 2020; Spiridonoff et al., 2020; Qu et al., 2020]
Õ (
Lν2
μ2MNK
+
L2
ν2
B
μ3N2K2
+
L2
τ2
B2
μ3N2K2 ) K ≳ κ
·Too large doesn't help : if , even when rate becomes
B B = Θ(N) τ = 0 Õ (
1
NK2 )
·If and small, local RR can match minibatch RR (can't outperform)
B τ
# machines , # local components/machine , # epochs , # gradients/machine/comm

49.
(e.g., same dataset for all machines)
∀i, f1
i = f2
i = ⋯ = fM
i =: fi
M

50. Results: Synchronized Shuffling
17
Assume component-wise homogeneity
(e.g., same dataset for all machines)
∀i, f1
i = f2
i = ⋯ = fM
i =: fi
M
For any permutation , summing over an epoch gives full gradient:
σ
N
∑
i=1
∇fσ(i)(x) = N∇F(x)

51. Results: Synchronized Shuffling
17
Assume component-wise homogeneity
(e.g., same dataset for all machines)
∀i, f1
i = f2
i = ⋯ = fM
i =: fi
M
For any permutation , summing over an epoch gives full gradient:
σ
N
∑
i=1
∇fσ(i)(x) = N∇F(x)
Any ways to exploit the permutation identity "more often"?

52. Permutation Identity
N
∑
i=1
∇fσm
k (i)(x) = N ∇F(x)
Results: Synchronized Shuffling
18
Machine 1: fσ1
k (i) f3 f4 f5 f2 f6
f1
Machine 2: fσ2
k (i) f4 f1 f2 f3 f5
f6
Machine 3: fσ3
k (i) f6 f1 f4 f3 f2
f5
Independent Shuffling
σm
k ∼ Unif[Perm(N)]

53. Permutation Identity
N
∑
i=1
∇fσm
k (i)(x) = N ∇F(x)
Results: Synchronized Shuffling
18
Machine 1: fσ1
k (i) f3 f4 f5 f2 f6
f1
Machine 2: fσ2
k (i) f4 f1 f2 f3 f5
f6
Machine 3: fσ3
k (i) f6 f1 f4 f3 f2
f5
Synchronized Shuffling
σ ∼ Unif[Perm(N)], σm
k (i) := σ((i+
mN
M ) mod N)
Independent Shuffling
σm
k ∼ Unif[Perm(N)]

54. Permutation Identity
N
∑
i=1
∇fσm
k (i)(x) = N ∇F(x)
Results: Synchronized Shuffling
18
Machine 1: fσ1
k (i) f3 f4 f5 f2 f6
f1
Machine 2: fσ2
k (i) f4 f1 f2 f3 f5
f6
Machine 3: fσ3
k (i) f6 f1 f4 f3 f2
f5
Machine 1: fσ1
k (i) f3 f4 f5 f2 f6
f1
Machine 2: fσ2
k (i) f2 f6 f4 f1 f5
f3
Machine 3: fσ3
k (i) f1 f5 f6 f3 f4
f2
Synchronized Shuffling
σ ∼ Unif[Perm(N)], σm
k (i) := σ((i+
mN
M ) mod N)
Independent Shuffling
σm
k ∼ Unif[Perm(N)]

55. Permutation Identity
N
∑
i=1
∇fσm
k (i)(x) = N ∇F(x)
Results: Synchronized Shuffling
18
Machine 1: fσ1
k (i) f3 f4 f5 f2 f6
f1
Machine 2: fσ2
k (i) f4 f1 f2 f3 f5
f6
Machine 3: fσ3
k (i) f6 f1 f4 f3 f2
f5
Machine 1: fσ1
k (i) f3 f4 f5 f2 f6
f1
Machine 2: fσ2
k (i) f2 f6 f4 f1 f5
f3
Machine 3: fσ3
k (i) f1 f5 f6 f3 f4
f2
Get every iterations
N ∇F(x)
N
M
Synchronized Shuffling
σ ∼ Unif[Perm(N)], σm
k (i) := σ((i+
mN
M ) mod N)
Independent Shuffling
σm
k ∼ Unif[Perm(N)]

56. Results: Synchronized Shuffling
19
Minibatch RR for
Õ (
L2
ν2
μ3MNK2 ) K ≳ κ Local RR for
Õ (
L2
ν2
μ3MNK2
+ L2
ν2
B
μ3N2K2 ) K ≳ κ

57. Results: Synchronized Shuffling
19
Minibatch RR for
Õ (
L2
ν2
μ3MNK2 ) K ≳ κ Local RR for
Õ (
L2
ν2
μ3MNK2
+ L2
ν2
B
μ3N2K2 ) K ≳ κ
Minibatch RR for
Õ (
L2
ν2
μ3M2NK2 ) K ≳ κ Local RR for
Õ (
L2
ν2
μ3M2NK2
+
L2
ν2
B
μ3N2K2 ) K ≳ κ
+SyncShuf +SyncShuf

58. Results: Synchronized Shuffling
19
Minibatch RR for
Õ (
L2
ν2
μ3MNK2 ) K ≳ κ Local RR for
Õ (
L2
ν2
μ3MNK2
+ L2
ν2
B
μ3N2K2 ) K ≳ κ
Minibatch RR for
Õ (
L2
ν2
μ3M2NK2 ) K ≳ κ Local RR for
Õ (
L2
ν2
μ3M2NK2
+
L2
ν2
B
μ3N2K2 ) K ≳ κ
+SyncShuf +SyncShuf
·Bypass the factors in lower bounds
1
M

59. Results: Synchronized Shuffling
19
Minibatch RR for
Õ (
L2
ν2
μ3MNK2 ) K ≳ κ Local RR for
Õ (
L2
ν2
μ3MNK2
+ L2
ν2
B
μ3N2K2 ) K ≳ κ
Minibatch RR for
Õ (
L2
ν2
μ3M2NK2 ) K ≳ κ Local RR for
Õ (
L2
ν2
μ3M2NK2
+
L2
ν2
B
μ3N2K2 ) K ≳ κ
+SyncShuf +SyncShuf
·Bypass the factors in lower bounds
1
M
·Can allow "slight" component-wise heterogeneity

60. Thank you!
Minibatch vs Local SGD with Shuffling: Tight Convergence Bounds and Beyond
Chulhee Yun, Shashank Rajput, Suvrit Sra
https://arxiv.org/abs/2110.10342