Recommended
More Related Content
Similar to Minibatch vs Local SGD with Shuffling: Tight Convergence Bounds and Beyond
Similar to Minibatch vs Local SGD with Shuffling: Tight Convergence Bounds and Beyond (20)
Recently uploaded
Recently uploaded (20)
Minibatch vs Local SGD with Shuffling: Tight Convergence Bounds and Beyond
- 4. Distributed/Federated Learning 2 … Central Server Devices F1 (x) := 1 N N ∑ i=1 f1 i (x) F2 (x) F3 (x) FM−1 (x) FM (x) minx F(x) := 1 M M ∑ m=1 Fm (x)
- 40. [Khaled et al., 2020; Spiridonoff et al., 2020; Qu et al., 2020] Õ ( Lν2 μ2MNK + L2 ν2 B μ3N2K2 + L2 τ2 B2 μ3N2K2 ) K ≳ κ # machines , # local components/machine , # epochs , # gradients/machine/comm
- 41. [Khaled et al., 2020; Spiridonoff et al., 2020; Qu et al., 2020] Õ ( Lν2 μ2MNK + L2 ν2 B μ3N2K2 + L2 τ2 B2 μ3N2K2 ) K ≳ κ ·Too large doesn't help : if , even when rate becomes B B = Θ(N) τ = 0 Õ ( 1 NK2 ) # machines , # local components/machine , # epochs , # gradients/machine/comm
- 42. [Khaled et al., 2020; Spiridonoff et al., 2020; Qu et al., 2020] Õ ( Lν2 μ2MNK + L2 ν2 B μ3N2K2 + L2 τ2 B2 μ3N2K2 ) K ≳ κ ·Too large doesn't help : if , even when rate becomes B B = Θ(N) τ = 0 Õ ( 1 NK2 ) ·If and small, local RR can match minibatch RR (can't outperform) B τ # machines , # local components/machine , # epochs , # gradients/machine/comm
- 49. (e.g., same dataset for all machines) ∀i, f1 i = f2 i = ⋯ = fM i =: fi M
- 50. Results: Synchronized Shuffling 17 Assume component-wise homogeneity (e.g., same dataset for all machines) ∀i, f1 i = f2 i = ⋯ = fM i =: fi M For any permutation , summing over an epoch gives full gradient: σ N ∑ i=1 ∇fσ(i)(x) = N∇F(x)
- 51. Results: Synchronized Shuffling 17 Assume component-wise homogeneity (e.g., same dataset for all machines) ∀i, f1 i = f2 i = ⋯ = fM i =: fi M For any permutation , summing over an epoch gives full gradient: σ N ∑ i=1 ∇fσ(i)(x) = N∇F(x) Any ways to exploit the permutation identity "more often"?
- 52. Permutation Identity N ∑ i=1 ∇fσm k (i)(x) = N ∇F(x) Results: Synchronized Shuffling 18 Machine 1: fσ1 k (i) f3 f4 f5 f2 f6 f1 Machine 2: fσ2 k (i) f4 f1 f2 f3 f5 f6 Machine 3: fσ3 k (i) f6 f1 f4 f3 f2 f5 Independent Shuffling σm k ∼ Unif[Perm(N)]
- 53. Permutation Identity N ∑ i=1 ∇fσm k (i)(x) = N ∇F(x) Results: Synchronized Shuffling 18 Machine 1: fσ1 k (i) f3 f4 f5 f2 f6 f1 Machine 2: fσ2 k (i) f4 f1 f2 f3 f5 f6 Machine 3: fσ3 k (i) f6 f1 f4 f3 f2 f5 Synchronized Shuffling σ ∼ Unif[Perm(N)], σm k (i) := σ((i+ mN M ) mod N) Independent Shuffling σm k ∼ Unif[Perm(N)]
- 54. Permutation Identity N ∑ i=1 ∇fσm k (i)(x) = N ∇F(x) Results: Synchronized Shuffling 18 Machine 1: fσ1 k (i) f3 f4 f5 f2 f6 f1 Machine 2: fσ2 k (i) f4 f1 f2 f3 f5 f6 Machine 3: fσ3 k (i) f6 f1 f4 f3 f2 f5 Machine 1: fσ1 k (i) f3 f4 f5 f2 f6 f1 Machine 2: fσ2 k (i) f2 f6 f4 f1 f5 f3 Machine 3: fσ3 k (i) f1 f5 f6 f3 f4 f2 Synchronized Shuffling σ ∼ Unif[Perm(N)], σm k (i) := σ((i+ mN M ) mod N) Independent Shuffling σm k ∼ Unif[Perm(N)]
- 55. Permutation Identity N ∑ i=1 ∇fσm k (i)(x) = N ∇F(x) Results: Synchronized Shuffling 18 Machine 1: fσ1 k (i) f3 f4 f5 f2 f6 f1 Machine 2: fσ2 k (i) f4 f1 f2 f3 f5 f6 Machine 3: fσ3 k (i) f6 f1 f4 f3 f2 f5 Machine 1: fσ1 k (i) f3 f4 f5 f2 f6 f1 Machine 2: fσ2 k (i) f2 f6 f4 f1 f5 f3 Machine 3: fσ3 k (i) f1 f5 f6 f3 f4 f2 Get every iterations N ∇F(x) N M Synchronized Shuffling σ ∼ Unif[Perm(N)], σm k (i) := σ((i+ mN M ) mod N) Independent Shuffling σm k ∼ Unif[Perm(N)]
- 56. Results: Synchronized Shuffling 19 Minibatch RR for Õ ( L2 ν2 μ3MNK2 ) K ≳ κ Local RR for Õ ( L2 ν2 μ3MNK2 + L2 ν2 B μ3N2K2 ) K ≳ κ
- 57. Results: Synchronized Shuffling 19 Minibatch RR for Õ ( L2 ν2 μ3MNK2 ) K ≳ κ Local RR for Õ ( L2 ν2 μ3MNK2 + L2 ν2 B μ3N2K2 ) K ≳ κ Minibatch RR for Õ ( L2 ν2 μ3M2NK2 ) K ≳ κ Local RR for Õ ( L2 ν2 μ3M2NK2 + L2 ν2 B μ3N2K2 ) K ≳ κ +SyncShuf +SyncShuf
- 58. Results: Synchronized Shuffling 19 Minibatch RR for Õ ( L2 ν2 μ3MNK2 ) K ≳ κ Local RR for Õ ( L2 ν2 μ3MNK2 + L2 ν2 B μ3N2K2 ) K ≳ κ Minibatch RR for Õ ( L2 ν2 μ3M2NK2 ) K ≳ κ Local RR for Õ ( L2 ν2 μ3M2NK2 + L2 ν2 B μ3N2K2 ) K ≳ κ +SyncShuf +SyncShuf ·Bypass the factors in lower bounds 1 M
- 59. Results: Synchronized Shuffling 19 Minibatch RR for Õ ( L2 ν2 μ3MNK2 ) K ≳ κ Local RR for Õ ( L2 ν2 μ3MNK2 + L2 ν2 B μ3N2K2 ) K ≳ κ Minibatch RR for Õ ( L2 ν2 μ3M2NK2 ) K ≳ κ Local RR for Õ ( L2 ν2 μ3M2NK2 + L2 ν2 B μ3N2K2 ) K ≳ κ +SyncShuf +SyncShuf ·Bypass the factors in lower bounds 1 M ·Can allow "slight" component-wise heterogeneity