This document discusses distributed optimization techniques in Spark. It provides examples of convex and spectral optimization problems and describes how gradient descent and PageRank algorithms can be optimized in a distributed setting. Specifically, it explains how caching neighbor lists in RAM and controlling RDD partitioning can avoid repeated network communication, improving performance of algorithms like PageRank that involve repeated joins or matrix multiplications.

1. Reza Zadeh
Introduction to Distributed
Optimization
@Reza_Zadeh | http://reza-zadeh.com

2. Optimization
At least two large classes of optimization
problems humans can solve:"
» Convex
» Spectral

3. Optimization Example: Gradient Descent

4. Logistic Regression
Already saw this with data scaling
Need to optimize with broadcast

5. Model Broadcast: LR

6. Model Broadcast: LR
Use
via
.value
Call
sc.broadcast

7. Separable Updates
Can be generalized for
» Unconstrained optimization
» Smooth or non-smooth
» LBFGS, Conjugate Gradient, Accelerated
Gradient methods, …

8. Optimization Example: Spectral Program

9. Spark PageRank
Given directed graph, compute node
importance. Two RDDs:
» Neighbors (a sparse graph/matrix)
» Current guess (a vector)
"
Using cache(), keep neighbor list in RAM

10. Spark PageRank
Using cache(), keep neighbor lists in RAM
Using partitioning, avoid repeated hashing
Neighbors
(id, edges)
Ranks
(id, rank)
join
partitionBy
join
join
…

11. Spark PageRank
Generalizes
to
Matrix
Multiplication,
opening
many
algorithms
from
Numerical
Linear
Algebra

12. Partitioning for PageRank
Recall from first lecture that network
bandwidth is ~100× as expensive as memory
bandwidth
One way Spark avoids using it is through
locality-aware scheduling for RAM and disk
Another important tool is controlling the
partitioning of RDD contents across nodes

13. Spark PageRank
Given directed graph, compute node
importance. Two RDDs:
» Neighbors (a sparse graph/matrix)
» Current guess (a vector)
Best representation for vector and matrix?

14. PageRank

15. Execution

16. Solution

17. New Execution

18. How it works
Each RDD has an optional Partitioner object"
Any shuffle operation on an RDD with a
Partitioner will respect that Partitioner"
Any shuffle operation on two RDDs will take
on the Partitioner of one of them, if one is set

19. Examples

20. Main Conclusion
Controlled partitioning can avoid unnecessary
all-to-all communication, saving computation
Repeated joins generalizes to repeated Matrix
Multiplication, opening many algorithms from
Numerical Linear Algebra

21. Performance

22. RDD partitioner

23. Custom Partitioning