Download as PDF, PPTX
![Distributed Vector
Driver
[2.5, 6.8, 3.1, 9.0, 0.7, 1.9, 2.6, 8.7, 6.2, 1.1, 5.0, 0.0]
worker0 worker1 worker2 worker3](https://image.slidesharecdn.com/5302yanboliang-170215231221/75/Scaling-Apache-Spark-MLlib-to-Billions-of-Parameters-Spark-Summit-East-talk-by-Yanbo-Liang-13-2048.jpg)
![Distributed Vector
Driver
[2.5, 6.8, 3.1, 9.0, 0.7, 1.9, 2.6, 8.7, 6.2, 1.1, 5.0, 0.0]
[2.5, 6.8, 3.1]
worker0
[9.0, 0.7, 1.9]
worker1
[2.6, 8.7, 6.2]
worker2
[1.1, 5.0, 0.0]
worker3](https://image.slidesharecdn.com/5302yanboliang-170215231221/75/Scaling-Apache-Spark-MLlib-to-Billions-of-Parameters-Spark-Summit-East-talk-by-Yanbo-Liang-14-2048.jpg)
![Distributed Vector
• Definition:
• Linear algebra operations:
– a * x + y
– a * x + b * y + c * z + …
– x.dot(y)
– x.norm
– …
class DistribuedVector(
val values: RDD[Vector],
val sizePerPart: Int,
val numPartitions: Int,
val size: Long)](https://image.slidesharecdn.com/5302yanboliang-170215231221/75/Scaling-Apache-Spark-MLlib-to-Billions-of-Parameters-Spark-Summit-East-talk-by-Yanbo-Liang-15-2048.jpg)
![APIs
val dataset: Dataset[_] =
spark.read.format("libsvm").load("data/a9a")
val trainer = new VLogisticRegression()
.setColsPerBlock(100)
.setRowsPerBlock(10)
.setColPartitions(3)
.setRowPartitions(3)
.setRegParam(0.5)
val model = trainer.fit(dataset)
println(s"Vector-free logistic regression coefficients:
${model.coefficients}")](https://image.slidesharecdn.com/5302yanboliang-170215231221/75/Scaling-Apache-Spark-MLlib-to-Billions-of-Parameters-Spark-Summit-East-talk-by-Yanbo-Liang-36-2048.jpg)
![APIs
val dataset: Dataset[_] =
spark.read.format("libsvm").load("data/a9a")
val trainer = new LogisticRegression()
.setColsPerBlock(100)
.setRowsPerBlock(10)
.setColPartitions(3)
.setRowPartitions(3)
.setRegParam(0.5)
.setSolver(“vl-bfgs”)
val model = trainer.fit(dataset)
println(s"Vector-free logistic regression coefficients:
${model.coefficients}")](https://image.slidesharecdn.com/5302yanboliang-170215231221/75/Scaling-Apache-Spark-MLlib-to-Billions-of-Parameters-Spark-Summit-East-talk-by-Yanbo-Liang-39-2048.jpg)
Uploaded bySpark Summit
Scaling Apache Spark MLlib to Billions of Parameters: Spark Summit East talk by Yanbo Liang
The document discusses scaling Apache Spark's MLlib to handle billions of parameters using a vector-free L-BFGS algorithm for logistic regression. It outlines performance metrics, integration with existing MLlib, and future work opportunities while emphasizing a full distributed computation model without requiring special resources. Key takeaways include the API consistency with Breeze L-BFGS and its deployment ease within Spark clusters.
More Related Content
Similar to Scaling Apache Spark MLlib to Billions of Parameters: Spark Summit East talk by Yanbo Liang
Scaling Apache Spark MLlib to Billions of Parameters: Spark Summit East talk by Yanbo Liang
- 1. Scaling Apache SparkMLlib to billions of parameters Yanbo Liang (Apache Spark committer @ Hortonworks) joint with Weichen Xu
- 2. Outline • Background • Vector-freeL-BFGS on Spark • Logistic regression on vector-free L-BFGS • Performance • Integrate with existing MLlib • Future work
- 3. Outline • Background • Vector-freeL-BFGS on Spark • Logistic regression on vector-free L-BFGS • Performance • Integrate with existing MLlib • Future work
- 4. Big data +big models = better accuracies
- 5. Spark: a unifiedplatform “Hidden Technical Debt in Machine Learning Systems”, Google
- 6. Spark: a unifiedplatform “Hidden Technical Debt in Machine Learning Systems”, Google
- 7.
- 8. MLlib logistic regression Initialize coefficients Broadcastcoefficients to Executors Comput loss and gradient for each instance, and sum them up locally Comput loss and gradient for each instance, and sum them up locally Comput loss and gradient for each instance, and sum them up locally Reduce from executors to get lossSum and gradientSum Handle regularization and use L-BFGS/OWL-QN to find next step Final model coefficients Executors/Workers Driver/Controller loop untial converge
- 9. Bottleneck 1 Initialize coefficients Broadcast coefficients toExecutors Comput loss and gradient for each instance, and sum them up locally Comput loss and gradient for each instance, and sum them up locally Comput loss and gradient for each instance, and sum them up locally Reduce from executors to get lossSum and gradientSum Handle regularization and use L-BFGS/OWL-QN to find next step Final model coefficients Executors/Workers Driver/Controller loop untial converge
- 10. Solution 1 Comput lossand gradient for each instance, and sum them up locally Comput loss and gradient for each instance, and sum them up locally Comput loss and gradient for each instance, and sum them up locally Reduce from executors to get final lossSum and gradientSum Comput loss and gradient for each instance, and sum them up locally Reduce from executors to get partial lossSum and gradientSum Reduce from executors to get partial lossSum and gradientSum
- 11. Bottleneck 2 Initialize coefficients Broadcast coefficients toExecutors Comput loss and gradient for each instance, and sum them up locally Comput loss and gradient for each instance, and sum them up locally Comput loss and gradient for each instance, and sum them up locally Reduce from executors to get lossSum and gradientSum Handle regularization and use L-BFGS/OWL-QN to find next step Final model coefficients Executors/Workers Driver/Controller loop untial converge
- 12. Outline • Background • Vector-freeL-BFGS on Spark • Logistic regression on vector-free L-BFGS • Performance • Integrate with existing MLlib • Future work
- 13. Distributed Vector Driver [2.5, 6.8,3.1, 9.0, 0.7, 1.9, 2.6, 8.7, 6.2, 1.1, 5.0, 0.0] worker0 worker1 worker2 worker3
- 14. Distributed Vector Driver [2.5, 6.8,3.1, 9.0, 0.7, 1.9, 2.6, 8.7, 6.2, 1.1, 5.0, 0.0] [2.5, 6.8, 3.1] worker0 [9.0, 0.7, 1.9] worker1 [2.6, 8.7, 6.2] worker2 [1.1, 5.0, 0.0] worker3
- 15. Distributed Vector • Definition: •Linear algebra operations: – a * x + y – a * x + b * y + c * z + … – x.dot(y) – x.norm – … class DistribuedVector( val values: RDD[Vector], val sizePerPart: Int, val numPartitions: Int, val size: Long)
- 16. L-BFGS Coefficients: X(k) Choose descentdirection (Two loop recursion) X(k+1) = alpha * dir(k) + X(k) Calculate G(k+1) and F(k+1)
- 17. L-BFGS Coefficients: X(k) Choose descentdirection (Two loop recursion) X(k+1) = alpha * dir(k) + X(k) Calculate G(k+1) and F(k+1) Worker Worker Worker Worker WorkerWorkerWorkerWorker Driver Space: (2m+1)d Time: 6md
- 18. Vector-free L-BFGS Coefficients: X(k) Choosedescent direction (Two loop recursion) X(k+1) = alpha * dir(k) + X(k) Calculate G(k+1) and F(k+1)
- 19. Vector-free L-BFGS Coefficients: X(k) Choosedescent direction (Two loop recursion) X(k+1) = alpha * dir(k) + X(k) Calculate G(k+1) and F(k+1) Worker Worker Worker Worker WorkerWorkerWorkerWorker Driver Space: (2m+1)*d Time: 6md Space: (2m+1)^2 Time: 8m^2
- 20. Outline • Background • Vector-freeL-BFGS on Spark • Logistic regression on vector-free L-BFGS • Performance • Integrate with existing MLlib • Future work
- 21.
- 22.
- 23.
- 24.
- 25.
- 26.
- 27.
- 28.
- 29.
- 30. grad0 grad1 grad2gradient: DistributedVector
- 31. Regularization and OWL-QN •Regularization: – L2 – L1 – elasticNet • Implement vector-free OWL-QN for objective with L1/elasticNet regularization.
- 32. Checkpoint Coefficients: X(k) Choose descentdirection (Two loop recursion) X(k+1) = alpha * dir(k) + X(k) Calculate G(k+1) and F(k+1) Worker Worker Worker Worker WorkerWorkerWorkerWorker Driver Space: (2m+1)*d Time: 6md Space: (2m+1)^2 Time: 8m^2
- 33. Outline • Background • Vector-freeL-BFGS on Spark • Logistic regression on vector-free L-BFGS • Performance • Integrate with existing MLlib • Future work
- 34. Performance(WIP) 0 20 40 60 80 100 120 1M 10M 50M100M 250M 500M 750M 1B Time for each epoch L-BFGS VL-BFGS
- 35. Outline • Background • Vector-freeL-BFGS on Spark • Logistic regression on vector-free L-BFGS • Performance • Integrate with existing MLlib • Future work
- 36. APIs val dataset: Dataset[_]= spark.read.format("libsvm").load("data/a9a") val trainer = new VLogisticRegression() .setColsPerBlock(100) .setRowsPerBlock(10) .setColPartitions(3) .setRowPartitions(3) .setRegParam(0.5) val model = trainer.fit(dataset) println(s"Vector-free logistic regression coefficients: ${model.coefficients}")
- 37. Adaptive logistic regression Numberof features Optimization method less than 4096 WLS/IRLS more than 4096, but less than 10 million L-BFGS more than 10 million VL-BFGS
- 38. Whole picture ofMLlib Application layer (Ads CTR, anti-fraud, data science, NLP, …) WLS/IRLS Optimization layer Spark core L-BFGS VL-BFGS LinearRegression Machine learning algorithm layer LogisticRegression MLP … SGD …
- 39. APIs val dataset: Dataset[_]= spark.read.format("libsvm").load("data/a9a") val trainer = new LogisticRegression() .setColsPerBlock(100) .setRowsPerBlock(10) .setColPartitions(3) .setRowPartitions(3) .setRegParam(0.5) .setSolver(“vl-bfgs”) val model = trainer.fit(dataset) println(s"Vector-free logistic regression coefficients: ${model.coefficients}")
- 40. Outline • Background • Vector-freeL-BFGS on Spark • Logistic regression on vector-free L-BFGS • Performance • Integrate with existing MLlib • Future work
- 41. Future work • Reducestages for each iteration. • Performance improvements. • Implement LinearRegression, SoftmaxRegression and MultilayerPerceptronClassifier base on vector-free L-BFGS/OWL-QN. • Real word use case for Ads CTR prediction with billions parameters and will share experiences and lessons we learned: – https://github.com/yanboliang/spark-vlbfgs
- 42. Key takeaways • Fulldistributed calculation, full distributed model, can run successfully without OOM. • VL-BFGS API is consistent with breeze L-BFGS. • VL-BFGS output exactly the same solution as breeze L-BFGS. • Does not require special components such as parameter servers. • Pure library and can be deployed and used easily. • Can benefit from the Spark cluster operation and development experience. • Use the same platform as the data size increasing.
- 43. Reference • https://papers.nips.cc/paper/5333-large-scale-l-bfgs-using- mapreduce • https://github.com/mengxr/spark-vl-bfgs •https://spark-summit.org/2015/events/large-scale-lasso-and-elastic- net-regularized-generalized-linear-models/
- 44.