Download as PDF, PPTX
Uploaded bySpark Summit
Spark Summit EU talk by Rolf Jagerman
This document discusses an asynchronous parameter server called Glint for Spark. It was created to address the problem of machine learning models exceeding the memory of a single machine. Glint distributes models over multiple machines and allows two operations - pulling and pushing model parameters. It was tested on topic modeling of a 27TB dataset using 1,000 topics, significantly outperforming MLLib in terms of quality, runtime, and scalability. Future work may include improved fault tolerance, custom aggregation functions, and implementing additional algorithms like deep learning.
More Related Content
Similar to Spark Summit EU talk by Rolf Jagerman
Spark Summit EU talk by Rolf Jagerman
- 1. An Asynchronous ParameterServer for Spark Rolf Jagerman University of Amsterdam
- 2.
- 3.
- 4.
- 5.
- 6.
- 7. Tourism Video gamesJavascript Biology hotel allianc write tag local hord docum protein car euro var hypothet holidai warcraft return gene area wow prop cytoplasm golf gold function prk wed warhamm subset locu Motivation Topic Modeling (LDA) Tourism Video games Javascript Biology hotel allianc write tag local hord docum protein car euro var hypothet holidai warcraft return gene area wow prop cytoplasm golf gold function prk wed warhamm subset locu Tourism Video games Javascript Biology hotel allianc write tag local hord docum protein car euro var hypothet holidai warcraft return gene area wow prop cytoplasm golf gold function prk wed warhamm subset locu
- 8. Problem Model size exceedsmemory of a single machine! Data ModelMachine Learning Algorithm
- 9. 9 Spark Worker Spark Driver SparkWorker Spark Worker Spark Worker
- 10. Spark Worker Spark Driver SparkWorker Spark Worker Spark Worker 10
- 11. Spark Worker Spark Driver SparkWorker Spark Worker Spark Worker 11
- 12. Spark Worker Spark Driver SparkWorker Spark Worker Spark Worker 12
- 13. Spark Worker Spark Driver SparkWorker Spark Worker Spark Worker 13
- 14. Spark Worker Spark Driver SparkWorker Spark Worker Spark Worker 14
- 15. Spark Worker Spark Driver SparkWorker Spark Worker Spark Worker 15
- 16. Parameter Server • Amachine learning framework • Distributes a model over multiple machines • Offers two operations: • Pull: query parts of the model • Push: update parts of the model 16
- 17. Parameter Server • Machinelearning update equation: wi ← wi + Δ • (Stochastic) gradient descent • Collapsed Gibbs sampling for topic modeling 17
- 18. Parameter Server • Machinelearning update equation: wi ← wi + Δ • (Stochastic) gradient descent • Collapsed Gibbs sampling for topic modeling • Aggregate push updates via addition (+) 18
- 19. 19 Spark Worker Spark Driver SparkWorker Spark Worker Spark Worker Parameter Server Parameter Server Parameter Server
- 20. 20 Spark Worker Spark Driver SparkWorker Spark Worker Spark Worker Parameter Server Parameter Server Parameter Server
- 21. 21 Spark Worker Spark Driver SparkWorker Spark Worker Spark Worker Parameter Server Parameter Server Parameter Server
- 22. 22 Spark Worker Spark Driver SparkWorker Spark Worker Spark Worker Parameter Server Parameter Server Parameter Server
- 23. 23 Spark Worker Spark Driver SparkWorker Spark Worker Spark Worker Parameter Server Parameter Server Parameter Server
- 24. Spark Worker Spark Driver SparkWorker Spark Worker Spark Worker Parameter Server Parameter Server Parameter Server 24
- 25. Spark Worker Spark Driver SparkWorker Spark Worker Spark Worker Parameter Server Parameter Server Parameter Server 25
- 26.
- 27. Experiments Task • LDA topicmodeling • 1,000 topics • 27TB data set (ClueWeb12) 27
- 28. Experiments Setup • 30 Sparkworkers (16 CPU cores each) • 3.7TB RAM total • Interconnected over 10Gb/s ethernet 28
- 29. Experiments Approach • Glint formodel storage (billions of parameters) • LDA approximation algorithm for runtime improvements • A small loss in model quality is acceptable 29
- 30. Experiments Comparing three methods: 1.Glint LDA 2. MLLib EM LDA 3. MLLib Online LDA 30
- 31. Experiments 31 Glint MLLib EMMLLib Online 6,108 -1.3% +0.4% 5,731 -5.4% -4.6% 5,427 -10.7% -8.2% 6,021 -5.1% +4.0% 5,813 +1.2% -1.3% 5,520 +4.8% -5.5% 5,861 +6.8% -0.5% Data (GB) # Topics 50 20 100 20 150 20 200 20 200 40 200 60 200 80 Perplexity (topic model quality)
- 32. Experiments 32 Glint MLLib EMMLLib Online 6.3 9.7 16.3 7.1 14.2 17.8 8.9 14.1 19.6 10.8 22.3 21.5 11.9 23.7 57.5 13.4 32.4 131.0 14.7 34.4 233.2 Data (GB) # Topics 50 20 100 20 150 20 200 20 200 40 200 60 200 80 Runtime (minutes)
- 33. Experiments 33 Glint MLLib EMMLLib Online 3.3 4.6 5.5 6.2 12.1 18.0 23.9 Data (GB) # Topics 50 20 100 20 150 20 200 20 200 40 200 60 200 80 Shuffle Write (GB) No Shuffle Write No Shuffle Write
- 34. • MLLib couldnot scale beyond 200GB or 100 topics due to task and job failures • Glint can compute a topic model on the full 27TB with 1,000 topics Experiments 34
- 35. Experiments 0 20 4060 Time (hours) Perplexity 104204304404504 35
- 36. Conclusion • Glint isa parameter server for Spark • Machine learning for very large models • Asynchrony enables highly flexible threading • Extremely easy to use • Outperforms MLLib on LDA topic modeling 36
- 37. Future work • Betterfault tolerance (using Chord/DHT) • User defined functions for aggregation • Support for sparse models • Implementing other algorithms • Deep learning • Linear models 37
- 38.