Download as PDF, PPTX
![Feature Extraction
Solution? “Stringify” + CountVectorizer
Row(i1=u'1', i2=u'1', i3=u'5', i4=u'0', i5=u’1382’,... )
Row(raw=[u'i1=1', u'i2=1', u'i3=5', u'i4=0', u'i5=1382', ...)
Convert set of
String featuresinto
Seq[String]](https://image.slidesharecdn.com/4anickpentreath-161103184439/75/Spark-Summit-EU-talk-by-Nick-Pentreath-19-2048.jpg)
Uploaded bySpark Summit
Spark Summit EU talk by Nick Pentreath
This document summarizes a presentation about scaling factorization machines on Apache Spark using parameter servers. The presentation introduces factorization machines as a method for modeling feature interactions in large datasets. It then discusses how distributed factorization machine models can be implemented on Spark using a parameter server approach, as demonstrated by the GlintFM library. Performance results show that GlintFM achieves significant speedups over the existing spark-libFM implementation on a large Criteo dataset, scaling to billions of parameters. Some remaining challenges are also outlined, along with ideas for future work.
More Related Content
Similar to Spark Summit EU talk by Nick Pentreath
![[DSC Europe 25] Dusan Jovicic - AI Story: From on-prem to cloud and back agai...](https://cdn.slidesharecdn.com/ss_thumbnails/8kp49m6uq22ifnbwhfnk-2-251205085715-964d11a6-thumbnail.jpg?width=640&height=640&fit=bounds)
![[DSC Europe 25] Boris Perkovic - Lost in performance.pptx](https://cdn.slidesharecdn.com/ss_thumbnails/uq5hrp7vsuahqkxzifux-1-251204082258-fd2ee09d-thumbnail.jpg?width=640&height=640&fit=bounds)
![[DSC Europe 25] Jim Sterne - Adopting Generative AI Capabilities Into the Ent...](https://cdn.slidesharecdn.com/ss_thumbnails/sxhpofuorcagxsaulkmt-3-251204082258-7e66bc48-thumbnail.jpg?width=640&height=640&fit=bounds)
![[DSC Europe 25] Dragana Ilic - AI for Big Data in Astronomy.pptx](https://cdn.slidesharecdn.com/ss_thumbnails/8palya86qaatvjhva1ms-2-dragana-ilic-ai-ilic-251208151906-652b819c-thumbnail.jpg?width=640&height=640&fit=bounds)
Spark Summit EU talk by Nick Pentreath
- 1. SPARK SUMMIT EUROPE2016 SCALING FACTORIZATION MACHINESON APACHE SPARK WITH PARAMETER SERVERS Nick Pentreath Principal Engineer, IBM
- 2. About • About me –@MLnick – Principal Engineer at IBM working on machine learning & Spark – Apache Spark PMC – Author of Machine Learning with Spark
- 3. Agenda • Brief Introto Factorization Machines • Distributed FMs with Spark and Glint • Results • Challenges • Future Work
- 4.
- 5.
- 6. Factorization Machines Linear Models 𝑤"+ $ 𝑤% 𝑥% ' %() Not expressive enough! w0 Bias terms
- 7. Factorization Machines Polynomial Regression 𝑤"+ $ 𝑤% 𝑥% + $ $ 𝑤%* 𝑥% 𝑥* ' *(%+) ' %() ' %() w0 Bias terms Interaction term O(d2) n << d
- 8. Factorization Machines Factorization Machine w0 Biasterms Factorized interaction term 𝑤" + $ 𝑤% 𝑥% + $ $ 𝑣% 𝑣* 𝑥% 𝑥* ' *(%+) ' %() ' %() O(d2k)
- 9. Factorization Machines Factorization Machine O(d2k)O(dk) Math hack aka reformulation Not convex, but efficient to train using SGD, coordinate descent, MCMC
- 10. Factorization Machines Factorization Machine Standardmatrix factorization (with biases) Contextual featuresMetadata features Model size can still be very large! e.g. video sharing, online ads, social networks
- 11. DISTRIBUTED FM MODELSON SPARK
- 12. Linear Models onSpark • Data parallel Master Send model to workers Update model Send Update ... Final model Worker Compute partial gradient using latest full model Compute partial gradient using latest full model … Worker Compute partial gradient using latest full model Compute partial gradient using latest full model … Tree aggregate Broadcast Bottleneck!
- 13. Parameter Servers • Model& data parallel Master Schedule work ... Worker Compute partial gradient using latest partial model Compute partial gradient using latest partial model … Parameter Server Update model Update model … Pull partial model Push partial gradient Only push/pull required features
- 14. Distributed FMs • spark-libFM –Uses old MLlib GradientDescent and LBFGS interfaces • DiFacto – Async SGD implementation using parameter server (ps-lite) – Adagrad, L1 regularization, frequency-adaptive model-size • Key is that most real-world datasets are highly sparse (especially high-cardinality categorical data), e.g. online ads, social network, recommender systems • Workers only need access to a small piece of the model
- 15. GlintFM • Procedure: 1. ConstructGlint Client 2. Create distributed parameters 3. Pre-compute required feature indices (per partition) 4. Iterate: • Pull partial model (blocking) • Compute partial gradient & update • Push partial update to parameter servers (can be async) 5. Done! Glint is a parameter server built using Akka For more info, see Rolf’s talk at 17:15!
- 16.
- 17. Data Criteo Display AdvertisingChallenge Dataset • 45m examples, 34m unique features, 48 nnz /example 0 2 4 6 8 10 12 Millions Unique Values per Feature 0% 20% 40% 60% 80% 100% Feature Occurence (%)
- 18. Feature Extraction Raw DataStringIndexer OneHotEncoder VectorAssembler OOM!
- 19. Feature Extraction Solution? “Stringify”+ CountVectorizer Row(i1=u'1', i2=u'1', i3=u'5', i4=u'0', i5=u’1382’,... ) Row(raw=[u'i1=1', u'i2=1', u'i3=5', u'i4=0', u'i5=1382', ...) Convert set of String featuresinto Seq[String]
- 20. Feature Extraction Raw DataStringify Count Vectorizer
- 21. Feature Extraction Raw DataStringify HashingTF
- 22. Performance 0 500 1000 1500 2000 2500 3000 3500 4000 4500 k=0 k=6 k=16k=32 Total run time (s)* Mllib FM Glint FM N/A *10 iterations, 48 partitions, fit intercept Model size 1.9GB Model size 4.6GB 2GB limit
- 23. Performance Broadcast Read *k =6, 10 iterations, 48 partitions, fit intercept Compute Gradient computation Aggregation & collect
- 24. Performance - 20 40 60 80 100 120 140 160 MLLib FM GlintFM Median time per iteration (s) Other Compute Communication 0 500 1000 1500 2000 2500 3000 3500 4000 4500 MB / iteration Data Transfer Mllib FM Glint FM *k = 6, 10 iterations, 48 partitions, fit intercept
- 25.
- 26. Challenges • Tuning configuration –Glint - models/server, message size, Akka frame size – Spark - data partitioning (can be seen as “mini-batch” size) • Lack of server-side processing in Glint – For L1 regularization, adaptive sparsity, Adagrad – These result in better performance, faster execution • Backpressure / concurrency control
- 27. Challenges • Tuning models/ server 0 500 1000 1500 2000 2500 3000 6 12 24 Models / parameter server Total runtime *k = 16, 10 iterations, 48 partitions, fit intercept 0 50 100 150 200 250 300 6 12 24 Models / parameter server Median iteration time Max iteration time
- 28. Challenges • Index partitioningfor “hot features” 0 500 1000 1500 2000 Total run time (s)* CV features Hashed features – CountVectorizer for features leads to hot spots & straggler tasks due to sorting by occurrence – OneHotEncoder OOMed... but can also face this problem – Spreading out features is critical (used feature hashing) *k = 16, 10 iterations, 48 partitions, fit intercept
- 29. Future Work • Glintenhancements – Add features from DiFacto, i.e. L1 regularization, Adagrad & memory-adaptive k – Requires support for UDFs on the server – Built-in backpressure (Akka Artery / Streams?) – Key caching – 2x decrease in message size • Mini-batch SGD within partitions • Distributed solvers for ALS, MCMC, CD • Relational data / block structure formulation – www.vldb.org/pvldb/vol6/p337-rendle.pdf
- 30. References • Factorization Machines –http://www.csie.ntu.edu.tw/~b97053/paper/Rendle2010FM.pdf – https://github.com/ibayer/fastFM – www.libfm.org – https://github.com/zhengruifeng/spark-libFM – https://github.com/scikit-learn-contrib/polylearn • DiFacto – https://github.com/dmlc/difacto – www.cs.cmu.edu/~yuxiangw/docs/fm.pdf • Glint / parameter servers – https://github.com/rjagerman/glint – https://github.com/dmlc/ps-lite
- 31.