Scaling and Unifying SciKit Learn and Apache Spark Pipelines
![Scaling Pipelines: I/O as List of Objects
Transform
Fit
[X1, X2, … XN]
[X1, X2, … XN]
[y1, y2, … yN]
[X1’, X2’, …, XN’]
[FM1, FM2, … FMN]](https://image.slidesharecdn.com/303raghuganti-210616155323/75/scaling-and-unifying-scikit-learn-and-apache-spark-pipelines-5-2048.jpg?cb=1667327115)
Scaling and Unifying SciKit Learn and Apache Spark Pipelines
Pipelines have become ubiquitous, as the need for stringing multiple functions to compose applications has gained adoption and popularity. Common pipeline abstractions such as “fit” and “transform” are even shared across divergent platforms such as Python Scikit-Learn and Apache Spark. Scaling pipelines at the level of simple functions is desirable for many AI applications, however is not directly supported by Ray’s parallelism primitives. In this talk, Raghu will describe a pipeline abstraction that takes advantage of Ray’s compute model to efficiently scale arbitrarily complex pipeline workflows. He will demonstrate how this abstraction cleanly unifies pipeline workflows across multiple platforms such as Scikit-Learn and Spark, and achieves nearly optimal scale-out parallelism on pipelined computations. Attendees will learn how pipelined workflows can be mapped to Ray’s compute model and how they can both unify and accelerate their pipelines with Ray.
Recommended
More Related Content
What's hot
What's hot(20)
Similar to Scaling and Unifying SciKit Learn and Apache Spark Pipelines
Similar to Scaling and Unifying SciKit Learn and Apache Spark Pipelines(20)
More from Databricks
More from Databricks(20)
Recently uploaded
![[DSC Europe 23] Aleksandar Tomcic - Adversarial Attacks](https://cdn.slidesharecdn.com/ss_thumbnails/aleksandartomcic-adversarialattacks-231128234241-ab5a6f11-thumbnail.jpg?width=384&fit=bounds)
![[DSC Europe 23][Cryptica] Martin_Summer_Digital_central_bank_money_Ideas_init...](https://cdn.slidesharecdn.com/ss_thumbnails/stkmqtaswwueondfjdiq-martin-summer-digital-central-bank-money-ideas-initiatives-and-issues-231130103304-3a90228a-thumbnail.jpg?width=384&fit=bounds)
Recently uploaded(20)
![[DSC Europe 23] Aleksandar Tomcic - Adversarial Attacks](https://cdn.slidesharecdn.com/ss_thumbnails/aleksandartomcic-adversarialattacks-231128234241-ab5a6f11-thumbnail.jpg?width=768&fit=bounds)
![[DSC Europe 23][Cryptica] Martin_Summer_Digital_central_bank_money_Ideas_init...](https://cdn.slidesharecdn.com/ss_thumbnails/stkmqtaswwueondfjdiq-martin-summer-digital-central-bank-money-ideas-initiatives-and-issues-231130103304-3a90228a-thumbnail.jpg?width=768&fit=bounds)
![[DSC Europe 23] Ivana Sesic - Use of AI in Public Health.pptx](https://cdn.slidesharecdn.com/ss_thumbnails/dsceurope23ivanasesic-useofaiinpublichealth-231128224848-bc49111d-thumbnail.jpg?width=768&fit=bounds)
![[DSC Europe 23] Ales Gros - Quantum and Today s security with Quantum.pdf](https://cdn.slidesharecdn.com/ss_thumbnails/dsceurope23alesgros-quantumandtodayssecuritywithquantum-231128224122-311adb0d-thumbnail.jpg?width=768&fit=bounds)
![[DSC Europe 23] Stefan Mrsic_Goran Savic - Evolving Technology Excellence.pptx](https://cdn.slidesharecdn.com/ss_thumbnails/pgsjzcgsfwagzsywivwo-stefan-mrsic-goran-savic-evolving-technology-excellence-231129142112-da067636-thumbnail.jpg?width=768&fit=bounds)
![[DSC Europe 23] Milos Grubjesic Empowering Business with Pepsico s Advanced M...](https://cdn.slidesharecdn.com/ss_thumbnails/dsceurope23milosgrubjesicempoweringbusinesswithpepsicosadvancedmlplatform3-231128225740-80a4b074-thumbnail.jpg?width=768&fit=bounds)
Scaling and Unifying SciKit Learn and Apache Spark Pipelines
- 1. Scaling and Unifying Scikit Learn and Spark Pipelines using Ray Raghu Ganti Principal Research Staff Member IBM T J Watson Research Center Team (IBM & Red Hat): Michael Behrendt, Linsong Chu, Carlos Costa, Erik Erlandson, Mudhakar Srivatsa
- 2. So many pipelines… And many more…
- 3. Ray.IO § Can we do pipelines on Ray? § Can we scale popular AI/ML pipelines on Ray? § Can we unify scikit learn and Spark pipelines?
- 4. Current pipeline API • Focus on scikit learn and Spark pipelines • Scikit learn missing scaling; Spark focus on data parallel scaling Transform Fit X X y X’ Fitted model
- 5. Scaling Pipelines: I/O as List of Objects Transform Fit [X1, X2, … XN] [X1, X2, … XN] [y1, y2, … yN] [X1’, X2’, …, XN’] [FM1, FM2, … FMN]
- 6. Scaling Pipelines: AND/OR Graphs And node X1 X2 XN X1’ X2’ XM’ Or node X Step1 Step2 StepN X’ X’ X’
- 7. Key Features ▪ Python function as unit of compute ▪ Intuitive for data scientist ▪ Follows transformer APIs ▪ MPI-style scaling ▪ Object references as I/O for unit of compute ▪ Sharing of objects using Plasma store ▪ Enables zero-copy object sharing • List of objects as I/O • Function as unit of compute ▪ Scikit learn typically in Python ▪ Ray.IO with RayDP enables efficient data exchange • Cross environment ▪ Enriched DAGs from plain pipelines ▪ OR nodes for fan- out expressions ▪ AND nodes for arbitrary lambdas • AND/OR Graphs
- 8. Illustrative Example 8 Preprocess Random Forest Gradient Boost Decision Tree Sample Pipeline Scikit learn Pipeline Our Pipeline
- 9. Pipelines Galore… Airflow Kubeflow Scikit learn Spark Pipeline Our pipeline Task parallelism ✓ ✓ ✗ ✓ ✓ Data parallelism ✗ ✗ ✗ ✓ ✓ And/Or Graphs ✓ ✓ ✗ ✗ ✓ Computational unit Container Container Python function Python/Java function Python/Java function Mutability of DAG ✗ ✗ ✓ ✓ ✓
- 10. What to expect? • Execution strategies based on graph traversals • Early stopping criteria • Mutability of execution pipelines • Current status: Proposal discussion with Ray and OSS community
- 11. Q&A Contacts: Raghu Ganti (rganti@us.ibm.com) Michael Behrendt (michaelbehrendt@de.ibm.com) Linsong Chu (lchu@us.ibm.com) Carlos Costa (chcost@us.ibm.com) Erik Erlandson (eerlands@redhat.com) Mudhakar Srivatsa (msrivats@us.ibm.com)
- 12. Feedback Your feedback is important to us. Don’t forget to rate and review the sessions.