Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Upcoming SlideShare
What to Upload to SlideShare
Next
Download to read offline and view in fullscreen.

Share

Scaling and Unifying SciKit Learn and Apache Spark Pipelines

Download to read offline

Pipelines have become ubiquitous, as the need for stringing multiple functions to compose applications has gained adoption and popularity. Common pipeline abstractions such as “fit” and “transform” are even shared across divergent platforms such as Python Scikit-Learn and Apache Spark.

Scaling pipelines at the level of simple functions is desirable for many AI applications, however is not directly supported by Ray’s parallelism primitives. In this talk, Raghu will describe a pipeline abstraction that takes advantage of Ray’s compute model to efficiently scale arbitrarily complex pipeline workflows. He will demonstrate how this abstraction cleanly unifies pipeline workflows across multiple platforms such as Scikit-Learn and Spark, and achieves nearly optimal scale-out parallelism on pipelined computations.

Attendees will learn how pipelined workflows can be mapped to Ray’s compute model and how they can both unify and accelerate their pipelines with Ray.

  • Be the first to like this

Scaling and Unifying SciKit Learn and Apache Spark Pipelines

  1. 1. Scaling and Unifying Scikit Learn and Spark Pipelines using Ray Raghu Ganti Principal Research Staff Member IBM T J Watson Research Center Team (IBM & Red Hat): Michael Behrendt, Linsong Chu, Carlos Costa, Erik Erlandson, Mudhakar Srivatsa
  2. 2. So many pipelines… And many more…
  3. 3. Ray.IO § Can we do pipelines on Ray? § Can we scale popular AI/ML pipelines on Ray? § Can we unify scikit learn and Spark pipelines?
  4. 4. Current pipeline API • Focus on scikit learn and Spark pipelines • Scikit learn missing scaling; Spark focus on data parallel scaling Transform Fit X X y X’ Fitted model
  5. 5. Scaling Pipelines: I/O as List of Objects Transform Fit [X1, X2, … XN] [X1, X2, … XN] [y1, y2, … yN] [X1’, X2’, …, XN’] [FM1, FM2, … FMN]
  6. 6. Scaling Pipelines: AND/OR Graphs And node X1 X2 XN X1’ X2’ XM’ Or node X Step1 Step2 StepN X’ X’ X’
  7. 7. Key Features ▪ Python function as unit of compute ▪ Intuitive for data scientist ▪ Follows transformer APIs ▪ MPI-style scaling ▪ Object references as I/O for unit of compute ▪ Sharing of objects using Plasma store ▪ Enables zero-copy object sharing • List of objects as I/O • Function as unit of compute ▪ Scikit learn typically in Python ▪ Ray.IO with RayDP enables efficient data exchange • Cross environment ▪ Enriched DAGs from plain pipelines ▪ OR nodes for fan- out expressions ▪ AND nodes for arbitrary lambdas • AND/OR Graphs
  8. 8. Illustrative Example 8 Preprocess Random Forest Gradient Boost Decision Tree Sample Pipeline Scikit learn Pipeline Our Pipeline
  9. 9. Pipelines Galore… Airflow Kubeflow Scikit learn Spark Pipeline Our pipeline Task parallelism ✓ ✓ ✗ ✓ ✓ Data parallelism ✗ ✗ ✗ ✓ ✓ And/Or Graphs ✓ ✓ ✗ ✗ ✓ Computational unit Container Container Python function Python/Java function Python/Java function Mutability of DAG ✗ ✗ ✓ ✓ ✓
  10. 10. What to expect? • Execution strategies based on graph traversals • Early stopping criteria • Mutability of execution pipelines • Current status: Proposal discussion with Ray and OSS community
  11. 11. Q&A Contacts: Raghu Ganti (rganti@us.ibm.com) Michael Behrendt (michaelbehrendt@de.ibm.com) Linsong Chu (lchu@us.ibm.com) Carlos Costa (chcost@us.ibm.com) Erik Erlandson (eerlands@redhat.com) Mudhakar Srivatsa (msrivats@us.ibm.com)
  12. 12. Feedback Your feedback is important to us. Don’t forget to rate and review the sessions.

Pipelines have become ubiquitous, as the need for stringing multiple functions to compose applications has gained adoption and popularity. Common pipeline abstractions such as “fit” and “transform” are even shared across divergent platforms such as Python Scikit-Learn and Apache Spark. Scaling pipelines at the level of simple functions is desirable for many AI applications, however is not directly supported by Ray’s parallelism primitives. In this talk, Raghu will describe a pipeline abstraction that takes advantage of Ray’s compute model to efficiently scale arbitrarily complex pipeline workflows. He will demonstrate how this abstraction cleanly unifies pipeline workflows across multiple platforms such as Scikit-Learn and Spark, and achieves nearly optimal scale-out parallelism on pipelined computations. Attendees will learn how pipelined workflows can be mapped to Ray’s compute model and how they can both unify and accelerate their pipelines with Ray.

Views

Total views

266

On Slideshare

0

From embeds

0

Number of embeds

0

Actions

Downloads

11

Shares

0

Comments

0

Likes

0

×