Skip to main content
UploadLanguage (EN)Support
BusinessMobileSocial MediaMarketingTechnologyArt & PhotosCareerDesignEducationPresentations & Public SpeakingGovernment & NonprofitHealthcareInternetLawLeadership & ManagementAutomotiveEngineeringSoftwareRecruiting & HRRetailSalesServicesScienceSmall Business & EntrepreneurshipFoodEnvironmentEconomy & FinanceData & AnalyticsInvestor RelationsSportsSpiritualNews & PoliticsTravelSelf ImprovementReal EstateEntertainment & HumorHealth & MedicineDevices & HardwareLifestyle
MK
Uploaded byM Waleed Kadous
PDF, PPTX592 views

Third Gen Production ML Architectures: Lessons from History, Experiences with Ray

Third generation machine learning architectures like Ray provide full programmability and flexibility, shifting the focus to libraries. Ray allows users to more easily build existing ML pipelines, parallelize high performance systems, make ML accessible to broader audiences, and undertake novel projects in reinforcement learning at large scale. It provides a simple way to distribute computation across resources without worrying about infrastructure details, analogous to how third generation GPU programming shifted the focus to libraries over hardware specifics. Customers are using Ray for applications like simplifying ML workflows at Uber and training self-driving boat designs for the America's Cup.

Engineering◦
Download as PDF, PPTX
Third Gen Production ML Architectures: Lessons from history, Experiences with Ray Waleed Kadous, Head of Engineering, Anyscale
Core Assertion - Production ML Architectures are evolving: - 1st Gen: “Fixed function” pipeline - 2nd Gen: Programmability within the pipeline - 3rd Gen: ? - Assertion - Learn from history - Evolution of GPU programming architectures a model? - → third generation is complete programmability - Introduce you to Ray - Are Ray and its siblings the next generation of production ML architecture?
Plan - Quickly review GPU programming history - Compare that with the evolution of production ML architectures - Discuss Ray in the context of a 3rd Generation system - Show how Ray’s users are building: - Existing applications in a new way. - New production ML applications that used to be difficult.
Production ML Architectures - ML systems that - Are deployed in production environments - Typically operate at considerable scale - Trained on large volume of data (minimally: gigabytes) - Do many inferences (minimally: hundreds of qps) - Require distributed training and frequently distributed inference
GPU Programming Architectures
1st Generation: Fixed Function Pipeline Input Transform & Lighting View & Clipping Fill in polygons (rasterize) Pixel operations (eg depth) Display Examples: OpenGL 1.0, Direct3D
Pros and Cons Pros - Amazing hardware accelerated graphics - Massively expanded possibilities - Enabled the 3D gaming revolution - Flexibility that came from choice of inputs (Textures, meshes, etc) Cons - Could not do complex effects - Lack of flexibility meant more advanced features out of reach
Second Generation: Programmability within Pipeline Programmable Vertex Shader Programmable Fragment Shader Examples: Direct3D 10, Open GL 2.0 Input Transform & Lighting View & Clipping Fill in polygons (rasterize) Pixel operations (eg depth) Display
Second generation: Programmability within Pipeline - Some programmability for different shaders - Had its own C-like languages (GLSL and HLSL) - But STILL had to basically conform to the existing pipeline - Especially interchange formats to each stage: - Scene in, vertices with properties out - Vertices with properties in, pixels (aka fragments) out - Starting to see applications that “hack” general applications - “GPGPU” movement - Turing complete so everything is possible - But “possible” and “easy” are not the same thing - Examples: Physics and first deep learning acceleration.
Fixed format interchange between stages Programmable Vertex Shader Programmable Fragment Shader Examples: Direct3D 10, Open GL 2.0 Input Transform & Lighting View & Clipping Fill in polygons (rasterize) Pixel operations (eg depth) Display Vertices, textures, etc Vertices with properties Pixels aka Fragments
Third generation: Complete programmability Programmable Vertex Shader Programmable Fragment Shader Examples: Cg, OpenCL Input Transform & Lighting View & Clipping Fill in polygons (rasterize) Pixel operations (eg depth) Display
Third Generation: Full Programmability - Can do everything First Gen and Second Gen do (they’re just libraries) - Interface is a programming language - Led to an explosion of applications - Deep Learning is built on this - cuDNN → Caffe → Torch → PyTorch! - Key change: focus shifts to libraries - Originally, games were written with Direct3D or OpenGL - Now people use an engine like Unity or Unreal Engine 4 - Focus is now on libraries, the GPU is just a detail now - Opened up the power of GPUs to a huge number of users - Nobody talks about OpenGL now. People talk about Unity, and it’s just a compute layer
And when I say focus has shifted …
OpenCL Example: Mandelbrot fractal in hardware
Wait, what does this have to do with ML (aside from Deep Learning)?
Surprising similarities - Will talk about Michelangelo (Uber’s ML system) - When I was working with Mike del Balso and Jeremy Hermann (founders of Tecton) at Uber as ML evangelist - But apply to your favorite framework
1st generation: Fixed Pipeline (2017 Blog Post)
Thoughts Input Transform & Lighting View & Clipping Fill in polygons (rasterize) Pixel operations (eg depth) Display
2nd generation: Michelangelo (blog post from 2019) “Michelangelo was initially launched with a monolithic architecture that managed tightly-coupled workflows … made adding support for new Spark transformers difficult and precluded serving of models trained outside of Michelangelo. [This motivated us to] develop a workflow and operator framework on top of common orchestration engines for the flexibility of composing custom servable pipeline models.”
Looks kinda similar … Programmable Vertex Shader Programmable Fragment Shader Input Transform & Lighting View & Clipping Fill in polygons (rasterize) Pixel operations (eg depth) Display
So … by analogy, what would 3rd gen look like? - Can do everything First Gen and Second Gen does (they’re just libraries) - Interface is a programming language - Could it lead to an explosion of applications? - Key change: focus shifts to libraries - Focus is now on libraries, the compute engine is just a detail now - Opens up the power of ML to a huge number of users - Nobody talks about p3.16xlarge EC2 instances any more, they just use a library
Ray is: 1. A simple and flexible framework for distributed computation a. Simple: simple annotation to make functions & classes distributable. b. Flexible: create new distributed function calls & instances. No batching required. 2. A cloud-provider independent compute launcher/autoscaler 3. An ecosystem of distributed computation libraries built with #1
What is Ray?
How does it “fit” with ML? 3rd generation: tackling the problem of production ML architectures by making it programmable Moves the focus to libraries instead of worried about distributed computation and underlying clusters (just like in 3rd generation of GPU architecture, the focus shifted to libraries like Unity etc).
How are customers using Ray?
Four different ways - Simpler way to build 1st gen/2nd gen pipelines - Parallelize high performance ML systems - Build ML applications that make ML accessible to non-specialists - Win the America’s Cup
As a simpler way to build 1st/2nd gen pipelines - Flexibility of a programming language to define pipelines - More easily allows for shared components (e.g. feature transformation during training vs real time) - “Out of the box” support for distributed ML
Uber: By leveraging Ray, we can combine the preprocessing, distributed training, and hyperparameter search all within a single job running a single training script.
Uber: We believe that Ray will continue to play an increasingly important role in bringing much needed common infrastructure and standardization to the production machine learning ecosystem, both within Uber and the industry at large.
To parallelize high performance ML systems - Robovision wanted to run vehicle detection using 5 stacked ML models, plus custom logic - Don’t all fit on one GPU or machine - Robovision tried vanilla Python implementation: 5 fps - Parallelized using Ray: 16 fps - Code is simple: one “actor” for each ML model, Ray schedules actors across multiple GPUs and machines - How would you do this without Ray? - Each ML model becomes a microservice?
Code is pretty easy to read …
Build apps that make ML accessible to non-specialists - Descartes Labs provides a geospatial data analysis platform - User defines a function/model to be applied to geospatial data. - Ray is used to apply that function to petabytes of data across thousands of cores
To undertake ML projects that don’t fit the ML Pipeline - Many important & practical ML projects don’t fit the ML pipeline. - Examples: - Reinforcement Learning: what’s train what’s test? Whole point is to mix. - Online Learning - Active/Semi-supervised learning - One example: QuantumBlack -- McKinsey
Winning America’s Cup - Built a “AI Sailor” using RL on RLLib which runs on Ray. Outperformed human sailors. - Built a “Boat Designer” using RL as well. - Used Ray to take both RL algorithms to test 10x as many designs - “Frameworks [like RLLib and Ray] can, for example, enable teams to run training jobs across dozens of systems using a single line of code, rather than having to program this capability from scratch.”
Conclusion - The history of GPU rendering architecture gives us hints to where production ML architectures are going. - 1st gen: fixed function pipeline - 2nd gen: programmability within the pipeline - 3rd gen: Full programmability. Led to massive expansion of applications - Ray is an example of a programmable, flexible production ML architecture - Leading to new and interesting production ML applications including: - Simplifying existing ML architectures - Parallelizing high performance ML systems - Making ML accessible to non-specialists - Highly scalable algorithms for deep reinforcement learning
More info ray.io: The Ray framework Ray Summit (Jun 22-24): anyscale.com/ray-summit Hear from Uber, McKinsey and others in greater detail mwk@anyscale.com: Please reach out if you have questions! Anyscale is hiring