Download as PDF, PPTX
Uploaded byKarthik Murugesan
Uber - Building Intelligent Applications, Experimental ML with Uber’s Data Science Workbench
This document summarizes Uber's data science workbench (DSW), which provides scalable infrastructure, tools, customization, and support for Uber's large data science community. The DSW allows data scientists to access internal data sources and compute engines through Jupyter notebooks or RStudio IDEs in a secure, hosted environment. It helps standardize workflows and facilitates collaboration, publishing of results, and model deployment to production. The DSW integrates with Uber's Spark and machine learning systems to enable large-scale data exploration, parallelized model training, and evaluation at Uber's massive scale. It has supported a wide range of use cases across safety, risk, recommendations, and operations.
More Related Content
Similar to Uber - Building Intelligent Applications, Experimental ML with Uber’s Data Science Workbench
Uber - Building Intelligent Applications, Experimental ML with Uber’s Data Science Workbench
- 1. Building Intelligent Applications& Experimental ML with Uber’s Data Science Workbench Felix Cheung & Atul Gupte Uber Technologies, Inc.
- 2. / Data atUber / Analytics Stack / Spark at Uber / Machine Learning at Uber / Data Science Workbench / Common User Flows & Impact Contents
- 3. Engineer turned ProductManager Previously: building FarmVille & the mobile advertising platform @ Zynga Currently: Product Manager for Data Science Workbench & Data Warehouse / About Atul
- 4. Apache Spark PMC& Committer Engineer, Tech Lead & Area Owner of Spark @ Uber / About Felix
- 5. / Data atUber
- 6. Uber's mission isto bring reliable transportation - to everyone, everywhere
- 7. Data informs everydecision at the company
- 8. Uber’s massive dataholds deep, hidden insights. We surface them
- 9. 6,000+ data scientists,engineers, and operations managers rely on us to support the business
- 10. Data is whatdifferentiates Uber but, data at Uber is unlike anywhere else.
- 11. Delicate marketplace with networkeffects Bits to atoms Business New LOBs spun up in a snap Pluggable mobility platform Spatio-temporal Analytics Sheer scale Real-time. Real-world. ML is Uber’s brain Apps/Machine generated queries Varied skills: BI to DNN Consumers Internal and external 6,000 and growing What makes Uber unique
- 12. MISSION Move the worldwith global data, local insights, and intelligent decisions. Data Platform Team
- 13. / Data AnalyticsStack
- 14. The Data Team Ingest Workflow Management Store ProduceModel Ad-Hoc & Streaming Analytics Business Intelligence Machine Learning Metadata/ Knowledge Experimentation/ Segmentation Visualization Data Infrastructure Data Platforms Data Services & Analytics Disperse
- 15. Kafka Schemaless SOA BI Apps Ad-hocExperimentationML Notebooks Cluster Management All-Active Observability Security Raw Data Raw Tables Hadoop Hive Presto Spark Modeled Tables Vertica Vertica Warehouse AthenaX Apollo Streaming Real-time Metadata/Workflow Management Data Infrastructure
- 16. / Spark AtUber
- 17. at Uber Scale 100,000+ Sparkjobs per day ~96% ETL pipelines ~98% YARN job resource use (in vcore-seconds) on Spark ● 11,000+ machines across multiple data-centers ● Many 10s-petabytes of data ● Runs on one of the largest production HDFS clusters
- 18. Introducing Uber’s SparkCompute Service Simplifies lives of developers & cluster operators Consolidate Infrastructure Investments YARN, Mesos Available across multiple data-centers Improve Developer Experience Standardized Spark builds across Uber Bring-your-own-stack (optional) Advanced monitoring & debugging Serve Multiple Use Cases Exploratory, bursty & scheduled batch Manage full Spark application lifecycle Proliferate Better language support (R/Python/Java) Consumption Interfaces (CLI/REST/GUI)
- 19. Session Recap (June5th) Karthikeyan Natarajan Senior Software Engineer Bo Yang Senior Software Engineer
- 20. / Machine LearningAt Uber
- 21. The hype ● Abilityof a machine to learn without being explicitly programmed ● Identify hidden patterns in the world based on current and historical data and use it to predict the future ● Ability of a machine to get better at a task with data and experience ● Learn from mistakes and improve when given newer/more information
- 22. Demand prediction Object detection/tracking Motionprediction Route planning Pick-up clustering Voice recognition Supply modeling Occupancy modeling Route planning, ETA, road modeling, low-latency image classifier Elasticity estimation, ETA, route optimization, demand prediction Speech generation, Natural language generations, image classifiers, drop-off clustering
- 23. 2. prototype 3. productionize 1.define 4. measure Launch and Iterate Typical ML Workflow
- 24. UNDERSTAND BUSINESS NEED(S) DEFINE MINIMUM VIABLEPRODUCT (MVP) ○ Customers + cross-functional team ○ Define objectives and key results ○ Data-driven ○ Research ○ Ruthless prioritization 2. prototype 3. productionize 4. measure 1. define Problem Definition
- 25. UNDERSTAND BUSINESS NEED(S) DEFINE MINIMUM VIABLEPRODUCT (MVP) 2. prototype 1. define GET DATA DATA PREPARATION TRAIN MODELS EVALUATE MODELS 3. productionize 4. measure validation computational cost interpretability SQL, Spark data cleansing and pre-processing, R / Python CPU or GPU Exploration
- 26. UNDERSTAND BUSINESS NEED(S) 2. prototype 1.define DATA PREPARATION TRAIN MODELS EVALUATE MODELS 4. measure GET DATA PRODUCTIONIZE MODELS 3. productionize DEPLOY MODELS Engineers + Data Scientists, Java or Go, unit tests MAKE PREDICTIONSReal-time or batch Experimentation and rollout monitoring; Retraining strategy DEFINE MINIMUM VIABLE PRODUCT (MVP) Production
- 27. UNDERSTAND BUSINESS NEED(S) DEFINE MINIMUM VIABLEPRODUCT (MVP) 2. prototype 1. define DATA PREPARATION TRAIN MODELS EVALUATE MODELS GET DATA DEPLOY MODELS PRODUCTIONIZE MODELS MONITOR PREDICTIONS 4. measure MAKE PREDICTIONS 3. productionize Automatically detect degradations GATHER AND ANALYZE INSIGHTS Deep-dive analyses inform future product roadmap Measure
- 28. 3x growth inData Science community Py and R Machine Learning was mostly DIY - and on laptops Moving a Py models to production was hard Proliferation of tools, libraries, infra None of which could scale to 1000s Collaboration and Sharing non-existent Security / Compliance / DC redundancy Our world in 2016
- 29.
- 30. Unleash the productivityof the Data Science community at Uber by providing scalable infrastructure, tools, customization, and support. Mission
- 32. Fully hosted environment- nothing to install One-click to Jupyter Notebook or RStudio IDE Pre-baked environment Session Customization (BYOP) Wired to all internal sources and compute engines Our world today Share/publish/comment on data/notebooks One-click publish to Shiny dashboards Multi-DC Secure and GDPR Compliant Support & documentation
- 33.
- 34. RStudio and Shinyare trademarks of RStudio, Inc "Jupyter" is a trademark of the NumFOCUS foundation, of which Project Jupyter is a part. "Python" is a registered trademark of the PSF. The Python logos (in several variants) are use trademarks of the PSF as well.
- 35. RStudio and Shinyare trademarks of RStudio, Inc
- 39.
- 40. DSW + SparkArchitecture Storage Service DataScientists FrontEnd Application Management DSW DSW cluster ContainerContainer Container RStudio Server Container Jupyter Server Compute Service Hadoop Cluster Hive Presto Spark HDFS SparkMagic Livy
- 41. DSW + SparkUse-cases ● Explore large-scale dataset ● Parallelise Python native packages for feature generation & model training ● Collaborate and review on a common interface for ad-hoc analysis & prototyping
- 42. Common DS Patterns(#1) PySpark Python Native packages PySpark Hive Tables Hive Tables scikit-learn Features DSW
- 43. Common DS Patterns(#2) Spark Scala mllib Hive Tables HDFS Trained Model Production DSW Evaluate
- 44. DSW + SparkImpact Safety Trip classification Risk Driver account check Driver referral risk scoring Uber Eats Restaurant recommendations Support NLP model for support tickets Operations Lifetime value (LTV) model more!
- 45. / … onelast thing
- 46. We’re hiring! Excited tobuild the data platform that moves the world? Come join us! http://t.uber.com/datahire San Francisco, Palo Alto, Seattle, Bangalore
- 47. Proprietary and confidential© 2018 Uber Technologies, Inc. All rights reserved. No part of this document may be reproduced or utilized in any form or by any means, electronic or mechanical, including photocopying, recording, or by any information storage or retrieval systems, without permission in writing from Uber. This document is intended only for the use of the individual or entity to whom it is addressed and contains information that is privileged, confidential or otherwise exempt from disclosure under applicable law. All recipients of this document are notified that the information contained herein includes proprietary and confidential information of Uber, and recipient may not make use of, disseminate, or in any way disclose this document or any of the enclosed information to any person other than employees of addressee to the extent necessary for consultations with authorized personnel of Uber. Questions? Thank you! and remember, t.uber.com/datahire