SF Big Analytics 2019-06-12: Managing uber's data workflows at scale
Talk 2. Managing Uber’s Data workflow at Scale. Uber microservices serving millions of rides a day, leading to 100+ PB of data. To democratize data pipelines, Uber needed a central tool that provides a way to author, manage, schedule, and deploy data workflows at scale. This talk details Uber’s journey toward a unified and scalable data workflow system used to manage this data and shares the challenges faced and how the company has rearchitected several components of the system—such as scheduling and serialization—to make them highly available and more scalable. Speaker Alex Kira (Uber) Alex Kira is an engineering tech lead at Uber, where he works on the data workflow management team. His team provides a data infrastructure platform. In 19-year, he’s had experience across several software disciplines, including distributed systems, data infrastructure, and full stack development.
Recommended
Recommended
More Related Content
Similar to SF Big Analytics 2019-06-12: Managing uber's data workflows at scale
Similar to SF Big Analytics 2019-06-12: Managing uber's data workflows at scale (20)
More from Chester Chen
More from Chester Chen (20)
Recently uploaded
Recently uploaded (20)
SF Big Analytics 2019-06-12: Managing uber's data workflows at scale
- 8. 01
- 16. Apache Kafka Schemaless SOA BI Apps NotebooksExperimentation ML Dashboards Raw Data Raw Tables Hadoop Apache Hive Presto Apache Spark Modeled Tables Vertica Vertica Warehouse AthenaX Apollo Streaming Real-time Metadata/Workflow Management
- 19. 02
- 20. Cron The Apache Ooozie and Apache Airflow logos are either a registered trademark or trademark of the Apache Software Foundation in the United States and/or other countries. No endorsement by The Apache Software Foundation is implied by the use of these logos. The Jenkins logo is released under the Creative Commons Attribution-ShareAlike 3.0 unported license and is available at: https://jenkins.io/. The Clojure logo is in the public domain and was designed by Tom Hickey.
- 40. ● ● ● ● ●
- 44. 04
- 52. 05
- 54. ● ● ● ● ●
- 63. 06
- 82. 07
- 87. Pawan Dixit Alex Kira Ankit Mody Atasi Panda Prakhar Garg Patrick Cullen Anthony Asta
- 89. Proprietary and confidential © 2019 Uber Technologies, Inc. All rights reserved. No part of this document may be reproduced or utilized in any form or by any means, electronic or mechanical, including photocopying, recording, or by any information storage or retrieval systems, without permission in writing from Uber. This document is intended only for the use of the individual or entity to whom it is addressed and contains information that is privileged, confidential or otherwise exempt from disclosure under applicable law. All recipients of this document are notified that the information contained herein includes proprietary and confidential information of Uber, and recipient may not make use of, disseminate, or in any way disclose this document or any of the enclosed information to any person other than employees of addressee to the extent necessary for consultations with authorized personnel of Uber.