Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Scaling DevOps of Microservices at Uber (Code Conf 2018)


Published on

Scaling DevOps at Uber - How Uber handles scaling of its microservices based platform including building, deploying, running, testing, and monitoring.

Published in: Software
  • Be the first to comment

  • Be the first to like this

Scaling DevOps of Microservices at Uber (Code Conf 2018)

  1. 1. Edit or delete footer text in Master ipsandella doloreium dem isciame ndaestia nessed quibus aut hiligenet ut ea debisci eturiate poresti vid min core, vercidigent. Scaling DevOps at Kiran Bondalapati, Uber Igniting opportunity by setting the world in motion
  2. 2. 10+ billion trips 15M+ trips per day 6 continents, 65 countries and 600+ cities 75M active monthly users 3M+ active drivers 16,000+ employees worldwide 3000+ developers worldwide
  3. 3. Bits + Atoms 0101010101010101010101 1010101010101011100111 0001110010101010101001
  4. 4. 5/20/17 - 5 Billion Trips 6/18/16 - 2 Billion Trips 12/31/15 - 1 Billion Trips 6/10/18 - 10 Billion Trips Business
  5. 5. 1000s of Microservices 1000s of builds per day 10000+ deployments per day 100K+ service containers per cluster ~1M batch containers per day DevOps
  7. 7. Pre-history PHP (outsourced) Marketplace Node.JS, moving to Go Core Services Python, moving to Go, Java Maps Python and Java Data Python and Java Metrics Go Code 20000+ repos Multiple languages and frameworks Multiple communication protocols
  8. 8. Microservices March 2016 March 2018
  9. 9. 4000+ builds per day Build times affect developer productivity Build sizes affect deployments Build Build without docker Optimize layer generation Distributed cache for intermediate layers
  10. 10. 100s of services pulling 1000s images from Registry Deploy Vertical Scaling Horizontal Scaling P2P Distribution - Scales with Load
  11. 11. Reproduce Halloween and New Year Systemic issues are hard in unit tests Cascading failures are common in real life Test Hailstorm load testing framework uDestroy random failure injection framework Regular failure and failover drills no testee … no workee
  12. 12. Containers are sized for peak load Dynamic utilization affects cluster efficiency Typical auto-scaling does not help Run Combine responsive and revocable tasks Oversubscribe resources Rate limiting of revocation
  13. 13. M3 metrics platform ~5B time series ~10M metrics/sec Changing services, metrics, infrastructure, ...Monitoring Rule based alert generators Git based review and update Measure oncall quality
  14. 14. HW/SW has tendency to have faults 100M+ alerts per month across Uber stack Many faults are transient/temporary Remediate Smart alert prioritization Automate manual tasks - reboot, restart, ... SLA aware remediation
  15. 15. Provision Deploy Config Scale Update Detect Recovery Remedy Auto-
  16. 16. Standards based innovation Layercake architecture Avoid cyclic dependencies Avoid cascading failures while designing Incremental deployments - code and config Test often … including production Add guardrails to automation Design for understandability Learnings
  17. 17. Larger systems Bigger impact of changeScale Larger teams Less each person knows Our understanding of systems breaks more often than actual systems do
  18. 18. Proprietary and confidential © 2018 Uber Technologies, Inc. All rights reserved. No part of this document may be reproduced or utilized in any form or by any means, electronic or mechanical, including photocopying, recording, or by any information storage or retrieval systems, without permission in writing from Uber. This document is intended only for the use of the individual or entity to whom it is addressed and contains information that is privileged, confidential or otherwise exempt from disclosure under applicable law. All recipients of this document are notified that the information contained herein includes proprietary and confidential information of Uber, and recipient may not make use of, disseminate, or in any way disclose this document or any of the enclosed information to any person other than employees of addressee to the extent necessary for consultations with authorized personnel of Uber. We are hiring!