Uploaded bySangeeta Narayanan
QConSF 2014 - How we learned to stop worrying and start deploying the Netflix API
This document discusses Netflix's approach to deploying its API service. It outlines how Netflix moved to a continuous delivery model with automated testing, deployment pipelines and canary analysis. This allowed Netflix to increase deployment velocity for its API server, releasing new features more frequently with rollbacks if needed. The key lessons are to build agility into architectures, embrace rather than fight change, and use insight from metrics to improve processes.
More Related Content
Similar to QConSF 2014 - How we learned to stop worrying and start deploying the Netflix API
![Agility and Control from AWS [FutureStack16]](https://cdn.slidesharecdn.com/ss_thumbnails/agilityandcontroldougleeaws-161201210211-thumbnail.jpg?width=640&height=640&fit=bounds)
QConSF 2014 - How we learned to stop worrying and start deploying the Netflix API
- 1. How we learnedto stop worrying and start deploying The Netflix API service Sangeeta Narayanan @sangeetan http://www.linkedin.com/in/sangeetanarayanan http://bit.ly/1wq2kkN
- 4. Global Streaming forMovies and TV Shows
- 5. High Quality OriginalContent
- 7. Over 50 MillionSubscribers Over 40 Countries
- 8. > 34% ofPeak Downstream Traffic in North America Over 2 billion streaming hours a month
- 10.
- 12. http://goo.gl/VhokZV Role ofAPI Enable rapid innovation Conduit for metadata between Devices and Services Implements business logic Scale with business growth Maintain resiliency
- 13. Going back intime… http://bit.ly/1yOWEjr
- 14. PM: When canI get my feature?
- 15. PM: When canI get my feature? Us: 2 -4 weeks
- 16. PM: When canI get my feature? Us: 2 -4 weeks - ish…
- 17. PM: When canI get my feature? Us: 2 -4 weeks - ish… IF all goes well…
- 18. 2 week releasecycle
- 19.
- 21. ! Stop beingthe bottleneck! http://bit.ly/1zmYbAy
- 22.
- 23. Heavy weight CodeManagement
- 24.
- 25.
- 26. Slow, unreliable tests Low coverage Manual on-device testing
- 27.
- 28.
- 29. Requirements for newsystem On-Demand, Rapid Feature Delivery Intuitive and painless Easy recovery from errors Insight and Communication Balance between Agility & Stability
- 30. Release Release Release Patch Patch Patch 2 week Releases + Ad-Hoc Patches http://bit.ly/1E6a9yn
- 31. MR MR IR IR IR IR 3 week Major Releases + Weekly Incremental Releases
- 32.
- 33.
- 34. Test Strategy Increasing confidence
- 35. Test Runtimes 60% No False Positives
- 36.
- 37. Automated Deployments InternalEnvironments Using Asgard API Connected to builds Driven from CI Server
- 38.
- 39. • Multiple deployments/day ! • Multiple internal environments ! • Multiple AWS regions http://bit.ly/13qrIfw
- 40. Team Cohesion ! • Shared ownership - no silos • Increased partner satisfaction • Greater productivity
- 41.
- 42. Faster, Better, Allthe way! Shorter Feedback Loop Increased Confidence Richer Insight & Communication
- 43. Build Bake Test Deploy Build Test Bake Deploy
- 44. Branching Strategy ! Modeled after github-flow ! Automated Pull Request Processing ! Automated Patch Branching
- 45. Single long-lived branch Always deploy-able Feature branches
- 46. More, Better, Faster& to Prod Shorter Feedback Loop Increased Confidence Richer Insight & Communication
- 47. Automated Canary Analysis Aggregate Health Score >1500 metrics Configurable Multiple regions ~1%$Traffic$ Old$Code$(Baseline)$ New$Code$(Canary)$
- 50. Developer Canaries (dynamicallyprovisioned)
- 51.
- 52. Deployed Not intendedfor deployment Not deployable; failed tests
- 53. Hands Free ProductionDeployments http://bit.ly/1wQ8fPQ
- 54.
- 55.
- 56. Production Traffic OldCode New Code
- 57. Production Traffic OldCode New Code
- 60.
- 61.
- 62. Old Code NewCode Re-enable traffic Production Traffic
- 63. Production Traffic OldCode New Code
- 64. More, Better, Faster& to Prod Shorter Feedback Loop Increased Confidence Richer Insight & Communication
- 69. API Server DeploymentVelocity #Deployments #Rollbacks Deployments & Rollbacks/week since Oct 2012
- 70. Take-aways Build Agilityinto Architecture Embrace Change; Don’t Fight it! Failure is inevitable Insight is key
- 71. What’s Next “Tiered”Canary Analysis Failure Injection Testing Throughput Trending http://goo.gl/zjiV6W
- 72.
- 73.
- 75.
- 76. How we learnedto stop worrying and start deploying The Netflix API service Sangeeta Narayanan @sangeetan http://www.linkedin.com/in/sangeetanarayanan http://bit.ly/1wq2kkN Thank You!