Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Open Business Conference: Continuous Delivery At Netflix -- Powered by Open Source


Published on

How can we engineer velocity by weaving together tooling and culture with software
development to expose and elevate highly effective practices? This talk describes various
components of Netflix’s continuous delivery platform -- much of which is available in open
source. I’ll show how these pieces fit together and allow us to build scaffolding so that we’re comfortable with software developers making the decision to push the button for prod deployment -- and helps them to recover if necessary. As a result, we can run fast, trusting our tooling and our culture. I’ll also describe how we test our resiliency through simulating failure, unleashing the monkeys (Simian Army) on our production environment. Because if you’re afraid of cute little monkeys, imagine how afraid you’ll be of a production environment that offers those same risks but doesn’t give you an opportunity to test your response to those dangers.

Throughout this talk, I hope that you will challenge yourself to consider how your company can "shift the curve" through tooling and to achieve a high velocity environment without negatively impacting reliability.

Published in: Software, Technology
  • Be the first to comment

Open Business Conference: Continuous Delivery At Netflix -- Powered by Open Source

  1. 1. Continuous Delivery at Netflix: Powered by Open Source Dianne Marsh San Francisco, May 6, 2014
  2. 2. Building Continuous Delivery
  3. 3. Netflix Goals High Availability But … Move fast Tools encourage Best Practices But … Freedom to do the right thing
  4. 4. Teams Deploy Their Own Code Run What You Wrote • Rapid Innovation • Rapid Detection • Rapid Response = Freedom + Responsibility
  5. 5. Nebula/Gradle • Netflix Build Language • Gradle plugins • Supports internal and open source builds BUILD
  6. 6. BUILD Jenkins Job DSL Configuration as Code Groovy Script Scripts go in Version Control
  7. 7. BUILD BAKE
  8. 8. Aminator • Create AMI from Base AMI • Image contains service and everything needed to run it • Unit of Deployment for Test and Prod • Abstracts Cloud Details BAKE
  10. 10. DEPLOY Asgard: AWS Deployment Tool Deploys Netflix to the Cloud Red/Black push
  11. 11. CANARY ANALYSIS Test, Int, Prod Choose where to deploy Run canary analysis Scale up new instances Turn on traffic to new ASG Turn off traffic to old ASG Wait … analyze … continue
  12. 12. Asgard Developer Portal
  13. 13. GLISTEN Extending Asgard’s Workflow Automated Red/Black Push Run canary/analysis Scale up new instances Turn on traffic/Run Tests/Turn off traffic Wait … analyze … continue
  14. 14. One Button Deployment?
  15. 15. Regional Isolation Limit Impact of Human Error • Stagger deployments? • Canary testing per region? Know your service!
  16. 16. Multi-Region Consistency Build Tooling to: • Schedule Deployments • Prefer off peak • Choose next available region automatically • Provide high visibility per region
  18. 18. Simian Army • Chaos Monkey • Latency Monkey • Janitor Monkey • Conformity Monkey (and more!) Test resiliency at runtime
  19. 19. Send in the Conformity Monkey Have deployments diverged?  Balance regional consistency with regional isolation  Provide meaningful thresholds  Build best practices into tooling and reporting
  20. 20. Clean up with the Janitor Monkey • Disassociate unused EIPs • Delete unassociated Amazon EBS volumes • Delete older Amazon EBS snapshots • Leverage Amazon S3 Object Expiration
  21. 21. Key Elements for Netflix • Value Self-service • Test Everywhere • Build Awareness of Multiple Regions • Avoid peak times • Roll back quickly and easily • Be Cloud Native
  22. 22. Netflix OSS More at
  23. 23. Thanks! Dianne Marsh (@dmarsh) Thanks!