Continuous Delivery at Netflix:
From Code to the Monkeys

QCon – San Francisco, November 2013
Dianne Marsh

@dmarsh
http:/...
Watch the video with slide
synchronization on InfoQ.com!
http://www.infoq.com/presentations
/netflix-continuous-delivery

...
Presented at QCon San Francisco
www.qconsf.com
Purpose of QCon
- to empower software development by facilitating the sprea...
BUILD

BAKE

DEPLOY
Netflix Goals
High Availability
But … Move fast
Tools encourage Best Practices
But … Freedom to do the right thing
Teams Deploy Their Own Code
Run What You Wrote
§  Rapid Innovation
§  Rapid Detection
§  Rapid Response
= Freedom + Res...
BUILD

Jenkins Job DSL
Configuration as Code
Groovy Script
Scripts go in Version Control

http://www.slideshare.net/quidry...
BUILD

BAKE
BAKE

Aminator
•  Create AMI from Base AMI
•  Image contains service and
everything needed to run it
•  Unit of Deployment...
BUILD

BAKE

DEPLOY
DEPLOY

Asgard: AWS Deployment
Tool
Deploys Netflix to the Cloud
Red/Black push

http://www.infoq.com/presentations/asgard
CANARY ANALYSIS

Test, Int, Prod
Choose where to deploy
Run canary analysis
Scale up new instances
Turn on traffic to new ...
Asgard Developer Portal
GLISTEN

Extending Asgard’s Workflow
Automated Red/Black Push
Test, Int, Prod stacks
Run canary/analysis
Scale up new inst...
BUILD

BAKE

DEPLOY
Simian Army
• 
• 
• 
• 
• 

Chaos Monkey
Latency Monkey
Janitor Monkey
Conformity Monkey
(and more!)

Test resiliency at r...
One Button Deployment?
Regional Isolation
Limit Impact of Human
Error
§  Stagger deployments
§  Canary testing per
region
Multi-Region Consistency
Build Tooling to:
§  Schedule
Deployments
§  Prefer off peak
§  Choose next available
region a...
Send in the Conformity Monkey
Have deployments diverged?
§  Balance regional consistency
with regional isolation
§  Prov...
Clean up with Janitor Monkey
§  Disassociate unused EIPs
§  Delete unassociated Amazon
EBS volumes
§  Delete older Amaz...
Key Elements for Netflix
§  Value Self-service
§  Test Everywhere
§  Build Awareness of Multiple Regions
§  Avoid peak...
Put NetflixOSS to Work for You
Netflix Platform

AMINATOR
Glisten

** And 30+ more projects at http://netflix.github.io/
Keep the Conversation Going

Continuous Delivery
Open Space
Ballroom B/C (here!)
1:35-2:25, immediately
following lunch
Thanks!

We’re always hiring!
Dianne Marsh (@dmarsh)
dmarsh@netflix.com
Watch the video with slide synchronization on
InfoQ.com!
http://www.infoq.com/presentations/netflixcontinuous-delivery
Upcoming SlideShare
Loading in...5
×

From Code to Monkeys: Continuous Delivery at Netflix

718

Published on

Video and slides synchronized, mp3 and slide download available at URL http://bit.ly/JUqwqz.

Dianne Marsh presents the open source tools used by Netflix to keep the continuous delivery wheels spinning. Filmed at qconsf.com.

Dianne Marsh is a Director of Engineering for Netflix in Los Gatos, CA, where she leads a team responsible for tools and systems used by nearly all engineers in the company for continuous integration, delivery and deployment to the AWS cloud. Dianne earned her Master of Science degree in computer science from Michigan Technological University.

From Code to Monkeys: Continuous Delivery at Netflix

  1. 1. Continuous Delivery at Netflix: From Code to the Monkeys QCon – San Francisco, November 2013 Dianne Marsh @dmarsh http://www.linkedin.com/in/diannemarsh 1
  2. 2. Watch the video with slide synchronization on InfoQ.com! http://www.infoq.com/presentations /netflix-continuous-delivery InfoQ.com: News & Community Site • 750,000 unique visitors/month • Published in 4 languages (English, Chinese, Japanese and Brazilian Portuguese) • Post content from our QCon conferences • News 15-20 / week • Articles 3-4 / week • Presentations (videos) 12-15 / week • Interviews 2-3 / week • Books 1 / month
  3. 3. Presented at QCon San Francisco www.qconsf.com Purpose of QCon - to empower software development by facilitating the spread of knowledge and innovation Strategy - practitioner-driven conference designed for YOU: influencers of change and innovation in your teams - speakers and topics driving the evolution and innovation - connecting and catalyzing the influencers and innovators Highlights - attended by more than 12,000 delegates since 2007 - held in 9 cities worldwide
  4. 4. BUILD BAKE DEPLOY
  5. 5. Netflix Goals High Availability But … Move fast Tools encourage Best Practices But … Freedom to do the right thing
  6. 6. Teams Deploy Their Own Code Run What You Wrote §  Rapid Innovation §  Rapid Detection §  Rapid Response = Freedom + Responsibility http://www.slideshare.net/garethbowles/self-servicebuilddeploymentagile2013
  7. 7. BUILD Jenkins Job DSL Configuration as Code Groovy Script Scripts go in Version Control http://www.slideshare.net/quidryan/configuration-as-code
  8. 8. BUILD BAKE
  9. 9. BAKE Aminator •  Create AMI from Base AMI •  Image contains service and everything needed to run it •  Unit of Deployment for Test and Prod •  Abstracts Cloud Details http://techblog.netflix.com/2013/03/ami-creation-with-aminator.html
  10. 10. BUILD BAKE DEPLOY
  11. 11. DEPLOY Asgard: AWS Deployment Tool Deploys Netflix to the Cloud Red/Black push http://www.infoq.com/presentations/asgard
  12. 12. CANARY ANALYSIS Test, Int, Prod Choose where to deploy Run canary analysis Scale up new instances Turn on traffic to new ASG Turn off traffic to old ASG Wait … analyze … continue
  13. 13. Asgard Developer Portal
  14. 14. GLISTEN Extending Asgard’s Workflow Automated Red/Black Push Test, Int, Prod stacks Run canary/analysis Scale up new instances Turn on traffic to new ASG Run more tests Turn off traffic to old ASG Wait … analyze … continue http://techblog.netflix.com/2013/09/glisten-groovy-way-to-use-amazons.html
  15. 15. BUILD BAKE DEPLOY
  16. 16. Simian Army •  •  •  •  •  Chaos Monkey Latency Monkey Janitor Monkey Conformity Monkey (and more!) Test resiliency at runtime http://www.infoq.com/presentations/netflix-resiliency-failure-cloud
  17. 17. One Button Deployment?
  18. 18. Regional Isolation Limit Impact of Human Error §  Stagger deployments §  Canary testing per region
  19. 19. Multi-Region Consistency Build Tooling to: §  Schedule Deployments §  Prefer off peak §  Choose next available region automatically §  Provide high visibility per region
  20. 20. Send in the Conformity Monkey Have deployments diverged? §  Balance regional consistency with regional isolation §  Provide meaningful thresholds §  Build best practices into tooling and reporting http://techblog.netflix.com/2013/05/conformity-monkey-keeping-yourcloud.html
  21. 21. Clean up with Janitor Monkey §  Disassociate unused EIPs §  Delete unassociated Amazon EBS volumes §  Delete older Amazon EBS snapshots §  Leverage Amazon S3 Object Expiration https://github.com/Netflix/SimianArmy/wiki/Janitor-Home
  22. 22. Key Elements for Netflix §  Value Self-service §  Test Everywhere §  Build Awareness of Multiple Regions §  Avoid peak times §  Roll back quickly and easily §  Be Cloud Native
  23. 23. Put NetflixOSS to Work for You Netflix Platform AMINATOR Glisten ** And 30+ more projects at http://netflix.github.io/
  24. 24. Keep the Conversation Going Continuous Delivery Open Space Ballroom B/C (here!) 1:35-2:25, immediately following lunch
  25. 25. Thanks! We’re always hiring! Dianne Marsh (@dmarsh) dmarsh@netflix.com
  26. 26. Watch the video with slide synchronization on InfoQ.com! http://www.infoq.com/presentations/netflixcontinuous-delivery

×