1.
Continuous Delivery at Netflix:
From Code to the Monkeys
QCon – San Francisco, November 2013
Dianne Marsh
@dmarsh
http://www.linkedin.com/in/diannemarsh
1

2.
Watch the video with slide
synchronization on InfoQ.com!
http://www.infoq.com/presentations
/netflix-continuous-delivery
InfoQ.com: News & Community Site
• 750,000 unique visitors/month
• Published in 4 languages (English, Chinese, Japanese and Brazilian
Portuguese)
• Post content from our QCon conferences
• News 15-20 / week
• Articles 3-4 / week
• Presentations (videos) 12-15 / week
• Interviews 2-3 / week
• Books 1 / month

3.
Presented at QCon San Francisco
www.qconsf.com
Purpose of QCon
- to empower software development by facilitating the spread of
knowledge and innovation
Strategy
- practitioner-driven conference designed for YOU: influencers of
change and innovation in your teams
- speakers and topics driving the evolution and innovation
- connecting and catalyzing the influencers and innovators
Highlights
- attended by more than 12,000 delegates since 2007
- held in 9 cities worldwide

4.
BUILD
BAKE
DEPLOY

5.
Netflix Goals
High Availability
But … Move fast
Tools encourage Best Practices
But … Freedom to do the right thing

6.
Teams Deploy Their Own Code
Run What You Wrote
§ Rapid Innovation
§ Rapid Detection
§ Rapid Response
= Freedom + Responsibility
http://www.slideshare.net/garethbowles/self-servicebuilddeploymentagile2013

7.
BUILD
Jenkins Job DSL
Configuration as Code
Groovy Script
Scripts go in Version Control
http://www.slideshare.net/quidryan/configuration-as-code

8.
BUILD
BAKE

9.
BAKE
Aminator
• Create AMI from Base AMI
• Image contains service and
everything needed to run it
• Unit of Deployment for Test and
Prod
• Abstracts Cloud Details
http://techblog.netflix.com/2013/03/ami-creation-with-aminator.html

10.
BUILD
BAKE
DEPLOY

11.
DEPLOY
Asgard: AWS Deployment
Tool
Deploys Netflix to the Cloud
Red/Black push
http://www.infoq.com/presentations/asgard

12.
CANARY ANALYSIS
Test, Int, Prod
Choose where to deploy
Run canary analysis
Scale up new instances
Turn on traffic to new ASG
Turn off traffic to old ASG
Wait … analyze … continue

13.
Asgard Developer Portal

14.
GLISTEN
Extending Asgard’s Workflow
Automated Red/Black Push
Test, Int, Prod stacks
Run canary/analysis
Scale up new instances
Turn on traffic to new ASG
Run more tests
Turn off traffic to old ASG
Wait … analyze … continue
http://techblog.netflix.com/2013/09/glisten-groovy-way-to-use-amazons.html

15.
BUILD
BAKE
DEPLOY

16.
Simian Army
•
•
•
•
•
Chaos Monkey
Latency Monkey
Janitor Monkey
Conformity Monkey
(and more!)
Test resiliency at runtime
http://www.infoq.com/presentations/netflix-resiliency-failure-cloud

17.
One Button Deployment?

18.
Regional Isolation
Limit Impact of Human
Error
§ Stagger deployments
§ Canary testing per
region

19.
Multi-Region Consistency
Build Tooling to:
§ Schedule
Deployments
§ Prefer off peak
§ Choose next available
region automatically
§ Provide high visibility
per region

20.
Send in the Conformity Monkey
Have deployments diverged?
§ Balance regional consistency
with regional isolation
§ Provide meaningful
thresholds
§ Build best practices into
tooling and reporting
http://techblog.netflix.com/2013/05/conformity-monkey-keeping-yourcloud.html

21.
Clean up with Janitor Monkey
§ Disassociate unused EIPs
§ Delete unassociated Amazon
EBS volumes
§ Delete older Amazon EBS
snapshots
§ Leverage Amazon S3 Object
Expiration
https://github.com/Netflix/SimianArmy/wiki/Janitor-Home

22.
Key Elements for Netflix
§ Value Self-service
§ Test Everywhere
§ Build Awareness of Multiple Regions
§ Avoid peak times
§ Roll back quickly and easily
§ Be Cloud Native

23.
Put NetflixOSS to Work for You
Netflix Platform
AMINATOR
Glisten
** And 30+ more projects at http://netflix.github.io/

24.
Keep the Conversation Going
Continuous Delivery
Open Space
Ballroom B/C (here!)
1:35-2:25, immediately
following lunch

25.
Thanks!
We’re always hiring!
Dianne Marsh (@dmarsh)
dmarsh@netflix.com

26.
Watch the video with slide synchronization on
InfoQ.com!
http://www.infoq.com/presentations/netflixcontinuous-delivery