SlideShare a Scribd company logo
Scaling Up
Continuous Deployment
    Timothy Fitz (.com)
“Continuous deployment involves
  deploying early and often, so as to
avoid the pitfalls of "deployment hell".
The practice aims to reduce timely rework and thus
       reduce cost and development time.”
Scalability
• Maintaining availability, performance and
  happiness
• As a function of # of people
• As a function of # of tests
Availability
• The build must always be green
• Set a “green SLA”
  – 99% green
  – Never red for > 15m
• Measure, track and report on these numbers
Performance
• Measure (intent to) commit to time of deploy
  – Goal: < 5 minutes
• Measure local development test loop
  – Goal: < 2s
Happiness
• CI/CD System is a product
• Software Engineers are the customer
• Keep your customer happy!
Testing Pyramid
How do you make tests fast?
• Tests can exercise large amounts of code
  without being slow
• Minimize system calls (no I/O, no disk)
• Minimize test data size
• Make sure all systems are cheap to
  instantiate/teardown
• No external state makes tests more reliable
Run Tests in Parallel
•   Multiprocess
•   Multimachine
•   Multi-VM
•   Instant multi-VM: http://circleci.com
Hardware Scale
• CI Cluster will get huge
  – Function of cumulative engineering man-months
  – Rule of thumb: 10% of your cluster size
• You will need a CI/CD DevOps person
  – CI cluster monitoring / alerting
  – Configuration Management critical
Scale testing infrastructure recap
• Write the right kind of tests
• Make those tests as fast as possible
• Run those tests in parallel
People / Roles
• Sheriff
  – Designated reverter / problem troubleshooter
  – Common pattern (IMVU, Chromium, Firefox)
• CD “Product Owner”
  – Held accountable for SLA / Performance
  – Manage infrastructure backlog
Single trunk
• Do this until it doesn’t work for you
• Gets painful in the 16 – 32 developer range
• Faster commit->deploy reduces the pain
  – But effort becomes prohibitive
“Try” pipeline
• Conceptually, a second tree that “doesn’t
  matter” but still gets tested for feedback
• Buildbot implements a patch-pushing version
• Takes a significant amount of pressure off of
  trunk builds
CI Server takes active role
• Server automatically reverts red commits
• Server merges green commits to trunk
Feature branches
• All incremental development happens on
  branches, branches land when feature is
  “ready”
• If “feature” is kept small, can be 2-3 per
  engineer per week on average
• Less continuous, but scales much better
  – Feature branches tested before merge
Merge tree
•   Tree per team / feature
•   Trees merged into trunk daily (if green)
•   Scale up via tree of trees (of trees…)
•   Again, less continuous
Federation
•   Each team gets their own deploy pipeline
•   Requires SOA / component architecture
•   Each team can set their own CD pace
•   “Enterprise Ready”
Recap
•   Single trunk + Try pipeline / Autorevert
•   Feature Branches
•   Merge Tree
•   Federation
Questions?

Timothy Fitz (.com)

More Related Content

What's hot

Continuous integration
Continuous integrationContinuous integration
Continuous integration
Basma Alkerm
 
CI/CD Overview
CI/CD OverviewCI/CD Overview
CI/CD Overview
An Nguyen
 
Challenges and best practices of database continuous delivery
Challenges and best practices of database continuous deliveryChallenges and best practices of database continuous delivery
Challenges and best practices of database continuous delivery
DBmaestro - Database DevOps
 
The Continuous delivery Value @ codemotion 2014
The Continuous delivery Value @ codemotion 2014The Continuous delivery Value @ codemotion 2014
The Continuous delivery Value @ codemotion 2014
David Funaro
 
Continuous Integration 101
Continuous Integration 101Continuous Integration 101
Continuous Integration 101
John Ferguson Smart Limited
 
Continuous integration
Continuous integrationContinuous integration
Continuous integration
Dennis van der Stelt
 
Continuous integration, delivery & deployment
Continuous integration,  delivery & deploymentContinuous integration,  delivery & deployment
Continuous integration, delivery & deployment
Martijn van der Kamp
 
Improving software quality using Continuous Integration
Improving software quality using Continuous IntegrationImproving software quality using Continuous Integration
Improving software quality using Continuous Integration
Wouter Konecny
 
Continuous Integration at T3CON08
Continuous Integration at T3CON08Continuous Integration at T3CON08
Continuous Integration at T3CON08
Sebastian Kurfürst
 
Continuous Integration and Builds
Continuous Integration and BuildsContinuous Integration and Builds
Continuous Integration and Builds
Bhavin Javia
 
Sailing through devlopment with legacy code
Sailing through devlopment with legacy codeSailing through devlopment with legacy code
Sailing through devlopment with legacy code
prasadkunte
 
Why NXTware Remote for Jenkins
Why NXTware Remote for JenkinsWhy NXTware Remote for Jenkins
Why NXTware Remote for Jenkins
ecubemarketing
 
WSO2 IoTS Device Manufacturer Guide
WSO2 IoTS Device Manufacturer GuideWSO2 IoTS Device Manufacturer Guide
WSO2 IoTS Device Manufacturer Guide
hugo lu
 
Production Ready WordPress #WPLDN
Production Ready WordPress #WPLDNProduction Ready WordPress #WPLDN
Production Ready WordPress #WPLDN
Edmund Turbin
 
Introduce to Git and Jenkins
Introduce to Git and JenkinsIntroduce to Git and Jenkins
Introduce to Git and Jenkins
An Nguyen
 
Trunk Based Development in the Enterprise - Its Relevance and Economics
Trunk Based Development in the Enterprise - Its Relevance and EconomicsTrunk Based Development in the Enterprise - Its Relevance and Economics
Trunk Based Development in the Enterprise - Its Relevance and Economics
Perforce
 
Simple ci cd with strider cd
Simple ci cd with strider cdSimple ci cd with strider cd
Simple ci cd with strider cd
Amandeep Singh
 
Distributed Release Management
Distributed Release ManagementDistributed Release Management
Distributed Release Management
Mike Brittain
 
Integration Testing as Validation and Monitoring
 Integration Testing as Validation and Monitoring Integration Testing as Validation and Monitoring
Integration Testing as Validation and Monitoring
Melissa Benua
 
Continuous Delivery
Continuous DeliveryContinuous Delivery
Continuous Delivery
Mikalai Alimenkou
 

What's hot (20)

Continuous integration
Continuous integrationContinuous integration
Continuous integration
 
CI/CD Overview
CI/CD OverviewCI/CD Overview
CI/CD Overview
 
Challenges and best practices of database continuous delivery
Challenges and best practices of database continuous deliveryChallenges and best practices of database continuous delivery
Challenges and best practices of database continuous delivery
 
The Continuous delivery Value @ codemotion 2014
The Continuous delivery Value @ codemotion 2014The Continuous delivery Value @ codemotion 2014
The Continuous delivery Value @ codemotion 2014
 
Continuous Integration 101
Continuous Integration 101Continuous Integration 101
Continuous Integration 101
 
Continuous integration
Continuous integrationContinuous integration
Continuous integration
 
Continuous integration, delivery & deployment
Continuous integration,  delivery & deploymentContinuous integration,  delivery & deployment
Continuous integration, delivery & deployment
 
Improving software quality using Continuous Integration
Improving software quality using Continuous IntegrationImproving software quality using Continuous Integration
Improving software quality using Continuous Integration
 
Continuous Integration at T3CON08
Continuous Integration at T3CON08Continuous Integration at T3CON08
Continuous Integration at T3CON08
 
Continuous Integration and Builds
Continuous Integration and BuildsContinuous Integration and Builds
Continuous Integration and Builds
 
Sailing through devlopment with legacy code
Sailing through devlopment with legacy codeSailing through devlopment with legacy code
Sailing through devlopment with legacy code
 
Why NXTware Remote for Jenkins
Why NXTware Remote for JenkinsWhy NXTware Remote for Jenkins
Why NXTware Remote for Jenkins
 
WSO2 IoTS Device Manufacturer Guide
WSO2 IoTS Device Manufacturer GuideWSO2 IoTS Device Manufacturer Guide
WSO2 IoTS Device Manufacturer Guide
 
Production Ready WordPress #WPLDN
Production Ready WordPress #WPLDNProduction Ready WordPress #WPLDN
Production Ready WordPress #WPLDN
 
Introduce to Git and Jenkins
Introduce to Git and JenkinsIntroduce to Git and Jenkins
Introduce to Git and Jenkins
 
Trunk Based Development in the Enterprise - Its Relevance and Economics
Trunk Based Development in the Enterprise - Its Relevance and EconomicsTrunk Based Development in the Enterprise - Its Relevance and Economics
Trunk Based Development in the Enterprise - Its Relevance and Economics
 
Simple ci cd with strider cd
Simple ci cd with strider cdSimple ci cd with strider cd
Simple ci cd with strider cd
 
Distributed Release Management
Distributed Release ManagementDistributed Release Management
Distributed Release Management
 
Integration Testing as Validation and Monitoring
 Integration Testing as Validation and Monitoring Integration Testing as Validation and Monitoring
Integration Testing as Validation and Monitoring
 
Continuous Delivery
Continuous DeliveryContinuous Delivery
Continuous Delivery
 

Viewers also liked

Continuous Delivery in the AWS Cloud
Continuous Delivery in the AWS CloudContinuous Delivery in the AWS Cloud
Continuous Delivery in the AWS Cloud
Nigel Fernandes
 
Analysis of TLS in SMTP World
Analysis of TLS in SMTP WorldAnalysis of TLS in SMTP World
Analysis of TLS in SMTP World
Binu Ramakrishnan
 
Infrastructure Continuous Delivery using CloudFormation
Infrastructure Continuous Delivery using CloudFormationInfrastructure Continuous Delivery using CloudFormation
Infrastructure Continuous Delivery using CloudFormation
joehack3r
 
AppSec++ Take the best of Agile, DevOps and CI/CD into your AppSec Program
AppSec++ Take the best of Agile, DevOps and CI/CD into your AppSec ProgramAppSec++ Take the best of Agile, DevOps and CI/CD into your AppSec Program
AppSec++ Take the best of Agile, DevOps and CI/CD into your AppSec Program
Matt Tesauro
 
Deep Dive: Infrastructure as Code
Deep Dive: Infrastructure as CodeDeep Dive: Infrastructure as Code
Deep Dive: Infrastructure as Code
Amazon Web Services
 
Infrastructure as Code with AWS CloudFormation
Infrastructure as Code with AWS CloudFormationInfrastructure as Code with AWS CloudFormation
Infrastructure as Code with AWS CloudFormation
Justyna Janczyszyn
 
Dod is not done
Dod is not doneDod is not done
Dod is not done
Kris Buytaert
 
IBM Innovate - Adoption of Continuous Delivery at Scale at a large telco v0 3
IBM Innovate - Adoption of Continuous Delivery at Scale at a large telco v0 3IBM Innovate - Adoption of Continuous Delivery at Scale at a large telco v0 3
IBM Innovate - Adoption of Continuous Delivery at Scale at a large telco v0 3
Mirco Hering
 
Jenkins CI + XebiaLabs for Release Orchestration: A Recipe for Continuous Del...
Jenkins CI + XebiaLabs for Release Orchestration: A Recipe for Continuous Del...Jenkins CI + XebiaLabs for Release Orchestration: A Recipe for Continuous Del...
Jenkins CI + XebiaLabs for Release Orchestration: A Recipe for Continuous Del...
XebiaLabs
 
CI&CD on AWS - Meetup Roma Oct 2016
CI&CD on AWS - Meetup Roma Oct 2016CI&CD on AWS - Meetup Roma Oct 2016
CI&CD on AWS - Meetup Roma Oct 2016
Paolo latella
 
Continuous Deployment: The Dirty Details
Continuous Deployment: The Dirty DetailsContinuous Deployment: The Dirty Details
Continuous Deployment: The Dirty Details
Mike Brittain
 
The Journey of devops and continuous delivery in a Large Financial Institution
The Journey of devops and continuous delivery in a Large Financial InstitutionThe Journey of devops and continuous delivery in a Large Financial Institution
The Journey of devops and continuous delivery in a Large Financial Institution
Kris Buytaert
 
Securing application deployments in multi-tenant CI/CD environments
Securing application deployments in multi-tenant CI/CD environmentsSecuring application deployments in multi-tenant CI/CD environments
Securing application deployments in multi-tenant CI/CD environments
Binu Ramakrishnan
 
Continuous Integration, Build Pipelines and Continuous Deployment
Continuous Integration, Build Pipelines and Continuous DeploymentContinuous Integration, Build Pipelines and Continuous Deployment
Continuous Integration, Build Pipelines and Continuous Deployment
Christopher Read
 
O'Reilly/Nginx 2016: "Continuous Delivery with Containers: The Trials and Tri...
O'Reilly/Nginx 2016: "Continuous Delivery with Containers: The Trials and Tri...O'Reilly/Nginx 2016: "Continuous Delivery with Containers: The Trials and Tri...
O'Reilly/Nginx 2016: "Continuous Delivery with Containers: The Trials and Tri...
Daniel Bryant
 
AWS May Webinar Series - Deep Dive: Infrastructure as Code
AWS May Webinar Series - Deep Dive: Infrastructure as CodeAWS May Webinar Series - Deep Dive: Infrastructure as Code
AWS May Webinar Series - Deep Dive: Infrastructure as Code
Amazon Web Services
 
Infrastructure Continuous Delivery Using AWS CloudFormation
Infrastructure Continuous Delivery Using AWS CloudFormationInfrastructure Continuous Delivery Using AWS CloudFormation
Infrastructure Continuous Delivery Using AWS CloudFormation
Amazon Web Services
 
Enterprise DevOps at Scale with AWS | AWS Public Sector Summit 2016
Enterprise DevOps at Scale with AWS | AWS Public Sector Summit 2016Enterprise DevOps at Scale with AWS | AWS Public Sector Summit 2016
Enterprise DevOps at Scale with AWS | AWS Public Sector Summit 2016
Amazon Web Services
 

Viewers also liked (18)

Continuous Delivery in the AWS Cloud
Continuous Delivery in the AWS CloudContinuous Delivery in the AWS Cloud
Continuous Delivery in the AWS Cloud
 
Analysis of TLS in SMTP World
Analysis of TLS in SMTP WorldAnalysis of TLS in SMTP World
Analysis of TLS in SMTP World
 
Infrastructure Continuous Delivery using CloudFormation
Infrastructure Continuous Delivery using CloudFormationInfrastructure Continuous Delivery using CloudFormation
Infrastructure Continuous Delivery using CloudFormation
 
AppSec++ Take the best of Agile, DevOps and CI/CD into your AppSec Program
AppSec++ Take the best of Agile, DevOps and CI/CD into your AppSec ProgramAppSec++ Take the best of Agile, DevOps and CI/CD into your AppSec Program
AppSec++ Take the best of Agile, DevOps and CI/CD into your AppSec Program
 
Deep Dive: Infrastructure as Code
Deep Dive: Infrastructure as CodeDeep Dive: Infrastructure as Code
Deep Dive: Infrastructure as Code
 
Infrastructure as Code with AWS CloudFormation
Infrastructure as Code with AWS CloudFormationInfrastructure as Code with AWS CloudFormation
Infrastructure as Code with AWS CloudFormation
 
Dod is not done
Dod is not doneDod is not done
Dod is not done
 
IBM Innovate - Adoption of Continuous Delivery at Scale at a large telco v0 3
IBM Innovate - Adoption of Continuous Delivery at Scale at a large telco v0 3IBM Innovate - Adoption of Continuous Delivery at Scale at a large telco v0 3
IBM Innovate - Adoption of Continuous Delivery at Scale at a large telco v0 3
 
Jenkins CI + XebiaLabs for Release Orchestration: A Recipe for Continuous Del...
Jenkins CI + XebiaLabs for Release Orchestration: A Recipe for Continuous Del...Jenkins CI + XebiaLabs for Release Orchestration: A Recipe for Continuous Del...
Jenkins CI + XebiaLabs for Release Orchestration: A Recipe for Continuous Del...
 
CI&CD on AWS - Meetup Roma Oct 2016
CI&CD on AWS - Meetup Roma Oct 2016CI&CD on AWS - Meetup Roma Oct 2016
CI&CD on AWS - Meetup Roma Oct 2016
 
Continuous Deployment: The Dirty Details
Continuous Deployment: The Dirty DetailsContinuous Deployment: The Dirty Details
Continuous Deployment: The Dirty Details
 
The Journey of devops and continuous delivery in a Large Financial Institution
The Journey of devops and continuous delivery in a Large Financial InstitutionThe Journey of devops and continuous delivery in a Large Financial Institution
The Journey of devops and continuous delivery in a Large Financial Institution
 
Securing application deployments in multi-tenant CI/CD environments
Securing application deployments in multi-tenant CI/CD environmentsSecuring application deployments in multi-tenant CI/CD environments
Securing application deployments in multi-tenant CI/CD environments
 
Continuous Integration, Build Pipelines and Continuous Deployment
Continuous Integration, Build Pipelines and Continuous DeploymentContinuous Integration, Build Pipelines and Continuous Deployment
Continuous Integration, Build Pipelines and Continuous Deployment
 
O'Reilly/Nginx 2016: "Continuous Delivery with Containers: The Trials and Tri...
O'Reilly/Nginx 2016: "Continuous Delivery with Containers: The Trials and Tri...O'Reilly/Nginx 2016: "Continuous Delivery with Containers: The Trials and Tri...
O'Reilly/Nginx 2016: "Continuous Delivery with Containers: The Trials and Tri...
 
AWS May Webinar Series - Deep Dive: Infrastructure as Code
AWS May Webinar Series - Deep Dive: Infrastructure as CodeAWS May Webinar Series - Deep Dive: Infrastructure as Code
AWS May Webinar Series - Deep Dive: Infrastructure as Code
 
Infrastructure Continuous Delivery Using AWS CloudFormation
Infrastructure Continuous Delivery Using AWS CloudFormationInfrastructure Continuous Delivery Using AWS CloudFormation
Infrastructure Continuous Delivery Using AWS CloudFormation
 
Enterprise DevOps at Scale with AWS | AWS Public Sector Summit 2016
Enterprise DevOps at Scale with AWS | AWS Public Sector Summit 2016Enterprise DevOps at Scale with AWS | AWS Public Sector Summit 2016
Enterprise DevOps at Scale with AWS | AWS Public Sector Summit 2016
 

Similar to Scaling Up Continuous Deployment

Introduction to continuous delivery
Introduction to continuous deliveryIntroduction to continuous delivery
Introduction to continuous delivery
OlympicSoftware
 
Test Driven Development & CI/CD
Test Driven Development & CI/CDTest Driven Development & CI/CD
Test Driven Development & CI/CD
Shanmuga S Muthu
 
Road to Continuous Delivery - Wix.com
Road to Continuous Delivery - Wix.comRoad to Continuous Delivery - Wix.com
Road to Continuous Delivery - Wix.com
Aviran Mordo
 
Continuous integration sql in the city
Continuous integration sql in the cityContinuous integration sql in the city
Continuous integration sql in the city
Ike Ellis
 
Panel Discussion Continuous Deployment in SaaS
Panel Discussion Continuous Deployment in SaaSPanel Discussion Continuous Deployment in SaaS
Panel Discussion Continuous Deployment in SaaS
Jonas Cheng
 
DevOps and AWS
DevOps and AWSDevOps and AWS
DevOps and AWS
Shiva Narayanaswamy
 
2016 09-dev opsjourney-devopsdaysoslo
2016 09-dev opsjourney-devopsdaysoslo2016 09-dev opsjourney-devopsdaysoslo
2016 09-dev opsjourney-devopsdaysoslo
Jon Arild Tørresdal
 
Make It Cooler: Using Decentralized Version Control
Make It Cooler: Using Decentralized Version ControlMake It Cooler: Using Decentralized Version Control
Make It Cooler: Using Decentralized Version Control
indiver
 
NYC MeetUp 10.9
NYC MeetUp 10.9NYC MeetUp 10.9
NYC MeetUp 10.9
Solano Labs
 
DevOps 101
DevOps 101DevOps 101
DevOps 101
Liran Levy
 
SCM Transformation Challenges and How to Overcome Them
SCM Transformation Challenges and How to Overcome ThemSCM Transformation Challenges and How to Overcome Them
SCM Transformation Challenges and How to Overcome Them
Compuware
 
Devops - why, what and how?
Devops - why, what and how?Devops - why, what and how?
Devops - why, what and how?
Malinda Kapuruge
 
DevOps, Common use cases, Architectures, Best Practices
DevOps, Common use cases, Architectures, Best PracticesDevOps, Common use cases, Architectures, Best Practices
DevOps, Common use cases, Architectures, Best Practices
Shiva Narayanaswamy
 
Got Problems? Let's Do a Health Check
Got Problems? Let's Do a Health CheckGot Problems? Let's Do a Health Check
Got Problems? Let's Do a Health Check
Luis Guirigay
 
DevOps / Agile Tools Seminar 2013
DevOps / Agile Tools Seminar 2013DevOps / Agile Tools Seminar 2013
DevOps / Agile Tools Seminar 2013
Ethan Ram
 
Continuous delivery - takeaways
Continuous delivery - takeawaysContinuous delivery - takeaways
Continuous delivery - takeaways
Manuela Grindei
 
The Rocky Cloud Road
The Rocky Cloud RoadThe Rocky Cloud Road
The Rocky Cloud Road
Gert Drapers
 
So you-want-to-go-faster
So you-want-to-go-fasterSo you-want-to-go-faster
So you-want-to-go-faster
Ooblioob
 
Boston MeetUp 10.10
Boston MeetUp 10.10Boston MeetUp 10.10
Boston MeetUp 10.10
Solano Labs
 
Jenkins Introduction
Jenkins IntroductionJenkins Introduction
Jenkins Introduction
Pavan Gupta
 

Similar to Scaling Up Continuous Deployment (20)

Introduction to continuous delivery
Introduction to continuous deliveryIntroduction to continuous delivery
Introduction to continuous delivery
 
Test Driven Development & CI/CD
Test Driven Development & CI/CDTest Driven Development & CI/CD
Test Driven Development & CI/CD
 
Road to Continuous Delivery - Wix.com
Road to Continuous Delivery - Wix.comRoad to Continuous Delivery - Wix.com
Road to Continuous Delivery - Wix.com
 
Continuous integration sql in the city
Continuous integration sql in the cityContinuous integration sql in the city
Continuous integration sql in the city
 
Panel Discussion Continuous Deployment in SaaS
Panel Discussion Continuous Deployment in SaaSPanel Discussion Continuous Deployment in SaaS
Panel Discussion Continuous Deployment in SaaS
 
DevOps and AWS
DevOps and AWSDevOps and AWS
DevOps and AWS
 
2016 09-dev opsjourney-devopsdaysoslo
2016 09-dev opsjourney-devopsdaysoslo2016 09-dev opsjourney-devopsdaysoslo
2016 09-dev opsjourney-devopsdaysoslo
 
Make It Cooler: Using Decentralized Version Control
Make It Cooler: Using Decentralized Version ControlMake It Cooler: Using Decentralized Version Control
Make It Cooler: Using Decentralized Version Control
 
NYC MeetUp 10.9
NYC MeetUp 10.9NYC MeetUp 10.9
NYC MeetUp 10.9
 
DevOps 101
DevOps 101DevOps 101
DevOps 101
 
SCM Transformation Challenges and How to Overcome Them
SCM Transformation Challenges and How to Overcome ThemSCM Transformation Challenges and How to Overcome Them
SCM Transformation Challenges and How to Overcome Them
 
Devops - why, what and how?
Devops - why, what and how?Devops - why, what and how?
Devops - why, what and how?
 
DevOps, Common use cases, Architectures, Best Practices
DevOps, Common use cases, Architectures, Best PracticesDevOps, Common use cases, Architectures, Best Practices
DevOps, Common use cases, Architectures, Best Practices
 
Got Problems? Let's Do a Health Check
Got Problems? Let's Do a Health CheckGot Problems? Let's Do a Health Check
Got Problems? Let's Do a Health Check
 
DevOps / Agile Tools Seminar 2013
DevOps / Agile Tools Seminar 2013DevOps / Agile Tools Seminar 2013
DevOps / Agile Tools Seminar 2013
 
Continuous delivery - takeaways
Continuous delivery - takeawaysContinuous delivery - takeaways
Continuous delivery - takeaways
 
The Rocky Cloud Road
The Rocky Cloud RoadThe Rocky Cloud Road
The Rocky Cloud Road
 
So you-want-to-go-faster
So you-want-to-go-fasterSo you-want-to-go-faster
So you-want-to-go-faster
 
Boston MeetUp 10.10
Boston MeetUp 10.10Boston MeetUp 10.10
Boston MeetUp 10.10
 
Jenkins Introduction
Jenkins IntroductionJenkins Introduction
Jenkins Introduction
 

More from Timothy Fitz

Good hypothesis testing is surprising!
Good hypothesis testing is surprising!Good hypothesis testing is surprising!
Good hypothesis testing is surprising!
Timothy Fitz
 
Gdc 2010 architecture final slideshare edition
Gdc 2010 architecture final slideshare editionGdc 2010 architecture final slideshare edition
Gdc 2010 architecture final slideshare edition
Timothy Fitz
 
Realtime web2012
Realtime web2012Realtime web2012
Realtime web2012
Timothy Fitz
 
Continuous Deployment
Continuous DeploymentContinuous Deployment
Continuous Deployment
Timothy Fitz
 
Continuous Deployment
Continuous DeploymentContinuous Deployment
Continuous Deployment
Timothy Fitz
 
Shdh
ShdhShdh
Socket.io
Socket.ioSocket.io
Socket.io
Timothy Fitz
 
Shdh
ShdhShdh
Just In Time Scalability Agile Methods To Support Massive Growth Presentation
Just In Time Scalability  Agile Methods To Support Massive Growth PresentationJust In Time Scalability  Agile Methods To Support Massive Growth Presentation
Just In Time Scalability Agile Methods To Support Massive Growth Presentation
Timothy Fitz
 

More from Timothy Fitz (9)

Good hypothesis testing is surprising!
Good hypothesis testing is surprising!Good hypothesis testing is surprising!
Good hypothesis testing is surprising!
 
Gdc 2010 architecture final slideshare edition
Gdc 2010 architecture final slideshare editionGdc 2010 architecture final slideshare edition
Gdc 2010 architecture final slideshare edition
 
Realtime web2012
Realtime web2012Realtime web2012
Realtime web2012
 
Continuous Deployment
Continuous DeploymentContinuous Deployment
Continuous Deployment
 
Continuous Deployment
Continuous DeploymentContinuous Deployment
Continuous Deployment
 
Shdh
ShdhShdh
Shdh
 
Socket.io
Socket.ioSocket.io
Socket.io
 
Shdh
ShdhShdh
Shdh
 
Just In Time Scalability Agile Methods To Support Massive Growth Presentation
Just In Time Scalability  Agile Methods To Support Massive Growth PresentationJust In Time Scalability  Agile Methods To Support Massive Growth Presentation
Just In Time Scalability Agile Methods To Support Massive Growth Presentation
 

Recently uploaded

Your One-Stop Shop for Python Success: Top 10 US Python Development Providers
Your One-Stop Shop for Python Success: Top 10 US Python Development ProvidersYour One-Stop Shop for Python Success: Top 10 US Python Development Providers
Your One-Stop Shop for Python Success: Top 10 US Python Development Providers
akankshawande
 
"Choosing proper type of scaling", Olena Syrota
"Choosing proper type of scaling", Olena Syrota"Choosing proper type of scaling", Olena Syrota
"Choosing proper type of scaling", Olena Syrota
Fwdays
 
Programming Foundation Models with DSPy - Meetup Slides
Programming Foundation Models with DSPy - Meetup SlidesProgramming Foundation Models with DSPy - Meetup Slides
Programming Foundation Models with DSPy - Meetup Slides
Zilliz
 
[OReilly Superstream] Occupy the Space: A grassroots guide to engineering (an...
[OReilly Superstream] Occupy the Space: A grassroots guide to engineering (an...[OReilly Superstream] Occupy the Space: A grassroots guide to engineering (an...
[OReilly Superstream] Occupy the Space: A grassroots guide to engineering (an...
Jason Yip
 
Monitoring and Managing Anomaly Detection on OpenShift.pdf
Monitoring and Managing Anomaly Detection on OpenShift.pdfMonitoring and Managing Anomaly Detection on OpenShift.pdf
Monitoring and Managing Anomaly Detection on OpenShift.pdf
Tosin Akinosho
 
Choosing The Best AWS Service For Your Website + API.pptx
Choosing The Best AWS Service For Your Website + API.pptxChoosing The Best AWS Service For Your Website + API.pptx
Choosing The Best AWS Service For Your Website + API.pptx
Brandon Minnick, MBA
 
Presentation of the OECD Artificial Intelligence Review of Germany
Presentation of the OECD Artificial Intelligence Review of GermanyPresentation of the OECD Artificial Intelligence Review of Germany
Presentation of the OECD Artificial Intelligence Review of Germany
innovationoecd
 
Introduction of Cybersecurity with OSS at Code Europe 2024
Introduction of Cybersecurity with OSS  at Code Europe 2024Introduction of Cybersecurity with OSS  at Code Europe 2024
Introduction of Cybersecurity with OSS at Code Europe 2024
Hiroshi SHIBATA
 
Skybuffer SAM4U tool for SAP license adoption
Skybuffer SAM4U tool for SAP license adoptionSkybuffer SAM4U tool for SAP license adoption
Skybuffer SAM4U tool for SAP license adoption
Tatiana Kojar
 
GraphRAG for LifeSciences Hands-On with the Clinical Knowledge Graph
GraphRAG for LifeSciences Hands-On with the Clinical Knowledge GraphGraphRAG for LifeSciences Hands-On with the Clinical Knowledge Graph
GraphRAG for LifeSciences Hands-On with the Clinical Knowledge Graph
Neo4j
 
WeTestAthens: Postman's AI & Automation Techniques
WeTestAthens: Postman's AI & Automation TechniquesWeTestAthens: Postman's AI & Automation Techniques
WeTestAthens: Postman's AI & Automation Techniques
Postman
 
Apps Break Data
Apps Break DataApps Break Data
Apps Break Data
Ivo Velitchkov
 
Deep Dive: AI-Powered Marketing to Get More Leads and Customers with HyperGro...
Deep Dive: AI-Powered Marketing to Get More Leads and Customers with HyperGro...Deep Dive: AI-Powered Marketing to Get More Leads and Customers with HyperGro...
Deep Dive: AI-Powered Marketing to Get More Leads and Customers with HyperGro...
saastr
 
Astute Business Solutions | Oracle Cloud Partner |
Astute Business Solutions | Oracle Cloud Partner |Astute Business Solutions | Oracle Cloud Partner |
Astute Business Solutions | Oracle Cloud Partner |
AstuteBusiness
 
Dandelion Hashtable: beyond billion requests per second on a commodity server
Dandelion Hashtable: beyond billion requests per second on a commodity serverDandelion Hashtable: beyond billion requests per second on a commodity server
Dandelion Hashtable: beyond billion requests per second on a commodity server
Antonios Katsarakis
 
Serial Arm Control in Real Time Presentation
Serial Arm Control in Real Time PresentationSerial Arm Control in Real Time Presentation
Serial Arm Control in Real Time Presentation
tolgahangng
 
Digital Banking in the Cloud: How Citizens Bank Unlocked Their Mainframe
Digital Banking in the Cloud: How Citizens Bank Unlocked Their MainframeDigital Banking in the Cloud: How Citizens Bank Unlocked Their Mainframe
Digital Banking in the Cloud: How Citizens Bank Unlocked Their Mainframe
Precisely
 
Main news related to the CCS TSI 2023 (2023/1695)
Main news related to the CCS TSI 2023 (2023/1695)Main news related to the CCS TSI 2023 (2023/1695)
Main news related to the CCS TSI 2023 (2023/1695)
Jakub Marek
 
Generating privacy-protected synthetic data using Secludy and Milvus
Generating privacy-protected synthetic data using Secludy and MilvusGenerating privacy-protected synthetic data using Secludy and Milvus
Generating privacy-protected synthetic data using Secludy and Milvus
Zilliz
 
5th LF Energy Power Grid Model Meet-up Slides
5th LF Energy Power Grid Model Meet-up Slides5th LF Energy Power Grid Model Meet-up Slides
5th LF Energy Power Grid Model Meet-up Slides
DanBrown980551
 

Recently uploaded (20)

Your One-Stop Shop for Python Success: Top 10 US Python Development Providers
Your One-Stop Shop for Python Success: Top 10 US Python Development ProvidersYour One-Stop Shop for Python Success: Top 10 US Python Development Providers
Your One-Stop Shop for Python Success: Top 10 US Python Development Providers
 
"Choosing proper type of scaling", Olena Syrota
"Choosing proper type of scaling", Olena Syrota"Choosing proper type of scaling", Olena Syrota
"Choosing proper type of scaling", Olena Syrota
 
Programming Foundation Models with DSPy - Meetup Slides
Programming Foundation Models with DSPy - Meetup SlidesProgramming Foundation Models with DSPy - Meetup Slides
Programming Foundation Models with DSPy - Meetup Slides
 
[OReilly Superstream] Occupy the Space: A grassroots guide to engineering (an...
[OReilly Superstream] Occupy the Space: A grassroots guide to engineering (an...[OReilly Superstream] Occupy the Space: A grassroots guide to engineering (an...
[OReilly Superstream] Occupy the Space: A grassroots guide to engineering (an...
 
Monitoring and Managing Anomaly Detection on OpenShift.pdf
Monitoring and Managing Anomaly Detection on OpenShift.pdfMonitoring and Managing Anomaly Detection on OpenShift.pdf
Monitoring and Managing Anomaly Detection on OpenShift.pdf
 
Choosing The Best AWS Service For Your Website + API.pptx
Choosing The Best AWS Service For Your Website + API.pptxChoosing The Best AWS Service For Your Website + API.pptx
Choosing The Best AWS Service For Your Website + API.pptx
 
Presentation of the OECD Artificial Intelligence Review of Germany
Presentation of the OECD Artificial Intelligence Review of GermanyPresentation of the OECD Artificial Intelligence Review of Germany
Presentation of the OECD Artificial Intelligence Review of Germany
 
Introduction of Cybersecurity with OSS at Code Europe 2024
Introduction of Cybersecurity with OSS  at Code Europe 2024Introduction of Cybersecurity with OSS  at Code Europe 2024
Introduction of Cybersecurity with OSS at Code Europe 2024
 
Skybuffer SAM4U tool for SAP license adoption
Skybuffer SAM4U tool for SAP license adoptionSkybuffer SAM4U tool for SAP license adoption
Skybuffer SAM4U tool for SAP license adoption
 
GraphRAG for LifeSciences Hands-On with the Clinical Knowledge Graph
GraphRAG for LifeSciences Hands-On with the Clinical Knowledge GraphGraphRAG for LifeSciences Hands-On with the Clinical Knowledge Graph
GraphRAG for LifeSciences Hands-On with the Clinical Knowledge Graph
 
WeTestAthens: Postman's AI & Automation Techniques
WeTestAthens: Postman's AI & Automation TechniquesWeTestAthens: Postman's AI & Automation Techniques
WeTestAthens: Postman's AI & Automation Techniques
 
Apps Break Data
Apps Break DataApps Break Data
Apps Break Data
 
Deep Dive: AI-Powered Marketing to Get More Leads and Customers with HyperGro...
Deep Dive: AI-Powered Marketing to Get More Leads and Customers with HyperGro...Deep Dive: AI-Powered Marketing to Get More Leads and Customers with HyperGro...
Deep Dive: AI-Powered Marketing to Get More Leads and Customers with HyperGro...
 
Astute Business Solutions | Oracle Cloud Partner |
Astute Business Solutions | Oracle Cloud Partner |Astute Business Solutions | Oracle Cloud Partner |
Astute Business Solutions | Oracle Cloud Partner |
 
Dandelion Hashtable: beyond billion requests per second on a commodity server
Dandelion Hashtable: beyond billion requests per second on a commodity serverDandelion Hashtable: beyond billion requests per second on a commodity server
Dandelion Hashtable: beyond billion requests per second on a commodity server
 
Serial Arm Control in Real Time Presentation
Serial Arm Control in Real Time PresentationSerial Arm Control in Real Time Presentation
Serial Arm Control in Real Time Presentation
 
Digital Banking in the Cloud: How Citizens Bank Unlocked Their Mainframe
Digital Banking in the Cloud: How Citizens Bank Unlocked Their MainframeDigital Banking in the Cloud: How Citizens Bank Unlocked Their Mainframe
Digital Banking in the Cloud: How Citizens Bank Unlocked Their Mainframe
 
Main news related to the CCS TSI 2023 (2023/1695)
Main news related to the CCS TSI 2023 (2023/1695)Main news related to the CCS TSI 2023 (2023/1695)
Main news related to the CCS TSI 2023 (2023/1695)
 
Generating privacy-protected synthetic data using Secludy and Milvus
Generating privacy-protected synthetic data using Secludy and MilvusGenerating privacy-protected synthetic data using Secludy and Milvus
Generating privacy-protected synthetic data using Secludy and Milvus
 
5th LF Energy Power Grid Model Meet-up Slides
5th LF Energy Power Grid Model Meet-up Slides5th LF Energy Power Grid Model Meet-up Slides
5th LF Energy Power Grid Model Meet-up Slides
 

Scaling Up Continuous Deployment

  • 1. Scaling Up Continuous Deployment Timothy Fitz (.com)
  • 2. “Continuous deployment involves deploying early and often, so as to avoid the pitfalls of "deployment hell". The practice aims to reduce timely rework and thus reduce cost and development time.”
  • 3. Scalability • Maintaining availability, performance and happiness • As a function of # of people • As a function of # of tests
  • 4. Availability • The build must always be green • Set a “green SLA” – 99% green – Never red for > 15m • Measure, track and report on these numbers
  • 5. Performance • Measure (intent to) commit to time of deploy – Goal: < 5 minutes • Measure local development test loop – Goal: < 2s
  • 6. Happiness • CI/CD System is a product • Software Engineers are the customer • Keep your customer happy!
  • 8. How do you make tests fast? • Tests can exercise large amounts of code without being slow • Minimize system calls (no I/O, no disk) • Minimize test data size • Make sure all systems are cheap to instantiate/teardown • No external state makes tests more reliable
  • 9. Run Tests in Parallel • Multiprocess • Multimachine • Multi-VM • Instant multi-VM: http://circleci.com
  • 10. Hardware Scale • CI Cluster will get huge – Function of cumulative engineering man-months – Rule of thumb: 10% of your cluster size • You will need a CI/CD DevOps person – CI cluster monitoring / alerting – Configuration Management critical
  • 11. Scale testing infrastructure recap • Write the right kind of tests • Make those tests as fast as possible • Run those tests in parallel
  • 12. People / Roles • Sheriff – Designated reverter / problem troubleshooter – Common pattern (IMVU, Chromium, Firefox) • CD “Product Owner” – Held accountable for SLA / Performance – Manage infrastructure backlog
  • 13. Single trunk • Do this until it doesn’t work for you • Gets painful in the 16 – 32 developer range • Faster commit->deploy reduces the pain – But effort becomes prohibitive
  • 14. “Try” pipeline • Conceptually, a second tree that “doesn’t matter” but still gets tested for feedback • Buildbot implements a patch-pushing version • Takes a significant amount of pressure off of trunk builds
  • 15. CI Server takes active role • Server automatically reverts red commits • Server merges green commits to trunk
  • 16. Feature branches • All incremental development happens on branches, branches land when feature is “ready” • If “feature” is kept small, can be 2-3 per engineer per week on average • Less continuous, but scales much better – Feature branches tested before merge
  • 17. Merge tree • Tree per team / feature • Trees merged into trunk daily (if green) • Scale up via tree of trees (of trees…) • Again, less continuous
  • 18. Federation • Each team gets their own deploy pipeline • Requires SOA / component architecture • Each team can set their own CD pace • “Enterprise Ready”
  • 19. Recap • Single trunk + Try pipeline / Autorevert • Feature Branches • Merge Tree • Federation

Editor's Notes

  1. About me: IMVU, Canvas, Continuous Deployment)Ground rules: I don’t demand your attention, please tweet / follow links while I’m talking. If you have questions, shoot up a hand. If I don’t see you, yell at me.
  2. Continous Deployment vs Continuous Delivery (next slide)
  3. Commit to deploy&lt; 5 minute: stay in flow5 – 15 minutes: can keep working on feature15 minutes: failures are surprises and require expensive rewindingLocal dev loop&lt; 2s: stay in tight flow2s – 10s: tab away from terminal, looser flow10s – 1min: start thinking or coding on next thing, failures require rewinding1m – 5min: significant rewinding, high distraction, painful testing (rolling chair jousting)
  4. Example: All metrics green, everything looks great, but got to metrics by shaming anyone who breaks build. Culture of tip-toeing through the bulid system leads to reduced happiness (and reduced throughput!)Sidenote: Measure throughput! Deploys per engineer (avg/median/extremes), is it scaling up with org or are you getting less deploys?
  5. GUI Testsi.e. SeleniumIntegration Testsi.e. uses the databaseUnit Testssmall fast tests with no external state
  6. i.e. Don’t reload your country code or ipgeo tables, even into memory from disk, on every test
  7. Highly recommend Buildbot when you hit scale. It’s proven at huge scale (500+ nodes) and growing, and allows way better pipeline customization at that scale
  8. Test time is O(Cumulative man-months)Doubling staff means halving test time, in spite of ever-marching-on increase in tests
  9. Github modelMost FOSS is moving to or using this model
  10. Lots of people/process overheadClassical way of scaling up to extremely large teams
  11. Fortune 500-proof.
  12. Remember: Don’t overinvest up front. Better to do something simple until it doesn’t work, than to overbuild.Don’t underbuild either. Use