SlideShare a Scribd company logo
1 of 36
Download to read offline
The Hard Problems
of Continuous Deployment
     Timothy Fitz (.com)
“Continuous deployment involves
  deploying early and often, so as to
avoid the pitfalls of "deployment hell".
The practice aims to reduce timely rework and thus
       reduce cost and development time.”
“Our highest priority is to satisfy
  the customer through early
  and continuous delivery of
      valuable software.”
- Principles behind the Agile Manifesto (2001)
The Hard Problems
           (The ones I’m going to cover)
• Data
• Mobile
• Scale
Data makes CD harder
•   Updating data is slow
•   Moving data is slow
•   Schemas live outside code deploy process
•   Rollbacks are often hard or impossible
Schema rev. N, N+1 compatible
• Add the column in production
• Push the code that writes to that column
• Optionally, run a data migration to populate
  the existing rows with data
• Push the code that reads from that column.
Apply only cheap changes
• Only apply changes that are cheap enough to
  not affect live traffic
• More complex changes split into tiny steps:
  – Create new table
  – Write to both
  – Cut over eventually
  – Drop old table
Apply change to standby
•   Run two DB instances
•   Apply change to standby
•   Failover if successfully applied
•   Might run 3rd db instance for availability
Blue/Green Deploy
• Run two copies of entire cluster
• All databases are replicated
• Lets you test, update and rollback both code
  and schema in one step
Schemaless NoSQL
• “Schemaless” really means schema is in your
  source code
  – Which is great! We can CD schema changes
Schemaless-style SQL
• Benefits of existing SQL databases while still
  being pseudoschemaless
• (thing_id, key, value) table
• (guid, blob) table
Recap
•   Apply only cheap changes
•   Apply changes to standby
•   Blue/Green Deploy
•   Schemaless NoSQL
•   Schemaless-style SQL
Mobile
• Users must opt-in to every update
• iOS submissions take a week to be approved
• Luckily, lots of tools aimed at the space
Remember the basics
• CI server / automated tests are critical
• Can’t fallback on production alarms / rollback
• Hosted options: http://cisimple.com
Data-drive everything
• Build your views / content from data files
• Ping server for updates
• Hosted: http://appgrok.com/
  – Lets you deploy txt, png and xib dynamically!
99% HTML
• Entire app is a single UIWebView
• Glue native code to allow access to APIs
• Clutch.io is awesome (and FOSS now)
  – Live reloading for local dev
  – Streamlined deploys
  – https://github.com/clutchio
Hybrid HTML/Native
• Core app is native
• Sections can be replaced by HTML
  – i.e. Facebook stream entries fallback to HTML
• Infrequently used sections are 100% HTML
Recap
•   Remember the basics
•   Data-drive everything
•   99% HTML
•   Hybrid HTML/Native
Scalability
• Maintaining availability, performance and
  happiness
• As a function of # of people
• As a function of # of tests
Availability
• The build must always be green
• Set a “green SLA”
  – 99% green
  – Never red for > 15m
• Measure, track and report on these numbers
Performance
• Measure (intent to) commit to time of deploy
  – Goal: < 5 minutes
• Measure local development test loop
  – Goal: < 2s
Happiness
• CI/CD System is a product
• Software Engineers are the customer
• Keep your customer happy!
Testing Pyramid
How do you make tests fast?
• Tests can exercise large amounts of code
  without being slow
• Minimize system calls (no I/O, no disk)
• Minimize test data size
• Make sure all systems are cheap to
  instantiate/teardown
• No external state makes tests more reliable
Run Tests in Parallel
•   Multiprocess
•   Multimachine
•   Multi-VM
•   Instant multi-VM: http://circleci.com
Hardware Scale
• CI Cluster will get huge
  – Function of cumulative engineering man-months
  – Rule of thumb: 10% of your cluster size
• You will need a CI/CD DevOps person
  – CI cluster monitoring / alerting
  – Configuration Management critical
Scale testing infrastructure recap
• Write the right kind of tests
• Make those tests as fast as possible
• Run those tests in parallel
People / Roles
• Sheriff
  – Designated reverter / problem troubleshooter
  – Common pattern (IMVU, Chromium, Firefox)
• CD “Product Owner”
  – Held accountable for SLA / Performance
  – Manage infrastructure backlog
Single trunk
• Do this until it doesn’t work for you
• Gets painful in the 16 – 32 developer range
• Faster commit->deploy reduces the pain
  – But effort becomes prohibitive
“Try” pipeline
• Conceptually, a second tree that “doesn’t
  matter” but still gets tested for feedback
• Buildbot implements a patch-pushing version
• Takes a significant amount of pressure off of
  trunk builds
CI Server takes active role
• Server automatically reverts red commits
• Server merges green commits to trunk
Feature branches
• All incremental development happens on
  branches, branches land when feature is
  “ready”
• If “feature” is kept small, can be 2-3 per
  engineer per week on average
• Less continuous, but scales much better
  – Feature branches tested before merge
Merge tree
•   Tree per team / feature
•   Trees merged into trunk daily (if green)
•   Scale up via tree of trees (of trees…)
•   Again, less continuous
Federation
•   Each team gets their own deploy pipeline
•   Requires SOA / component architecture
•   Each team can set their own CD pace
•   “Enterprise Ready”
Recap
•   Single trunk + Try pipeline / Autorevert
•   Feature Branches
•   Merge Tree
•   Federation
Questions?

Timothy Fitz (.com)

More Related Content

What's hot

Continuous delivery @wcap 5-09-2013
Continuous delivery   @wcap 5-09-2013Continuous delivery   @wcap 5-09-2013
Continuous delivery @wcap 5-09-2013David Funaro
 
The Continuous delivery Value @ codemotion 2014
The Continuous delivery Value @ codemotion 2014The Continuous delivery Value @ codemotion 2014
The Continuous delivery Value @ codemotion 2014David Funaro
 
Introduction to Continuous Integration
Introduction to Continuous IntegrationIntroduction to Continuous Integration
Introduction to Continuous IntegrationSomkiat Puisungnoen
 
Continuous Deployment of Rails Applications
Continuous Deployment of Rails ApplicationsContinuous Deployment of Rails Applications
Continuous Deployment of Rails ApplicationsDiane Cessna
 
Continuous integration
Continuous integrationContinuous integration
Continuous integrationhugo lu
 
Continuous Integration at T3CON08
Continuous Integration at T3CON08Continuous Integration at T3CON08
Continuous Integration at T3CON08Sebastian Kurfürst
 
Best Practices for Database Deployments
Best Practices for Database DeploymentsBest Practices for Database Deployments
Best Practices for Database DeploymentsRed Gate Software
 
Database Deployment Pipeline - SQL In The City Workshop 2014
Database Deployment Pipeline - SQL In The City Workshop 2014Database Deployment Pipeline - SQL In The City Workshop 2014
Database Deployment Pipeline - SQL In The City Workshop 2014Red Gate Software
 
The Continuous delivery value - Funaro
The Continuous delivery value - FunaroThe Continuous delivery value - Funaro
The Continuous delivery value - FunaroCodemotion
 
Security & DevOps- Ways To Make Sure Your Apps & Infrastructure Are Secure
Security & DevOps- Ways To Make Sure Your Apps & Infrastructure Are SecureSecurity & DevOps- Ways To Make Sure Your Apps & Infrastructure Are Secure
Security & DevOps- Ways To Make Sure Your Apps & Infrastructure Are SecurePuppet
 
Performance monitoring - Adoniram Mishra, Rupesh Dubey, ThoughtWorks
Performance monitoring - Adoniram Mishra, Rupesh Dubey, ThoughtWorksPerformance monitoring - Adoniram Mishra, Rupesh Dubey, ThoughtWorks
Performance monitoring - Adoniram Mishra, Rupesh Dubey, ThoughtWorksThoughtworks
 
WSO2 IoTS Device Manufacturer Guide
WSO2 IoTS Device Manufacturer GuideWSO2 IoTS Device Manufacturer Guide
WSO2 IoTS Device Manufacturer Guidehugo lu
 
Integration Testing as Validation and Monitoring
 Integration Testing as Validation and Monitoring Integration Testing as Validation and Monitoring
Integration Testing as Validation and MonitoringMelissa Benua
 
Agile Engineering Best Practices by Richard Cheng
Agile Engineering Best Practices by Richard ChengAgile Engineering Best Practices by Richard Cheng
Agile Engineering Best Practices by Richard ChengExcella
 
The Evolution of Agile - Continuous Delivery - Extending Agile out to Product...
The Evolution of Agile - Continuous Delivery - Extending Agile out to Product...The Evolution of Agile - Continuous Delivery - Extending Agile out to Product...
The Evolution of Agile - Continuous Delivery - Extending Agile out to Product...Burns Sheehan
 
Continuous integration
Continuous integrationContinuous integration
Continuous integrationBasma Alkerm
 
Bootstrapping Quality
Bootstrapping QualityBootstrapping Quality
Bootstrapping QualityMichael Roufa
 

What's hot (20)

Continuous delivery @wcap 5-09-2013
Continuous delivery   @wcap 5-09-2013Continuous delivery   @wcap 5-09-2013
Continuous delivery @wcap 5-09-2013
 
The Continuous delivery Value @ codemotion 2014
The Continuous delivery Value @ codemotion 2014The Continuous delivery Value @ codemotion 2014
The Continuous delivery Value @ codemotion 2014
 
Introduction to Continuous Integration
Introduction to Continuous IntegrationIntroduction to Continuous Integration
Introduction to Continuous Integration
 
Continuous integration
Continuous integrationContinuous integration
Continuous integration
 
Continuous Deployment of Rails Applications
Continuous Deployment of Rails ApplicationsContinuous Deployment of Rails Applications
Continuous Deployment of Rails Applications
 
Continuous integration
Continuous integrationContinuous integration
Continuous integration
 
Continuous Integration at T3CON08
Continuous Integration at T3CON08Continuous Integration at T3CON08
Continuous Integration at T3CON08
 
NYC MeetUp 10.9
NYC MeetUp 10.9NYC MeetUp 10.9
NYC MeetUp 10.9
 
Best Practices for Database Deployments
Best Practices for Database DeploymentsBest Practices for Database Deployments
Best Practices for Database Deployments
 
Database Deployment Pipeline - SQL In The City Workshop 2014
Database Deployment Pipeline - SQL In The City Workshop 2014Database Deployment Pipeline - SQL In The City Workshop 2014
Database Deployment Pipeline - SQL In The City Workshop 2014
 
The Continuous delivery value - Funaro
The Continuous delivery value - FunaroThe Continuous delivery value - Funaro
The Continuous delivery value - Funaro
 
Continuous Integration 101
Continuous Integration 101Continuous Integration 101
Continuous Integration 101
 
Security & DevOps- Ways To Make Sure Your Apps & Infrastructure Are Secure
Security & DevOps- Ways To Make Sure Your Apps & Infrastructure Are SecureSecurity & DevOps- Ways To Make Sure Your Apps & Infrastructure Are Secure
Security & DevOps- Ways To Make Sure Your Apps & Infrastructure Are Secure
 
Performance monitoring - Adoniram Mishra, Rupesh Dubey, ThoughtWorks
Performance monitoring - Adoniram Mishra, Rupesh Dubey, ThoughtWorksPerformance monitoring - Adoniram Mishra, Rupesh Dubey, ThoughtWorks
Performance monitoring - Adoniram Mishra, Rupesh Dubey, ThoughtWorks
 
WSO2 IoTS Device Manufacturer Guide
WSO2 IoTS Device Manufacturer GuideWSO2 IoTS Device Manufacturer Guide
WSO2 IoTS Device Manufacturer Guide
 
Integration Testing as Validation and Monitoring
 Integration Testing as Validation and Monitoring Integration Testing as Validation and Monitoring
Integration Testing as Validation and Monitoring
 
Agile Engineering Best Practices by Richard Cheng
Agile Engineering Best Practices by Richard ChengAgile Engineering Best Practices by Richard Cheng
Agile Engineering Best Practices by Richard Cheng
 
The Evolution of Agile - Continuous Delivery - Extending Agile out to Product...
The Evolution of Agile - Continuous Delivery - Extending Agile out to Product...The Evolution of Agile - Continuous Delivery - Extending Agile out to Product...
The Evolution of Agile - Continuous Delivery - Extending Agile out to Product...
 
Continuous integration
Continuous integrationContinuous integration
Continuous integration
 
Bootstrapping Quality
Bootstrapping QualityBootstrapping Quality
Bootstrapping Quality
 

Viewers also liked

Failing Continuous Delivery, JDays, 2015
Failing Continuous Delivery, JDays, 2015Failing Continuous Delivery, JDays, 2015
Failing Continuous Delivery, JDays, 2015Daniel Sawano
 
Continuous Delivery in the AWS Cloud
Continuous Delivery in the AWS CloudContinuous Delivery in the AWS Cloud
Continuous Delivery in the AWS CloudNigel Fernandes
 
Analysis of TLS in SMTP World
Analysis of TLS in SMTP WorldAnalysis of TLS in SMTP World
Analysis of TLS in SMTP WorldBinu Ramakrishnan
 
AppSec++ Take the best of Agile, DevOps and CI/CD into your AppSec Program
AppSec++ Take the best of Agile, DevOps and CI/CD into your AppSec ProgramAppSec++ Take the best of Agile, DevOps and CI/CD into your AppSec Program
AppSec++ Take the best of Agile, DevOps and CI/CD into your AppSec ProgramMatt Tesauro
 
Infrastructure Continuous Delivery using CloudFormation
Infrastructure Continuous Delivery using CloudFormationInfrastructure Continuous Delivery using CloudFormation
Infrastructure Continuous Delivery using CloudFormationjoehack3r
 
Deep Dive: Infrastructure as Code
Deep Dive: Infrastructure as CodeDeep Dive: Infrastructure as Code
Deep Dive: Infrastructure as CodeAmazon Web Services
 
Infrastructure as Code with AWS CloudFormation
Infrastructure as Code with AWS CloudFormationInfrastructure as Code with AWS CloudFormation
Infrastructure as Code with AWS CloudFormationJustyna Janczyszyn
 
IBM Innovate - Adoption of Continuous Delivery at Scale at a large telco v0 3
IBM Innovate - Adoption of Continuous Delivery at Scale at a large telco v0 3IBM Innovate - Adoption of Continuous Delivery at Scale at a large telco v0 3
IBM Innovate - Adoption of Continuous Delivery at Scale at a large telco v0 3Mirco Hering
 
Jenkins CI + XebiaLabs for Release Orchestration: A Recipe for Continuous Del...
Jenkins CI + XebiaLabs for Release Orchestration: A Recipe for Continuous Del...Jenkins CI + XebiaLabs for Release Orchestration: A Recipe for Continuous Del...
Jenkins CI + XebiaLabs for Release Orchestration: A Recipe for Continuous Del...XebiaLabs
 
CI&CD on AWS - Meetup Roma Oct 2016
CI&CD on AWS - Meetup Roma Oct 2016CI&CD on AWS - Meetup Roma Oct 2016
CI&CD on AWS - Meetup Roma Oct 2016Paolo latella
 
Continuous Deployment: The Dirty Details
Continuous Deployment: The Dirty DetailsContinuous Deployment: The Dirty Details
Continuous Deployment: The Dirty DetailsMike Brittain
 
The Journey of devops and continuous delivery in a Large Financial Institution
The Journey of devops and continuous delivery in a Large Financial InstitutionThe Journey of devops and continuous delivery in a Large Financial Institution
The Journey of devops and continuous delivery in a Large Financial InstitutionKris Buytaert
 
Securing application deployments in multi-tenant CI/CD environments
Securing application deployments in multi-tenant CI/CD environmentsSecuring application deployments in multi-tenant CI/CD environments
Securing application deployments in multi-tenant CI/CD environmentsBinu Ramakrishnan
 
Continuous Integration, Build Pipelines and Continuous Deployment
Continuous Integration, Build Pipelines and Continuous DeploymentContinuous Integration, Build Pipelines and Continuous Deployment
Continuous Integration, Build Pipelines and Continuous DeploymentChristopher Read
 
O'Reilly/Nginx 2016: "Continuous Delivery with Containers: The Trials and Tri...
O'Reilly/Nginx 2016: "Continuous Delivery with Containers: The Trials and Tri...O'Reilly/Nginx 2016: "Continuous Delivery with Containers: The Trials and Tri...
O'Reilly/Nginx 2016: "Continuous Delivery with Containers: The Trials and Tri...Daniel Bryant
 
AWS May Webinar Series - Deep Dive: Infrastructure as Code
AWS May Webinar Series - Deep Dive: Infrastructure as CodeAWS May Webinar Series - Deep Dive: Infrastructure as Code
AWS May Webinar Series - Deep Dive: Infrastructure as CodeAmazon Web Services
 
Infrastructure Continuous Delivery Using AWS CloudFormation
Infrastructure Continuous Delivery Using AWS CloudFormationInfrastructure Continuous Delivery Using AWS CloudFormation
Infrastructure Continuous Delivery Using AWS CloudFormationAmazon Web Services
 
Enterprise DevOps at Scale with AWS | AWS Public Sector Summit 2016
Enterprise DevOps at Scale with AWS | AWS Public Sector Summit 2016Enterprise DevOps at Scale with AWS | AWS Public Sector Summit 2016
Enterprise DevOps at Scale with AWS | AWS Public Sector Summit 2016Amazon Web Services
 
Pitfalls of Continuous Deployment
Pitfalls of Continuous DeploymentPitfalls of Continuous Deployment
Pitfalls of Continuous Deploymentzeeg
 

Viewers also liked (20)

Failing Continuous Delivery, JDays, 2015
Failing Continuous Delivery, JDays, 2015Failing Continuous Delivery, JDays, 2015
Failing Continuous Delivery, JDays, 2015
 
Continuous Delivery in the AWS Cloud
Continuous Delivery in the AWS CloudContinuous Delivery in the AWS Cloud
Continuous Delivery in the AWS Cloud
 
Analysis of TLS in SMTP World
Analysis of TLS in SMTP WorldAnalysis of TLS in SMTP World
Analysis of TLS in SMTP World
 
AppSec++ Take the best of Agile, DevOps and CI/CD into your AppSec Program
AppSec++ Take the best of Agile, DevOps and CI/CD into your AppSec ProgramAppSec++ Take the best of Agile, DevOps and CI/CD into your AppSec Program
AppSec++ Take the best of Agile, DevOps and CI/CD into your AppSec Program
 
Infrastructure Continuous Delivery using CloudFormation
Infrastructure Continuous Delivery using CloudFormationInfrastructure Continuous Delivery using CloudFormation
Infrastructure Continuous Delivery using CloudFormation
 
Deep Dive: Infrastructure as Code
Deep Dive: Infrastructure as CodeDeep Dive: Infrastructure as Code
Deep Dive: Infrastructure as Code
 
Infrastructure as Code with AWS CloudFormation
Infrastructure as Code with AWS CloudFormationInfrastructure as Code with AWS CloudFormation
Infrastructure as Code with AWS CloudFormation
 
IBM Innovate - Adoption of Continuous Delivery at Scale at a large telco v0 3
IBM Innovate - Adoption of Continuous Delivery at Scale at a large telco v0 3IBM Innovate - Adoption of Continuous Delivery at Scale at a large telco v0 3
IBM Innovate - Adoption of Continuous Delivery at Scale at a large telco v0 3
 
Dod is not done
Dod is not doneDod is not done
Dod is not done
 
Jenkins CI + XebiaLabs for Release Orchestration: A Recipe for Continuous Del...
Jenkins CI + XebiaLabs for Release Orchestration: A Recipe for Continuous Del...Jenkins CI + XebiaLabs for Release Orchestration: A Recipe for Continuous Del...
Jenkins CI + XebiaLabs for Release Orchestration: A Recipe for Continuous Del...
 
CI&CD on AWS - Meetup Roma Oct 2016
CI&CD on AWS - Meetup Roma Oct 2016CI&CD on AWS - Meetup Roma Oct 2016
CI&CD on AWS - Meetup Roma Oct 2016
 
Continuous Deployment: The Dirty Details
Continuous Deployment: The Dirty DetailsContinuous Deployment: The Dirty Details
Continuous Deployment: The Dirty Details
 
The Journey of devops and continuous delivery in a Large Financial Institution
The Journey of devops and continuous delivery in a Large Financial InstitutionThe Journey of devops and continuous delivery in a Large Financial Institution
The Journey of devops and continuous delivery in a Large Financial Institution
 
Securing application deployments in multi-tenant CI/CD environments
Securing application deployments in multi-tenant CI/CD environmentsSecuring application deployments in multi-tenant CI/CD environments
Securing application deployments in multi-tenant CI/CD environments
 
Continuous Integration, Build Pipelines and Continuous Deployment
Continuous Integration, Build Pipelines and Continuous DeploymentContinuous Integration, Build Pipelines and Continuous Deployment
Continuous Integration, Build Pipelines and Continuous Deployment
 
O'Reilly/Nginx 2016: "Continuous Delivery with Containers: The Trials and Tri...
O'Reilly/Nginx 2016: "Continuous Delivery with Containers: The Trials and Tri...O'Reilly/Nginx 2016: "Continuous Delivery with Containers: The Trials and Tri...
O'Reilly/Nginx 2016: "Continuous Delivery with Containers: The Trials and Tri...
 
AWS May Webinar Series - Deep Dive: Infrastructure as Code
AWS May Webinar Series - Deep Dive: Infrastructure as CodeAWS May Webinar Series - Deep Dive: Infrastructure as Code
AWS May Webinar Series - Deep Dive: Infrastructure as Code
 
Infrastructure Continuous Delivery Using AWS CloudFormation
Infrastructure Continuous Delivery Using AWS CloudFormationInfrastructure Continuous Delivery Using AWS CloudFormation
Infrastructure Continuous Delivery Using AWS CloudFormation
 
Enterprise DevOps at Scale with AWS | AWS Public Sector Summit 2016
Enterprise DevOps at Scale with AWS | AWS Public Sector Summit 2016Enterprise DevOps at Scale with AWS | AWS Public Sector Summit 2016
Enterprise DevOps at Scale with AWS | AWS Public Sector Summit 2016
 
Pitfalls of Continuous Deployment
Pitfalls of Continuous DeploymentPitfalls of Continuous Deployment
Pitfalls of Continuous Deployment
 

Similar to The Hard Problems of Continuous Deployment

Road to Continuous Delivery - Wix.com
Road to Continuous Delivery - Wix.comRoad to Continuous Delivery - Wix.com
Road to Continuous Delivery - Wix.comAviran Mordo
 
The Rocky Cloud Road
The Rocky Cloud RoadThe Rocky Cloud Road
The Rocky Cloud RoadGert Drapers
 
2016 09-dev opsjourney-devopsdaysoslo
2016 09-dev opsjourney-devopsdaysoslo2016 09-dev opsjourney-devopsdaysoslo
2016 09-dev opsjourney-devopsdaysosloJon Arild Tørresdal
 
Make It Cooler: Using Decentralized Version Control
Make It Cooler: Using Decentralized Version ControlMake It Cooler: Using Decentralized Version Control
Make It Cooler: Using Decentralized Version Controlindiver
 
DevOps / Agile Tools Seminar 2013
DevOps / Agile Tools Seminar 2013DevOps / Agile Tools Seminar 2013
DevOps / Agile Tools Seminar 2013Ethan Ram
 
Lessons Learned Replatforming A Large Machine Learning Application To Apache ...
Lessons Learned Replatforming A Large Machine Learning Application To Apache ...Lessons Learned Replatforming A Large Machine Learning Application To Apache ...
Lessons Learned Replatforming A Large Machine Learning Application To Apache ...Databricks
 
Introduction to continuous delivery
Introduction to continuous deliveryIntroduction to continuous delivery
Introduction to continuous deliveryOlympicSoftware
 
SynapseIndia drupal presentation on drupal info
SynapseIndia drupal  presentation on drupal infoSynapseIndia drupal  presentation on drupal info
SynapseIndia drupal presentation on drupal infoSynapseindiappsdevelopment
 
Test Driven Development & CI/CD
Test Driven Development & CI/CDTest Driven Development & CI/CD
Test Driven Development & CI/CDShanmuga S Muthu
 
How to Build a Compute Cluster
How to Build a Compute ClusterHow to Build a Compute Cluster
How to Build a Compute ClusterRamsay Key
 
Denver devops : enabling DevOps with data virtualization
Denver devops : enabling DevOps with data virtualizationDenver devops : enabling DevOps with data virtualization
Denver devops : enabling DevOps with data virtualizationKyle Hailey
 
E2 evc 3-2-1-rule - mikeresseler
E2 evc   3-2-1-rule - mikeresselerE2 evc   3-2-1-rule - mikeresseler
E2 evc 3-2-1-rule - mikeresselerMike Resseler
 
Getting to Walk with DevOps
Getting to Walk with DevOpsGetting to Walk with DevOps
Getting to Walk with DevOpsEklove Mohan
 
Microservices: The Best Practices
Microservices: The Best PracticesMicroservices: The Best Practices
Microservices: The Best PracticesPavel Mička
 
Towards Continuous Delivery in Mobile Apps
Towards Continuous Delivery in Mobile AppsTowards Continuous Delivery in Mobile Apps
Towards Continuous Delivery in Mobile AppsPhillip Wheatley
 
Hpc lunch and learn
Hpc lunch and learnHpc lunch and learn
Hpc lunch and learnJohn D Almon
 
Sailing through devlopment with legacy code
Sailing through devlopment with legacy codeSailing through devlopment with legacy code
Sailing through devlopment with legacy codeprasadkunte
 
Why retail companies can't afford database downtime
Why retail companies can't afford database downtimeWhy retail companies can't afford database downtime
Why retail companies can't afford database downtimeDBmaestro - Database DevOps
 
Mixing d ps building architecture on the cross cutting example
Mixing d ps building architecture on the cross cutting exampleMixing d ps building architecture on the cross cutting example
Mixing d ps building architecture on the cross cutting examplecorehard_by
 

Similar to The Hard Problems of Continuous Deployment (20)

Road to Continuous Delivery - Wix.com
Road to Continuous Delivery - Wix.comRoad to Continuous Delivery - Wix.com
Road to Continuous Delivery - Wix.com
 
The Rocky Cloud Road
The Rocky Cloud RoadThe Rocky Cloud Road
The Rocky Cloud Road
 
2016 09-dev opsjourney-devopsdaysoslo
2016 09-dev opsjourney-devopsdaysoslo2016 09-dev opsjourney-devopsdaysoslo
2016 09-dev opsjourney-devopsdaysoslo
 
Make It Cooler: Using Decentralized Version Control
Make It Cooler: Using Decentralized Version ControlMake It Cooler: Using Decentralized Version Control
Make It Cooler: Using Decentralized Version Control
 
Extreme Makeover OnBase Edition
Extreme Makeover OnBase EditionExtreme Makeover OnBase Edition
Extreme Makeover OnBase Edition
 
DevOps / Agile Tools Seminar 2013
DevOps / Agile Tools Seminar 2013DevOps / Agile Tools Seminar 2013
DevOps / Agile Tools Seminar 2013
 
Lessons Learned Replatforming A Large Machine Learning Application To Apache ...
Lessons Learned Replatforming A Large Machine Learning Application To Apache ...Lessons Learned Replatforming A Large Machine Learning Application To Apache ...
Lessons Learned Replatforming A Large Machine Learning Application To Apache ...
 
Introduction to continuous delivery
Introduction to continuous deliveryIntroduction to continuous delivery
Introduction to continuous delivery
 
SynapseIndia drupal presentation on drupal info
SynapseIndia drupal  presentation on drupal infoSynapseIndia drupal  presentation on drupal info
SynapseIndia drupal presentation on drupal info
 
Test Driven Development & CI/CD
Test Driven Development & CI/CDTest Driven Development & CI/CD
Test Driven Development & CI/CD
 
How to Build a Compute Cluster
How to Build a Compute ClusterHow to Build a Compute Cluster
How to Build a Compute Cluster
 
Denver devops : enabling DevOps with data virtualization
Denver devops : enabling DevOps with data virtualizationDenver devops : enabling DevOps with data virtualization
Denver devops : enabling DevOps with data virtualization
 
E2 evc 3-2-1-rule - mikeresseler
E2 evc   3-2-1-rule - mikeresselerE2 evc   3-2-1-rule - mikeresseler
E2 evc 3-2-1-rule - mikeresseler
 
Getting to Walk with DevOps
Getting to Walk with DevOpsGetting to Walk with DevOps
Getting to Walk with DevOps
 
Microservices: The Best Practices
Microservices: The Best PracticesMicroservices: The Best Practices
Microservices: The Best Practices
 
Towards Continuous Delivery in Mobile Apps
Towards Continuous Delivery in Mobile AppsTowards Continuous Delivery in Mobile Apps
Towards Continuous Delivery in Mobile Apps
 
Hpc lunch and learn
Hpc lunch and learnHpc lunch and learn
Hpc lunch and learn
 
Sailing through devlopment with legacy code
Sailing through devlopment with legacy codeSailing through devlopment with legacy code
Sailing through devlopment with legacy code
 
Why retail companies can't afford database downtime
Why retail companies can't afford database downtimeWhy retail companies can't afford database downtime
Why retail companies can't afford database downtime
 
Mixing d ps building architecture on the cross cutting example
Mixing d ps building architecture on the cross cutting exampleMixing d ps building architecture on the cross cutting example
Mixing d ps building architecture on the cross cutting example
 

More from Timothy Fitz

Good hypothesis testing is surprising!
Good hypothesis testing is surprising!Good hypothesis testing is surprising!
Good hypothesis testing is surprising!Timothy Fitz
 
Gdc 2010 architecture final slideshare edition
Gdc 2010 architecture final slideshare editionGdc 2010 architecture final slideshare edition
Gdc 2010 architecture final slideshare editionTimothy Fitz
 
Continuous Deployment
Continuous DeploymentContinuous Deployment
Continuous DeploymentTimothy Fitz
 
Just In Time Scalability Agile Methods To Support Massive Growth Presentation
Just In Time Scalability  Agile Methods To Support Massive Growth PresentationJust In Time Scalability  Agile Methods To Support Massive Growth Presentation
Just In Time Scalability Agile Methods To Support Massive Growth PresentationTimothy Fitz
 

More from Timothy Fitz (8)

Good hypothesis testing is surprising!
Good hypothesis testing is surprising!Good hypothesis testing is surprising!
Good hypothesis testing is surprising!
 
Gdc 2010 architecture final slideshare edition
Gdc 2010 architecture final slideshare editionGdc 2010 architecture final slideshare edition
Gdc 2010 architecture final slideshare edition
 
Realtime web2012
Realtime web2012Realtime web2012
Realtime web2012
 
Continuous Deployment
Continuous DeploymentContinuous Deployment
Continuous Deployment
 
Shdh
ShdhShdh
Shdh
 
Socket.io
Socket.ioSocket.io
Socket.io
 
Shdh
ShdhShdh
Shdh
 
Just In Time Scalability Agile Methods To Support Massive Growth Presentation
Just In Time Scalability  Agile Methods To Support Massive Growth PresentationJust In Time Scalability  Agile Methods To Support Massive Growth Presentation
Just In Time Scalability Agile Methods To Support Massive Growth Presentation
 

Recently uploaded

Patch notes explaining DISARM Version 1.4 update
Patch notes explaining DISARM Version 1.4 updatePatch notes explaining DISARM Version 1.4 update
Patch notes explaining DISARM Version 1.4 updateadam112203
 
Automation Ops Series: Session 2 - Governance for UiPath projects
Automation Ops Series: Session 2 - Governance for UiPath projectsAutomation Ops Series: Session 2 - Governance for UiPath projects
Automation Ops Series: Session 2 - Governance for UiPath projectsDianaGray10
 
UiPath Studio Web workshop series - Day 4
UiPath Studio Web workshop series - Day 4UiPath Studio Web workshop series - Day 4
UiPath Studio Web workshop series - Day 4DianaGray10
 
The New Cloud World Order Is FinOps (Slideshow)
The New Cloud World Order Is FinOps (Slideshow)The New Cloud World Order Is FinOps (Slideshow)
The New Cloud World Order Is FinOps (Slideshow)codyslingerland1
 
The Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and InsightThe Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and InsightSafe Software
 
Flow Control | Block Size | ST Min | First Frame
Flow Control | Block Size | ST Min | First FrameFlow Control | Block Size | ST Min | First Frame
Flow Control | Block Size | ST Min | First FrameKapil Thakar
 
March Patch Tuesday
March Patch TuesdayMarch Patch Tuesday
March Patch TuesdayIvanti
 
TrustArc Webinar - How to Live in a Post Third-Party Cookie World
TrustArc Webinar - How to Live in a Post Third-Party Cookie WorldTrustArc Webinar - How to Live in a Post Third-Party Cookie World
TrustArc Webinar - How to Live in a Post Third-Party Cookie WorldTrustArc
 
.NET 8 ChatBot with Azure OpenAI Services.pptx
.NET 8 ChatBot with Azure OpenAI Services.pptx.NET 8 ChatBot with Azure OpenAI Services.pptx
.NET 8 ChatBot with Azure OpenAI Services.pptxHansamali Gamage
 
Q4 2023 Quarterly Investor Presentation - FINAL - v1.pdf
Q4 2023 Quarterly Investor Presentation - FINAL - v1.pdfQ4 2023 Quarterly Investor Presentation - FINAL - v1.pdf
Q4 2023 Quarterly Investor Presentation - FINAL - v1.pdfTejal81
 
How to release an Open Source Dataweave Library
How to release an Open Source Dataweave LibraryHow to release an Open Source Dataweave Library
How to release an Open Source Dataweave Libraryshyamraj55
 
Technical SEO for Improved Accessibility WTS FEST
Technical SEO for Improved Accessibility  WTS FESTTechnical SEO for Improved Accessibility  WTS FEST
Technical SEO for Improved Accessibility WTS FESTBillieHyde
 
My key hands-on projects in Quantum, and QAI
My key hands-on projects in Quantum, and QAIMy key hands-on projects in Quantum, and QAI
My key hands-on projects in Quantum, and QAIVijayananda Mohire
 
Outage Analysis: March 5th/6th 2024 Meta, Comcast, and LinkedIn
Outage Analysis: March 5th/6th 2024 Meta, Comcast, and LinkedInOutage Analysis: March 5th/6th 2024 Meta, Comcast, and LinkedIn
Outage Analysis: March 5th/6th 2024 Meta, Comcast, and LinkedInThousandEyes
 
20140402 - Smart house demo kit
20140402 - Smart house demo kit20140402 - Smart house demo kit
20140402 - Smart house demo kitJamie (Taka) Wang
 
Planetek Italia Srl - Corporate Profile Brochure
Planetek Italia Srl - Corporate Profile BrochurePlanetek Italia Srl - Corporate Profile Brochure
Planetek Italia Srl - Corporate Profile BrochurePlanetek Italia Srl
 
Top 10 Squarespace Development Companies
Top 10 Squarespace Development CompaniesTop 10 Squarespace Development Companies
Top 10 Squarespace Development CompaniesTopCSSGallery
 
Key Trends Shaping the Future of Infrastructure.pdf
Key Trends Shaping the Future of Infrastructure.pdfKey Trends Shaping the Future of Infrastructure.pdf
Key Trends Shaping the Future of Infrastructure.pdfCheryl Hung
 
IT Service Management (ITSM) Best Practices for Advanced Computing
IT Service Management (ITSM) Best Practices for Advanced ComputingIT Service Management (ITSM) Best Practices for Advanced Computing
IT Service Management (ITSM) Best Practices for Advanced ComputingMAGNIntelligence
 

Recently uploaded (20)

Patch notes explaining DISARM Version 1.4 update
Patch notes explaining DISARM Version 1.4 updatePatch notes explaining DISARM Version 1.4 update
Patch notes explaining DISARM Version 1.4 update
 
Automation Ops Series: Session 2 - Governance for UiPath projects
Automation Ops Series: Session 2 - Governance for UiPath projectsAutomation Ops Series: Session 2 - Governance for UiPath projects
Automation Ops Series: Session 2 - Governance for UiPath projects
 
UiPath Studio Web workshop series - Day 4
UiPath Studio Web workshop series - Day 4UiPath Studio Web workshop series - Day 4
UiPath Studio Web workshop series - Day 4
 
The New Cloud World Order Is FinOps (Slideshow)
The New Cloud World Order Is FinOps (Slideshow)The New Cloud World Order Is FinOps (Slideshow)
The New Cloud World Order Is FinOps (Slideshow)
 
The Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and InsightThe Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and Insight
 
Flow Control | Block Size | ST Min | First Frame
Flow Control | Block Size | ST Min | First FrameFlow Control | Block Size | ST Min | First Frame
Flow Control | Block Size | ST Min | First Frame
 
March Patch Tuesday
March Patch TuesdayMarch Patch Tuesday
March Patch Tuesday
 
TrustArc Webinar - How to Live in a Post Third-Party Cookie World
TrustArc Webinar - How to Live in a Post Third-Party Cookie WorldTrustArc Webinar - How to Live in a Post Third-Party Cookie World
TrustArc Webinar - How to Live in a Post Third-Party Cookie World
 
.NET 8 ChatBot with Azure OpenAI Services.pptx
.NET 8 ChatBot with Azure OpenAI Services.pptx.NET 8 ChatBot with Azure OpenAI Services.pptx
.NET 8 ChatBot with Azure OpenAI Services.pptx
 
Q4 2023 Quarterly Investor Presentation - FINAL - v1.pdf
Q4 2023 Quarterly Investor Presentation - FINAL - v1.pdfQ4 2023 Quarterly Investor Presentation - FINAL - v1.pdf
Q4 2023 Quarterly Investor Presentation - FINAL - v1.pdf
 
How to release an Open Source Dataweave Library
How to release an Open Source Dataweave LibraryHow to release an Open Source Dataweave Library
How to release an Open Source Dataweave Library
 
Technical SEO for Improved Accessibility WTS FEST
Technical SEO for Improved Accessibility  WTS FESTTechnical SEO for Improved Accessibility  WTS FEST
Technical SEO for Improved Accessibility WTS FEST
 
My key hands-on projects in Quantum, and QAI
My key hands-on projects in Quantum, and QAIMy key hands-on projects in Quantum, and QAI
My key hands-on projects in Quantum, and QAI
 
Outage Analysis: March 5th/6th 2024 Meta, Comcast, and LinkedIn
Outage Analysis: March 5th/6th 2024 Meta, Comcast, and LinkedInOutage Analysis: March 5th/6th 2024 Meta, Comcast, and LinkedIn
Outage Analysis: March 5th/6th 2024 Meta, Comcast, and LinkedIn
 
SheDev 2024
SheDev 2024SheDev 2024
SheDev 2024
 
20140402 - Smart house demo kit
20140402 - Smart house demo kit20140402 - Smart house demo kit
20140402 - Smart house demo kit
 
Planetek Italia Srl - Corporate Profile Brochure
Planetek Italia Srl - Corporate Profile BrochurePlanetek Italia Srl - Corporate Profile Brochure
Planetek Italia Srl - Corporate Profile Brochure
 
Top 10 Squarespace Development Companies
Top 10 Squarespace Development CompaniesTop 10 Squarespace Development Companies
Top 10 Squarespace Development Companies
 
Key Trends Shaping the Future of Infrastructure.pdf
Key Trends Shaping the Future of Infrastructure.pdfKey Trends Shaping the Future of Infrastructure.pdf
Key Trends Shaping the Future of Infrastructure.pdf
 
IT Service Management (ITSM) Best Practices for Advanced Computing
IT Service Management (ITSM) Best Practices for Advanced ComputingIT Service Management (ITSM) Best Practices for Advanced Computing
IT Service Management (ITSM) Best Practices for Advanced Computing
 

The Hard Problems of Continuous Deployment

  • 1. The Hard Problems of Continuous Deployment Timothy Fitz (.com)
  • 2. “Continuous deployment involves deploying early and often, so as to avoid the pitfalls of "deployment hell". The practice aims to reduce timely rework and thus reduce cost and development time.”
  • 3. “Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.” - Principles behind the Agile Manifesto (2001)
  • 4. The Hard Problems (The ones I’m going to cover) • Data • Mobile • Scale
  • 5. Data makes CD harder • Updating data is slow • Moving data is slow • Schemas live outside code deploy process • Rollbacks are often hard or impossible
  • 6. Schema rev. N, N+1 compatible • Add the column in production • Push the code that writes to that column • Optionally, run a data migration to populate the existing rows with data • Push the code that reads from that column.
  • 7. Apply only cheap changes • Only apply changes that are cheap enough to not affect live traffic • More complex changes split into tiny steps: – Create new table – Write to both – Cut over eventually – Drop old table
  • 8. Apply change to standby • Run two DB instances • Apply change to standby • Failover if successfully applied • Might run 3rd db instance for availability
  • 9. Blue/Green Deploy • Run two copies of entire cluster • All databases are replicated • Lets you test, update and rollback both code and schema in one step
  • 10. Schemaless NoSQL • “Schemaless” really means schema is in your source code – Which is great! We can CD schema changes
  • 11. Schemaless-style SQL • Benefits of existing SQL databases while still being pseudoschemaless • (thing_id, key, value) table • (guid, blob) table
  • 12. Recap • Apply only cheap changes • Apply changes to standby • Blue/Green Deploy • Schemaless NoSQL • Schemaless-style SQL
  • 13. Mobile • Users must opt-in to every update • iOS submissions take a week to be approved • Luckily, lots of tools aimed at the space
  • 14. Remember the basics • CI server / automated tests are critical • Can’t fallback on production alarms / rollback • Hosted options: http://cisimple.com
  • 15. Data-drive everything • Build your views / content from data files • Ping server for updates • Hosted: http://appgrok.com/ – Lets you deploy txt, png and xib dynamically!
  • 16. 99% HTML • Entire app is a single UIWebView • Glue native code to allow access to APIs • Clutch.io is awesome (and FOSS now) – Live reloading for local dev – Streamlined deploys – https://github.com/clutchio
  • 17. Hybrid HTML/Native • Core app is native • Sections can be replaced by HTML – i.e. Facebook stream entries fallback to HTML • Infrequently used sections are 100% HTML
  • 18. Recap • Remember the basics • Data-drive everything • 99% HTML • Hybrid HTML/Native
  • 19. Scalability • Maintaining availability, performance and happiness • As a function of # of people • As a function of # of tests
  • 20. Availability • The build must always be green • Set a “green SLA” – 99% green – Never red for > 15m • Measure, track and report on these numbers
  • 21. Performance • Measure (intent to) commit to time of deploy – Goal: < 5 minutes • Measure local development test loop – Goal: < 2s
  • 22. Happiness • CI/CD System is a product • Software Engineers are the customer • Keep your customer happy!
  • 24. How do you make tests fast? • Tests can exercise large amounts of code without being slow • Minimize system calls (no I/O, no disk) • Minimize test data size • Make sure all systems are cheap to instantiate/teardown • No external state makes tests more reliable
  • 25. Run Tests in Parallel • Multiprocess • Multimachine • Multi-VM • Instant multi-VM: http://circleci.com
  • 26. Hardware Scale • CI Cluster will get huge – Function of cumulative engineering man-months – Rule of thumb: 10% of your cluster size • You will need a CI/CD DevOps person – CI cluster monitoring / alerting – Configuration Management critical
  • 27. Scale testing infrastructure recap • Write the right kind of tests • Make those tests as fast as possible • Run those tests in parallel
  • 28. People / Roles • Sheriff – Designated reverter / problem troubleshooter – Common pattern (IMVU, Chromium, Firefox) • CD “Product Owner” – Held accountable for SLA / Performance – Manage infrastructure backlog
  • 29. Single trunk • Do this until it doesn’t work for you • Gets painful in the 16 – 32 developer range • Faster commit->deploy reduces the pain – But effort becomes prohibitive
  • 30. “Try” pipeline • Conceptually, a second tree that “doesn’t matter” but still gets tested for feedback • Buildbot implements a patch-pushing version • Takes a significant amount of pressure off of trunk builds
  • 31. CI Server takes active role • Server automatically reverts red commits • Server merges green commits to trunk
  • 32. Feature branches • All incremental development happens on branches, branches land when feature is “ready” • If “feature” is kept small, can be 2-3 per engineer per week on average • Less continuous, but scales much better – Feature branches tested before merge
  • 33. Merge tree • Tree per team / feature • Trees merged into trunk daily (if green) • Scale up via tree of trees (of trees…) • Again, less continuous
  • 34. Federation • Each team gets their own deploy pipeline • Requires SOA / component architecture • Each team can set their own CD pace • “Enterprise Ready”
  • 35. Recap • Single trunk + Try pipeline / Autorevert • Feature Branches • Merge Tree • Federation

Editor's Notes

  1. About me: IMVU, Canvas, Continuous Deployment)Ground rules: I don’t demand your attention, please tweet / follow links while I’m talking. If you have questions, shoot up a hand. If I don’t see you, yell at me.
  2. ContinousDeployment vs Continuous Delivery (next slide)
  3. Very similarContinuous Delivery is “agile as it should be”Continuous Delivery does NOT advocate for deploy-on-commit (Book and term reaction to Continuous Deployment)Continuous Deployment advocates for MORE THAN deploy-on-commit. (August 2007, 5 years old!)At the end of the day, I don’t care. People seem to be using both terms interchangeably. Whatever you call it, however far you take it, it’s better than not deploying very often.
  4. These are the problems I get asked about all the time.No silver bullets.
  5. In general I assume some sort of LAMP stack.
  6. Seems straightforward, but often means what would’ve been a 1 step process (with downtime) is now a 15 step process. BUT! If you follow it well, you can always step backwards, one small step at a time, in the event that it doesn’t work out / is wrong
  7. Sharding your data and keeping your shards balanced (and measured) means you can quantitatively assess this, especially if you keep an offline copy of a single shard.i.e. keep 100k users per shard, letting you do almost anything to any O(user) table.
  8. Not hitting on master-master / master-standby / etc distinctions, because the exact setup depends on your db / replication / required availability.
  9. http://martinfowler.com/bliki/BlueGreenDeployment.html
  10. Hybrid SQL db and NoSQLdb, use each where appropriate
  11. Most real world scenarios are a hybrid
  12. Experience heavily biased to iOS, would love to hear Android perspective
  13. Don’t just throw up your arms and say “CD can’t be done”Apparently cozying up with Apple can lower your submission turnaround time from 1 week to 1 day.
  14. Trades development speed for performanceFor most apps that’s probably the right trade (users don’t really notice)
  15. Get back the performance you were missing, and native scrollingLose some of the dynamic propertiesMuch more expensive to develop
  16. Commit to deploy&lt; 5 minute: stay in flow5 – 15 minutes: can keep working on feature15 minutes: failures are surprises and require expensive rewindingLocal dev loop&lt; 2s: stay in tight flow2s – 10s: tab away from terminal, looser flow10s – 1min: start thinking or coding on next thing, failures require rewinding1m – 5min: significant rewinding, high distraction, painful testing (rolling chair jousting)
  17. Example: All metrics green, everything looks great, but got to metrics by shaming anyone who breaks build. Culture of tip-toeing through the bulid system leads to reduced happiness (and reduced throughput!)Sidenote: Measure throughput! Deploys per engineer (avg/median/extremes), is it scaling up with org or are you getting less deploys?
  18. GUI Testsi.e. SeleniumIntegration Testsi.e. uses the databaseUnit Testssmall fast tests with no external state
  19. i.e. Don’t reload your country code or ipgeo tables, even into memory from disk, on every test
  20. Highly recommend Buildbot when you hit scale. It’s proven at huge scale (500+ nodes) and growing, and allows way better pipeline customization at that scale
  21. Test time is O(Cumulative man-months)Doubling staff means halving test time, in spite of ever-marching-on increase in tests
  22. Github modelMost FOSS is moving to or using this model
  23. Lots of people/process overheadClassical way of scaling up to extremely large teams
  24. Fortune 500-proof.
  25. Remember: Don’t overinvest up front. Better to do something simple until it doesn’t work, than to overbuild.Don’t underbuild either. Use