SlideShare a Scribd company logo
The Hard Problems
of Continuous Deployment
     Timothy Fitz (.com)
“Continuous deployment involves
  deploying early and often, so as to
avoid the pitfalls of "deployment hell".
The practice aims to reduce timely rework and thus
       reduce cost and development time.”
“Our highest priority is to satisfy
  the customer through early
  and continuous delivery of
      valuable software.”
- Principles behind the Agile Manifesto (2001)
The Hard Problems
           (The ones I’m going to cover)
• Data
• Mobile
• Scale
Data makes CD harder
•   Updating data is slow
•   Moving data is slow
•   Schemas live outside code deploy process
•   Rollbacks are often hard or impossible
Schema rev. N, N+1 compatible
• Add the column in production
• Push the code that writes to that column
• Optionally, run a data migration to populate
  the existing rows with data
• Push the code that reads from that column.
Apply only cheap changes
• Only apply changes that are cheap enough to
  not affect live traffic
• More complex changes split into tiny steps:
  – Create new table
  – Write to both
  – Cut over eventually
  – Drop old table
Apply change to standby
•   Run two DB instances
•   Apply change to standby
•   Failover if successfully applied
•   Might run 3rd db instance for availability
Blue/Green Deploy
• Run two copies of entire cluster
• All databases are replicated
• Lets you test, update and rollback both code
  and schema in one step
Schemaless NoSQL
• “Schemaless” really means schema is in your
  source code
  – Which is great! We can CD schema changes
Schemaless-style SQL
• Benefits of existing SQL databases while still
  being pseudoschemaless
• (thing_id, key, value) table
• (guid, blob) table
Recap
•   Apply only cheap changes
•   Apply changes to standby
•   Blue/Green Deploy
•   Schemaless NoSQL
•   Schemaless-style SQL
Mobile
• Users must opt-in to every update
• iOS submissions take a week to be approved
• Luckily, lots of tools aimed at the space
Remember the basics
• CI server / automated tests are critical
• Can’t fallback on production alarms / rollback
• Hosted options: http://cisimple.com
Data-drive everything
• Build your views / content from data files
• Ping server for updates
• Hosted: http://appgrok.com/
  – Lets you deploy txt, png and xib dynamically!
99% HTML
• Entire app is a single UIWebView
• Glue native code to allow access to APIs
• Clutch.io is awesome (and FOSS now)
  – Live reloading for local dev
  – Streamlined deploys
  – https://github.com/clutchio
Hybrid HTML/Native
• Core app is native
• Sections can be replaced by HTML
  – i.e. Facebook stream entries fallback to HTML
• Infrequently used sections are 100% HTML
Recap
•   Remember the basics
•   Data-drive everything
•   99% HTML
•   Hybrid HTML/Native
Scalability
• Maintaining availability, performance and
  happiness
• As a function of # of people
• As a function of # of tests
Availability
• The build must always be green
• Set a “green SLA”
  – 99% green
  – Never red for > 15m
• Measure, track and report on these numbers
Performance
• Measure (intent to) commit to time of deploy
  – Goal: < 5 minutes
• Measure local development test loop
  – Goal: < 2s
Happiness
• CI/CD System is a product
• Software Engineers are the customer
• Keep your customer happy!
Testing Pyramid
How do you make tests fast?
• Tests can exercise large amounts of code
  without being slow
• Minimize system calls (no I/O, no disk)
• Minimize test data size
• Make sure all systems are cheap to
  instantiate/teardown
• No external state makes tests more reliable
Run Tests in Parallel
•   Multiprocess
•   Multimachine
•   Multi-VM
•   Instant multi-VM: http://circleci.com
Hardware Scale
• CI Cluster will get huge
  – Function of cumulative engineering man-months
  – Rule of thumb: 10% of your cluster size
• You will need a CI/CD DevOps person
  – CI cluster monitoring / alerting
  – Configuration Management critical
Scale testing infrastructure recap
• Write the right kind of tests
• Make those tests as fast as possible
• Run those tests in parallel
People / Roles
• Sheriff
  – Designated reverter / problem troubleshooter
  – Common pattern (IMVU, Chromium, Firefox)
• CD “Product Owner”
  – Held accountable for SLA / Performance
  – Manage infrastructure backlog
Single trunk
• Do this until it doesn’t work for you
• Gets painful in the 16 – 32 developer range
• Faster commit->deploy reduces the pain
  – But effort becomes prohibitive
“Try” pipeline
• Conceptually, a second tree that “doesn’t
  matter” but still gets tested for feedback
• Buildbot implements a patch-pushing version
• Takes a significant amount of pressure off of
  trunk builds
CI Server takes active role
• Server automatically reverts red commits
• Server merges green commits to trunk
Feature branches
• All incremental development happens on
  branches, branches land when feature is
  “ready”
• If “feature” is kept small, can be 2-3 per
  engineer per week on average
• Less continuous, but scales much better
  – Feature branches tested before merge
Merge tree
•   Tree per team / feature
•   Trees merged into trunk daily (if green)
•   Scale up via tree of trees (of trees…)
•   Again, less continuous
Federation
•   Each team gets their own deploy pipeline
•   Requires SOA / component architecture
•   Each team can set their own CD pace
•   “Enterprise Ready”
Recap
•   Single trunk + Try pipeline / Autorevert
•   Feature Branches
•   Merge Tree
•   Federation
Questions?

Timothy Fitz (.com)

More Related Content

What's hot

Continuous delivery @wcap 5-09-2013
Continuous delivery   @wcap 5-09-2013Continuous delivery   @wcap 5-09-2013
Continuous delivery @wcap 5-09-2013
David Funaro
 
The Continuous delivery Value @ codemotion 2014
The Continuous delivery Value @ codemotion 2014The Continuous delivery Value @ codemotion 2014
The Continuous delivery Value @ codemotion 2014
David Funaro
 
Introduction to Continuous Integration
Introduction to Continuous IntegrationIntroduction to Continuous Integration
Introduction to Continuous Integration
Somkiat Puisungnoen
 
Continuous integration
Continuous integrationContinuous integration
Continuous integration
Dennis van der Stelt
 
Continuous Deployment of Rails Applications
Continuous Deployment of Rails ApplicationsContinuous Deployment of Rails Applications
Continuous Deployment of Rails Applications
Diane Cessna
 
Continuous integration
Continuous integrationContinuous integration
Continuous integration
hugo lu
 
Continuous Integration at T3CON08
Continuous Integration at T3CON08Continuous Integration at T3CON08
Continuous Integration at T3CON08
Sebastian Kurfürst
 
NYC MeetUp 10.9
NYC MeetUp 10.9NYC MeetUp 10.9
NYC MeetUp 10.9
Solano Labs
 
Best Practices for Database Deployments
Best Practices for Database DeploymentsBest Practices for Database Deployments
Best Practices for Database Deployments
Red Gate Software
 
Database Deployment Pipeline - SQL In The City Workshop 2014
Database Deployment Pipeline - SQL In The City Workshop 2014Database Deployment Pipeline - SQL In The City Workshop 2014
Database Deployment Pipeline - SQL In The City Workshop 2014
Red Gate Software
 
The Continuous delivery value - Funaro
The Continuous delivery value - FunaroThe Continuous delivery value - Funaro
The Continuous delivery value - Funaro
Codemotion
 
Continuous Integration 101
Continuous Integration 101Continuous Integration 101
Continuous Integration 101
John Ferguson Smart Limited
 
Security & DevOps- Ways To Make Sure Your Apps & Infrastructure Are Secure
Security & DevOps- Ways To Make Sure Your Apps & Infrastructure Are SecureSecurity & DevOps- Ways To Make Sure Your Apps & Infrastructure Are Secure
Security & DevOps- Ways To Make Sure Your Apps & Infrastructure Are Secure
Puppet
 
Performance monitoring - Adoniram Mishra, Rupesh Dubey, ThoughtWorks
Performance monitoring - Adoniram Mishra, Rupesh Dubey, ThoughtWorksPerformance monitoring - Adoniram Mishra, Rupesh Dubey, ThoughtWorks
Performance monitoring - Adoniram Mishra, Rupesh Dubey, ThoughtWorks
Thoughtworks
 
WSO2 IoTS Device Manufacturer Guide
WSO2 IoTS Device Manufacturer GuideWSO2 IoTS Device Manufacturer Guide
WSO2 IoTS Device Manufacturer Guide
hugo lu
 
Integration Testing as Validation and Monitoring
 Integration Testing as Validation and Monitoring Integration Testing as Validation and Monitoring
Integration Testing as Validation and Monitoring
Melissa Benua
 
Agile Engineering Best Practices by Richard Cheng
Agile Engineering Best Practices by Richard ChengAgile Engineering Best Practices by Richard Cheng
Agile Engineering Best Practices by Richard Cheng
Excella
 
The Evolution of Agile - Continuous Delivery - Extending Agile out to Product...
The Evolution of Agile - Continuous Delivery - Extending Agile out to Product...The Evolution of Agile - Continuous Delivery - Extending Agile out to Product...
The Evolution of Agile - Continuous Delivery - Extending Agile out to Product...
Burns Sheehan
 
Continuous integration
Continuous integrationContinuous integration
Continuous integration
Basma Alkerm
 
Bootstrapping Quality
Bootstrapping QualityBootstrapping Quality
Bootstrapping Quality
Michael Roufa
 

What's hot (20)

Continuous delivery @wcap 5-09-2013
Continuous delivery   @wcap 5-09-2013Continuous delivery   @wcap 5-09-2013
Continuous delivery @wcap 5-09-2013
 
The Continuous delivery Value @ codemotion 2014
The Continuous delivery Value @ codemotion 2014The Continuous delivery Value @ codemotion 2014
The Continuous delivery Value @ codemotion 2014
 
Introduction to Continuous Integration
Introduction to Continuous IntegrationIntroduction to Continuous Integration
Introduction to Continuous Integration
 
Continuous integration
Continuous integrationContinuous integration
Continuous integration
 
Continuous Deployment of Rails Applications
Continuous Deployment of Rails ApplicationsContinuous Deployment of Rails Applications
Continuous Deployment of Rails Applications
 
Continuous integration
Continuous integrationContinuous integration
Continuous integration
 
Continuous Integration at T3CON08
Continuous Integration at T3CON08Continuous Integration at T3CON08
Continuous Integration at T3CON08
 
NYC MeetUp 10.9
NYC MeetUp 10.9NYC MeetUp 10.9
NYC MeetUp 10.9
 
Best Practices for Database Deployments
Best Practices for Database DeploymentsBest Practices for Database Deployments
Best Practices for Database Deployments
 
Database Deployment Pipeline - SQL In The City Workshop 2014
Database Deployment Pipeline - SQL In The City Workshop 2014Database Deployment Pipeline - SQL In The City Workshop 2014
Database Deployment Pipeline - SQL In The City Workshop 2014
 
The Continuous delivery value - Funaro
The Continuous delivery value - FunaroThe Continuous delivery value - Funaro
The Continuous delivery value - Funaro
 
Continuous Integration 101
Continuous Integration 101Continuous Integration 101
Continuous Integration 101
 
Security & DevOps- Ways To Make Sure Your Apps & Infrastructure Are Secure
Security & DevOps- Ways To Make Sure Your Apps & Infrastructure Are SecureSecurity & DevOps- Ways To Make Sure Your Apps & Infrastructure Are Secure
Security & DevOps- Ways To Make Sure Your Apps & Infrastructure Are Secure
 
Performance monitoring - Adoniram Mishra, Rupesh Dubey, ThoughtWorks
Performance monitoring - Adoniram Mishra, Rupesh Dubey, ThoughtWorksPerformance monitoring - Adoniram Mishra, Rupesh Dubey, ThoughtWorks
Performance monitoring - Adoniram Mishra, Rupesh Dubey, ThoughtWorks
 
WSO2 IoTS Device Manufacturer Guide
WSO2 IoTS Device Manufacturer GuideWSO2 IoTS Device Manufacturer Guide
WSO2 IoTS Device Manufacturer Guide
 
Integration Testing as Validation and Monitoring
 Integration Testing as Validation and Monitoring Integration Testing as Validation and Monitoring
Integration Testing as Validation and Monitoring
 
Agile Engineering Best Practices by Richard Cheng
Agile Engineering Best Practices by Richard ChengAgile Engineering Best Practices by Richard Cheng
Agile Engineering Best Practices by Richard Cheng
 
The Evolution of Agile - Continuous Delivery - Extending Agile out to Product...
The Evolution of Agile - Continuous Delivery - Extending Agile out to Product...The Evolution of Agile - Continuous Delivery - Extending Agile out to Product...
The Evolution of Agile - Continuous Delivery - Extending Agile out to Product...
 
Continuous integration
Continuous integrationContinuous integration
Continuous integration
 
Bootstrapping Quality
Bootstrapping QualityBootstrapping Quality
Bootstrapping Quality
 

Viewers also liked

Failing Continuous Delivery, JDays, 2015
Failing Continuous Delivery, JDays, 2015Failing Continuous Delivery, JDays, 2015
Failing Continuous Delivery, JDays, 2015
Daniel Sawano
 
Continuous Delivery in the AWS Cloud
Continuous Delivery in the AWS CloudContinuous Delivery in the AWS Cloud
Continuous Delivery in the AWS Cloud
Nigel Fernandes
 
Analysis of TLS in SMTP World
Analysis of TLS in SMTP WorldAnalysis of TLS in SMTP World
Analysis of TLS in SMTP World
Binu Ramakrishnan
 
AppSec++ Take the best of Agile, DevOps and CI/CD into your AppSec Program
AppSec++ Take the best of Agile, DevOps and CI/CD into your AppSec ProgramAppSec++ Take the best of Agile, DevOps and CI/CD into your AppSec Program
AppSec++ Take the best of Agile, DevOps and CI/CD into your AppSec Program
Matt Tesauro
 
Infrastructure Continuous Delivery using CloudFormation
Infrastructure Continuous Delivery using CloudFormationInfrastructure Continuous Delivery using CloudFormation
Infrastructure Continuous Delivery using CloudFormation
joehack3r
 
Deep Dive: Infrastructure as Code
Deep Dive: Infrastructure as CodeDeep Dive: Infrastructure as Code
Deep Dive: Infrastructure as Code
Amazon Web Services
 
Infrastructure as Code with AWS CloudFormation
Infrastructure as Code with AWS CloudFormationInfrastructure as Code with AWS CloudFormation
Infrastructure as Code with AWS CloudFormation
Justyna Janczyszyn
 
IBM Innovate - Adoption of Continuous Delivery at Scale at a large telco v0 3
IBM Innovate - Adoption of Continuous Delivery at Scale at a large telco v0 3IBM Innovate - Adoption of Continuous Delivery at Scale at a large telco v0 3
IBM Innovate - Adoption of Continuous Delivery at Scale at a large telco v0 3
Mirco Hering
 
Dod is not done
Dod is not doneDod is not done
Dod is not done
Kris Buytaert
 
Jenkins CI + XebiaLabs for Release Orchestration: A Recipe for Continuous Del...
Jenkins CI + XebiaLabs for Release Orchestration: A Recipe for Continuous Del...Jenkins CI + XebiaLabs for Release Orchestration: A Recipe for Continuous Del...
Jenkins CI + XebiaLabs for Release Orchestration: A Recipe for Continuous Del...
XebiaLabs
 
CI&CD on AWS - Meetup Roma Oct 2016
CI&CD on AWS - Meetup Roma Oct 2016CI&CD on AWS - Meetup Roma Oct 2016
CI&CD on AWS - Meetup Roma Oct 2016
Paolo latella
 
Continuous Deployment: The Dirty Details
Continuous Deployment: The Dirty DetailsContinuous Deployment: The Dirty Details
Continuous Deployment: The Dirty Details
Mike Brittain
 
The Journey of devops and continuous delivery in a Large Financial Institution
The Journey of devops and continuous delivery in a Large Financial InstitutionThe Journey of devops and continuous delivery in a Large Financial Institution
The Journey of devops and continuous delivery in a Large Financial Institution
Kris Buytaert
 
Securing application deployments in multi-tenant CI/CD environments
Securing application deployments in multi-tenant CI/CD environmentsSecuring application deployments in multi-tenant CI/CD environments
Securing application deployments in multi-tenant CI/CD environments
Binu Ramakrishnan
 
Continuous Integration, Build Pipelines and Continuous Deployment
Continuous Integration, Build Pipelines and Continuous DeploymentContinuous Integration, Build Pipelines and Continuous Deployment
Continuous Integration, Build Pipelines and Continuous Deployment
Christopher Read
 
O'Reilly/Nginx 2016: "Continuous Delivery with Containers: The Trials and Tri...
O'Reilly/Nginx 2016: "Continuous Delivery with Containers: The Trials and Tri...O'Reilly/Nginx 2016: "Continuous Delivery with Containers: The Trials and Tri...
O'Reilly/Nginx 2016: "Continuous Delivery with Containers: The Trials and Tri...
Daniel Bryant
 
AWS May Webinar Series - Deep Dive: Infrastructure as Code
AWS May Webinar Series - Deep Dive: Infrastructure as CodeAWS May Webinar Series - Deep Dive: Infrastructure as Code
AWS May Webinar Series - Deep Dive: Infrastructure as Code
Amazon Web Services
 
Infrastructure Continuous Delivery Using AWS CloudFormation
Infrastructure Continuous Delivery Using AWS CloudFormationInfrastructure Continuous Delivery Using AWS CloudFormation
Infrastructure Continuous Delivery Using AWS CloudFormation
Amazon Web Services
 
Enterprise DevOps at Scale with AWS | AWS Public Sector Summit 2016
Enterprise DevOps at Scale with AWS | AWS Public Sector Summit 2016Enterprise DevOps at Scale with AWS | AWS Public Sector Summit 2016
Enterprise DevOps at Scale with AWS | AWS Public Sector Summit 2016
Amazon Web Services
 
Pitfalls of Continuous Deployment
Pitfalls of Continuous DeploymentPitfalls of Continuous Deployment
Pitfalls of Continuous Deployment
zeeg
 

Viewers also liked (20)

Failing Continuous Delivery, JDays, 2015
Failing Continuous Delivery, JDays, 2015Failing Continuous Delivery, JDays, 2015
Failing Continuous Delivery, JDays, 2015
 
Continuous Delivery in the AWS Cloud
Continuous Delivery in the AWS CloudContinuous Delivery in the AWS Cloud
Continuous Delivery in the AWS Cloud
 
Analysis of TLS in SMTP World
Analysis of TLS in SMTP WorldAnalysis of TLS in SMTP World
Analysis of TLS in SMTP World
 
AppSec++ Take the best of Agile, DevOps and CI/CD into your AppSec Program
AppSec++ Take the best of Agile, DevOps and CI/CD into your AppSec ProgramAppSec++ Take the best of Agile, DevOps and CI/CD into your AppSec Program
AppSec++ Take the best of Agile, DevOps and CI/CD into your AppSec Program
 
Infrastructure Continuous Delivery using CloudFormation
Infrastructure Continuous Delivery using CloudFormationInfrastructure Continuous Delivery using CloudFormation
Infrastructure Continuous Delivery using CloudFormation
 
Deep Dive: Infrastructure as Code
Deep Dive: Infrastructure as CodeDeep Dive: Infrastructure as Code
Deep Dive: Infrastructure as Code
 
Infrastructure as Code with AWS CloudFormation
Infrastructure as Code with AWS CloudFormationInfrastructure as Code with AWS CloudFormation
Infrastructure as Code with AWS CloudFormation
 
IBM Innovate - Adoption of Continuous Delivery at Scale at a large telco v0 3
IBM Innovate - Adoption of Continuous Delivery at Scale at a large telco v0 3IBM Innovate - Adoption of Continuous Delivery at Scale at a large telco v0 3
IBM Innovate - Adoption of Continuous Delivery at Scale at a large telco v0 3
 
Dod is not done
Dod is not doneDod is not done
Dod is not done
 
Jenkins CI + XebiaLabs for Release Orchestration: A Recipe for Continuous Del...
Jenkins CI + XebiaLabs for Release Orchestration: A Recipe for Continuous Del...Jenkins CI + XebiaLabs for Release Orchestration: A Recipe for Continuous Del...
Jenkins CI + XebiaLabs for Release Orchestration: A Recipe for Continuous Del...
 
CI&CD on AWS - Meetup Roma Oct 2016
CI&CD on AWS - Meetup Roma Oct 2016CI&CD on AWS - Meetup Roma Oct 2016
CI&CD on AWS - Meetup Roma Oct 2016
 
Continuous Deployment: The Dirty Details
Continuous Deployment: The Dirty DetailsContinuous Deployment: The Dirty Details
Continuous Deployment: The Dirty Details
 
The Journey of devops and continuous delivery in a Large Financial Institution
The Journey of devops and continuous delivery in a Large Financial InstitutionThe Journey of devops and continuous delivery in a Large Financial Institution
The Journey of devops and continuous delivery in a Large Financial Institution
 
Securing application deployments in multi-tenant CI/CD environments
Securing application deployments in multi-tenant CI/CD environmentsSecuring application deployments in multi-tenant CI/CD environments
Securing application deployments in multi-tenant CI/CD environments
 
Continuous Integration, Build Pipelines and Continuous Deployment
Continuous Integration, Build Pipelines and Continuous DeploymentContinuous Integration, Build Pipelines and Continuous Deployment
Continuous Integration, Build Pipelines and Continuous Deployment
 
O'Reilly/Nginx 2016: "Continuous Delivery with Containers: The Trials and Tri...
O'Reilly/Nginx 2016: "Continuous Delivery with Containers: The Trials and Tri...O'Reilly/Nginx 2016: "Continuous Delivery with Containers: The Trials and Tri...
O'Reilly/Nginx 2016: "Continuous Delivery with Containers: The Trials and Tri...
 
AWS May Webinar Series - Deep Dive: Infrastructure as Code
AWS May Webinar Series - Deep Dive: Infrastructure as CodeAWS May Webinar Series - Deep Dive: Infrastructure as Code
AWS May Webinar Series - Deep Dive: Infrastructure as Code
 
Infrastructure Continuous Delivery Using AWS CloudFormation
Infrastructure Continuous Delivery Using AWS CloudFormationInfrastructure Continuous Delivery Using AWS CloudFormation
Infrastructure Continuous Delivery Using AWS CloudFormation
 
Enterprise DevOps at Scale with AWS | AWS Public Sector Summit 2016
Enterprise DevOps at Scale with AWS | AWS Public Sector Summit 2016Enterprise DevOps at Scale with AWS | AWS Public Sector Summit 2016
Enterprise DevOps at Scale with AWS | AWS Public Sector Summit 2016
 
Pitfalls of Continuous Deployment
Pitfalls of Continuous DeploymentPitfalls of Continuous Deployment
Pitfalls of Continuous Deployment
 

Similar to The Hard Problems of Continuous Deployment

Road to Continuous Delivery - Wix.com
Road to Continuous Delivery - Wix.comRoad to Continuous Delivery - Wix.com
Road to Continuous Delivery - Wix.com
Aviran Mordo
 
The Rocky Cloud Road
The Rocky Cloud RoadThe Rocky Cloud Road
The Rocky Cloud Road
Gert Drapers
 
2016 09-dev opsjourney-devopsdaysoslo
2016 09-dev opsjourney-devopsdaysoslo2016 09-dev opsjourney-devopsdaysoslo
2016 09-dev opsjourney-devopsdaysoslo
Jon Arild Tørresdal
 
Make It Cooler: Using Decentralized Version Control
Make It Cooler: Using Decentralized Version ControlMake It Cooler: Using Decentralized Version Control
Make It Cooler: Using Decentralized Version Control
indiver
 
Extreme Makeover OnBase Edition
Extreme Makeover OnBase EditionExtreme Makeover OnBase Edition
Extreme Makeover OnBase Edition
DataBank, A KYOCERA Group Company
 
DevOps / Agile Tools Seminar 2013
DevOps / Agile Tools Seminar 2013DevOps / Agile Tools Seminar 2013
DevOps / Agile Tools Seminar 2013
Ethan Ram
 
Lessons Learned Replatforming A Large Machine Learning Application To Apache ...
Lessons Learned Replatforming A Large Machine Learning Application To Apache ...Lessons Learned Replatforming A Large Machine Learning Application To Apache ...
Lessons Learned Replatforming A Large Machine Learning Application To Apache ...
Databricks
 
Introduction to continuous delivery
Introduction to continuous deliveryIntroduction to continuous delivery
Introduction to continuous delivery
OlympicSoftware
 
SynapseIndia drupal presentation on drupal info
SynapseIndia drupal  presentation on drupal infoSynapseIndia drupal  presentation on drupal info
SynapseIndia drupal presentation on drupal info
Synapseindiappsdevelopment
 
Test Driven Development & CI/CD
Test Driven Development & CI/CDTest Driven Development & CI/CD
Test Driven Development & CI/CD
Shanmuga S Muthu
 
How to Build a Compute Cluster
How to Build a Compute ClusterHow to Build a Compute Cluster
How to Build a Compute Cluster
Ramsay Key
 
Denver devops : enabling DevOps with data virtualization
Denver devops : enabling DevOps with data virtualizationDenver devops : enabling DevOps with data virtualization
Denver devops : enabling DevOps with data virtualization
Kyle Hailey
 
E2 evc 3-2-1-rule - mikeresseler
E2 evc   3-2-1-rule - mikeresselerE2 evc   3-2-1-rule - mikeresseler
E2 evc 3-2-1-rule - mikeresseler
Mike Resseler
 
Getting to Walk with DevOps
Getting to Walk with DevOpsGetting to Walk with DevOps
Getting to Walk with DevOps
Eklove Mohan
 
Microservices: The Best Practices
Microservices: The Best PracticesMicroservices: The Best Practices
Microservices: The Best Practices
Pavel Mička
 
Towards Continuous Delivery in Mobile Apps
Towards Continuous Delivery in Mobile AppsTowards Continuous Delivery in Mobile Apps
Towards Continuous Delivery in Mobile Apps
Phillip Wheatley
 
Hpc lunch and learn
Hpc lunch and learnHpc lunch and learn
Hpc lunch and learn
John D Almon
 
Sailing through devlopment with legacy code
Sailing through devlopment with legacy codeSailing through devlopment with legacy code
Sailing through devlopment with legacy code
prasadkunte
 
Why retail companies can't afford database downtime
Why retail companies can't afford database downtimeWhy retail companies can't afford database downtime
Why retail companies can't afford database downtime
DBmaestro - Database DevOps
 
Mixing d ps building architecture on the cross cutting example
Mixing d ps building architecture on the cross cutting exampleMixing d ps building architecture on the cross cutting example
Mixing d ps building architecture on the cross cutting example
corehard_by
 

Similar to The Hard Problems of Continuous Deployment (20)

Road to Continuous Delivery - Wix.com
Road to Continuous Delivery - Wix.comRoad to Continuous Delivery - Wix.com
Road to Continuous Delivery - Wix.com
 
The Rocky Cloud Road
The Rocky Cloud RoadThe Rocky Cloud Road
The Rocky Cloud Road
 
2016 09-dev opsjourney-devopsdaysoslo
2016 09-dev opsjourney-devopsdaysoslo2016 09-dev opsjourney-devopsdaysoslo
2016 09-dev opsjourney-devopsdaysoslo
 
Make It Cooler: Using Decentralized Version Control
Make It Cooler: Using Decentralized Version ControlMake It Cooler: Using Decentralized Version Control
Make It Cooler: Using Decentralized Version Control
 
Extreme Makeover OnBase Edition
Extreme Makeover OnBase EditionExtreme Makeover OnBase Edition
Extreme Makeover OnBase Edition
 
DevOps / Agile Tools Seminar 2013
DevOps / Agile Tools Seminar 2013DevOps / Agile Tools Seminar 2013
DevOps / Agile Tools Seminar 2013
 
Lessons Learned Replatforming A Large Machine Learning Application To Apache ...
Lessons Learned Replatforming A Large Machine Learning Application To Apache ...Lessons Learned Replatforming A Large Machine Learning Application To Apache ...
Lessons Learned Replatforming A Large Machine Learning Application To Apache ...
 
Introduction to continuous delivery
Introduction to continuous deliveryIntroduction to continuous delivery
Introduction to continuous delivery
 
SynapseIndia drupal presentation on drupal info
SynapseIndia drupal  presentation on drupal infoSynapseIndia drupal  presentation on drupal info
SynapseIndia drupal presentation on drupal info
 
Test Driven Development & CI/CD
Test Driven Development & CI/CDTest Driven Development & CI/CD
Test Driven Development & CI/CD
 
How to Build a Compute Cluster
How to Build a Compute ClusterHow to Build a Compute Cluster
How to Build a Compute Cluster
 
Denver devops : enabling DevOps with data virtualization
Denver devops : enabling DevOps with data virtualizationDenver devops : enabling DevOps with data virtualization
Denver devops : enabling DevOps with data virtualization
 
E2 evc 3-2-1-rule - mikeresseler
E2 evc   3-2-1-rule - mikeresselerE2 evc   3-2-1-rule - mikeresseler
E2 evc 3-2-1-rule - mikeresseler
 
Getting to Walk with DevOps
Getting to Walk with DevOpsGetting to Walk with DevOps
Getting to Walk with DevOps
 
Microservices: The Best Practices
Microservices: The Best PracticesMicroservices: The Best Practices
Microservices: The Best Practices
 
Towards Continuous Delivery in Mobile Apps
Towards Continuous Delivery in Mobile AppsTowards Continuous Delivery in Mobile Apps
Towards Continuous Delivery in Mobile Apps
 
Hpc lunch and learn
Hpc lunch and learnHpc lunch and learn
Hpc lunch and learn
 
Sailing through devlopment with legacy code
Sailing through devlopment with legacy codeSailing through devlopment with legacy code
Sailing through devlopment with legacy code
 
Why retail companies can't afford database downtime
Why retail companies can't afford database downtimeWhy retail companies can't afford database downtime
Why retail companies can't afford database downtime
 
Mixing d ps building architecture on the cross cutting example
Mixing d ps building architecture on the cross cutting exampleMixing d ps building architecture on the cross cutting example
Mixing d ps building architecture on the cross cutting example
 

More from Timothy Fitz

Good hypothesis testing is surprising!
Good hypothesis testing is surprising!Good hypothesis testing is surprising!
Good hypothesis testing is surprising!
Timothy Fitz
 
Gdc 2010 architecture final slideshare edition
Gdc 2010 architecture final slideshare editionGdc 2010 architecture final slideshare edition
Gdc 2010 architecture final slideshare edition
Timothy Fitz
 
Realtime web2012
Realtime web2012Realtime web2012
Realtime web2012
Timothy Fitz
 
Continuous Deployment
Continuous DeploymentContinuous Deployment
Continuous Deployment
Timothy Fitz
 
Shdh
ShdhShdh
Socket.io
Socket.ioSocket.io
Socket.io
Timothy Fitz
 
Shdh
ShdhShdh
Just In Time Scalability Agile Methods To Support Massive Growth Presentation
Just In Time Scalability  Agile Methods To Support Massive Growth PresentationJust In Time Scalability  Agile Methods To Support Massive Growth Presentation
Just In Time Scalability Agile Methods To Support Massive Growth Presentation
Timothy Fitz
 

More from Timothy Fitz (8)

Good hypothesis testing is surprising!
Good hypothesis testing is surprising!Good hypothesis testing is surprising!
Good hypothesis testing is surprising!
 
Gdc 2010 architecture final slideshare edition
Gdc 2010 architecture final slideshare editionGdc 2010 architecture final slideshare edition
Gdc 2010 architecture final slideshare edition
 
Realtime web2012
Realtime web2012Realtime web2012
Realtime web2012
 
Continuous Deployment
Continuous DeploymentContinuous Deployment
Continuous Deployment
 
Shdh
ShdhShdh
Shdh
 
Socket.io
Socket.ioSocket.io
Socket.io
 
Shdh
ShdhShdh
Shdh
 
Just In Time Scalability Agile Methods To Support Massive Growth Presentation
Just In Time Scalability  Agile Methods To Support Massive Growth PresentationJust In Time Scalability  Agile Methods To Support Massive Growth Presentation
Just In Time Scalability Agile Methods To Support Massive Growth Presentation
 

Recently uploaded

WeTestAthens: Postman's AI & Automation Techniques
WeTestAthens: Postman's AI & Automation TechniquesWeTestAthens: Postman's AI & Automation Techniques
WeTestAthens: Postman's AI & Automation Techniques
Postman
 
Best 20 SEO Techniques To Improve Website Visibility In SERP
Best 20 SEO Techniques To Improve Website Visibility In SERPBest 20 SEO Techniques To Improve Website Visibility In SERP
Best 20 SEO Techniques To Improve Website Visibility In SERP
Pixlogix Infotech
 
Finale of the Year: Apply for Next One!
Finale of the Year: Apply for Next One!Finale of the Year: Apply for Next One!
Finale of the Year: Apply for Next One!
GDSC PJATK
 
Ocean lotus Threat actors project by John Sitima 2024 (1).pptx
Ocean lotus Threat actors project by John Sitima 2024 (1).pptxOcean lotus Threat actors project by John Sitima 2024 (1).pptx
Ocean lotus Threat actors project by John Sitima 2024 (1).pptx
SitimaJohn
 
Presentation of the OECD Artificial Intelligence Review of Germany
Presentation of the OECD Artificial Intelligence Review of GermanyPresentation of the OECD Artificial Intelligence Review of Germany
Presentation of the OECD Artificial Intelligence Review of Germany
innovationoecd
 
Operating System Used by Users in day-to-day life.pptx
Operating System Used by Users in day-to-day life.pptxOperating System Used by Users in day-to-day life.pptx
Operating System Used by Users in day-to-day life.pptx
Pravash Chandra Das
 
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with SlackLet's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
shyamraj55
 
Programming Foundation Models with DSPy - Meetup Slides
Programming Foundation Models with DSPy - Meetup SlidesProgramming Foundation Models with DSPy - Meetup Slides
Programming Foundation Models with DSPy - Meetup Slides
Zilliz
 
Nordic Marketo Engage User Group_June 13_ 2024.pptx
Nordic Marketo Engage User Group_June 13_ 2024.pptxNordic Marketo Engage User Group_June 13_ 2024.pptx
Nordic Marketo Engage User Group_June 13_ 2024.pptx
MichaelKnudsen27
 
Serial Arm Control in Real Time Presentation
Serial Arm Control in Real Time PresentationSerial Arm Control in Real Time Presentation
Serial Arm Control in Real Time Presentation
tolgahangng
 
HCL Notes and Domino License Cost Reduction in the World of DLAU
HCL Notes and Domino License Cost Reduction in the World of DLAUHCL Notes and Domino License Cost Reduction in the World of DLAU
HCL Notes and Domino License Cost Reduction in the World of DLAU
panagenda
 
Overcoming the PLG Trap: Lessons from Canva's Head of Sales & Head of EMEA Da...
Overcoming the PLG Trap: Lessons from Canva's Head of Sales & Head of EMEA Da...Overcoming the PLG Trap: Lessons from Canva's Head of Sales & Head of EMEA Da...
Overcoming the PLG Trap: Lessons from Canva's Head of Sales & Head of EMEA Da...
saastr
 
Columbus Data & Analytics Wednesdays - June 2024
Columbus Data & Analytics Wednesdays - June 2024Columbus Data & Analytics Wednesdays - June 2024
Columbus Data & Analytics Wednesdays - June 2024
Jason Packer
 
Main news related to the CCS TSI 2023 (2023/1695)
Main news related to the CCS TSI 2023 (2023/1695)Main news related to the CCS TSI 2023 (2023/1695)
Main news related to the CCS TSI 2023 (2023/1695)
Jakub Marek
 
Skybuffer AI: Advanced Conversational and Generative AI Solution on SAP Busin...
Skybuffer AI: Advanced Conversational and Generative AI Solution on SAP Busin...Skybuffer AI: Advanced Conversational and Generative AI Solution on SAP Busin...
Skybuffer AI: Advanced Conversational and Generative AI Solution on SAP Busin...
Tatiana Kojar
 
leewayhertz.com-AI in predictive maintenance Use cases technologies benefits ...
leewayhertz.com-AI in predictive maintenance Use cases technologies benefits ...leewayhertz.com-AI in predictive maintenance Use cases technologies benefits ...
leewayhertz.com-AI in predictive maintenance Use cases technologies benefits ...
alexjohnson7307
 
Choosing The Best AWS Service For Your Website + API.pptx
Choosing The Best AWS Service For Your Website + API.pptxChoosing The Best AWS Service For Your Website + API.pptx
Choosing The Best AWS Service For Your Website + API.pptx
Brandon Minnick, MBA
 
Artificial Intelligence for XMLDevelopment
Artificial Intelligence for XMLDevelopmentArtificial Intelligence for XMLDevelopment
Artificial Intelligence for XMLDevelopment
Octavian Nadolu
 
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdfHow to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
Chart Kalyan
 
GenAI Pilot Implementation in the organizations
GenAI Pilot Implementation in the organizationsGenAI Pilot Implementation in the organizations
GenAI Pilot Implementation in the organizations
kumardaparthi1024
 

Recently uploaded (20)

WeTestAthens: Postman's AI & Automation Techniques
WeTestAthens: Postman's AI & Automation TechniquesWeTestAthens: Postman's AI & Automation Techniques
WeTestAthens: Postman's AI & Automation Techniques
 
Best 20 SEO Techniques To Improve Website Visibility In SERP
Best 20 SEO Techniques To Improve Website Visibility In SERPBest 20 SEO Techniques To Improve Website Visibility In SERP
Best 20 SEO Techniques To Improve Website Visibility In SERP
 
Finale of the Year: Apply for Next One!
Finale of the Year: Apply for Next One!Finale of the Year: Apply for Next One!
Finale of the Year: Apply for Next One!
 
Ocean lotus Threat actors project by John Sitima 2024 (1).pptx
Ocean lotus Threat actors project by John Sitima 2024 (1).pptxOcean lotus Threat actors project by John Sitima 2024 (1).pptx
Ocean lotus Threat actors project by John Sitima 2024 (1).pptx
 
Presentation of the OECD Artificial Intelligence Review of Germany
Presentation of the OECD Artificial Intelligence Review of GermanyPresentation of the OECD Artificial Intelligence Review of Germany
Presentation of the OECD Artificial Intelligence Review of Germany
 
Operating System Used by Users in day-to-day life.pptx
Operating System Used by Users in day-to-day life.pptxOperating System Used by Users in day-to-day life.pptx
Operating System Used by Users in day-to-day life.pptx
 
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with SlackLet's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
 
Programming Foundation Models with DSPy - Meetup Slides
Programming Foundation Models with DSPy - Meetup SlidesProgramming Foundation Models with DSPy - Meetup Slides
Programming Foundation Models with DSPy - Meetup Slides
 
Nordic Marketo Engage User Group_June 13_ 2024.pptx
Nordic Marketo Engage User Group_June 13_ 2024.pptxNordic Marketo Engage User Group_June 13_ 2024.pptx
Nordic Marketo Engage User Group_June 13_ 2024.pptx
 
Serial Arm Control in Real Time Presentation
Serial Arm Control in Real Time PresentationSerial Arm Control in Real Time Presentation
Serial Arm Control in Real Time Presentation
 
HCL Notes and Domino License Cost Reduction in the World of DLAU
HCL Notes and Domino License Cost Reduction in the World of DLAUHCL Notes and Domino License Cost Reduction in the World of DLAU
HCL Notes and Domino License Cost Reduction in the World of DLAU
 
Overcoming the PLG Trap: Lessons from Canva's Head of Sales & Head of EMEA Da...
Overcoming the PLG Trap: Lessons from Canva's Head of Sales & Head of EMEA Da...Overcoming the PLG Trap: Lessons from Canva's Head of Sales & Head of EMEA Da...
Overcoming the PLG Trap: Lessons from Canva's Head of Sales & Head of EMEA Da...
 
Columbus Data & Analytics Wednesdays - June 2024
Columbus Data & Analytics Wednesdays - June 2024Columbus Data & Analytics Wednesdays - June 2024
Columbus Data & Analytics Wednesdays - June 2024
 
Main news related to the CCS TSI 2023 (2023/1695)
Main news related to the CCS TSI 2023 (2023/1695)Main news related to the CCS TSI 2023 (2023/1695)
Main news related to the CCS TSI 2023 (2023/1695)
 
Skybuffer AI: Advanced Conversational and Generative AI Solution on SAP Busin...
Skybuffer AI: Advanced Conversational and Generative AI Solution on SAP Busin...Skybuffer AI: Advanced Conversational and Generative AI Solution on SAP Busin...
Skybuffer AI: Advanced Conversational and Generative AI Solution on SAP Busin...
 
leewayhertz.com-AI in predictive maintenance Use cases technologies benefits ...
leewayhertz.com-AI in predictive maintenance Use cases technologies benefits ...leewayhertz.com-AI in predictive maintenance Use cases technologies benefits ...
leewayhertz.com-AI in predictive maintenance Use cases technologies benefits ...
 
Choosing The Best AWS Service For Your Website + API.pptx
Choosing The Best AWS Service For Your Website + API.pptxChoosing The Best AWS Service For Your Website + API.pptx
Choosing The Best AWS Service For Your Website + API.pptx
 
Artificial Intelligence for XMLDevelopment
Artificial Intelligence for XMLDevelopmentArtificial Intelligence for XMLDevelopment
Artificial Intelligence for XMLDevelopment
 
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdfHow to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
 
GenAI Pilot Implementation in the organizations
GenAI Pilot Implementation in the organizationsGenAI Pilot Implementation in the organizations
GenAI Pilot Implementation in the organizations
 

The Hard Problems of Continuous Deployment

  • 1. The Hard Problems of Continuous Deployment Timothy Fitz (.com)
  • 2. “Continuous deployment involves deploying early and often, so as to avoid the pitfalls of "deployment hell". The practice aims to reduce timely rework and thus reduce cost and development time.”
  • 3. “Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.” - Principles behind the Agile Manifesto (2001)
  • 4. The Hard Problems (The ones I’m going to cover) • Data • Mobile • Scale
  • 5. Data makes CD harder • Updating data is slow • Moving data is slow • Schemas live outside code deploy process • Rollbacks are often hard or impossible
  • 6. Schema rev. N, N+1 compatible • Add the column in production • Push the code that writes to that column • Optionally, run a data migration to populate the existing rows with data • Push the code that reads from that column.
  • 7. Apply only cheap changes • Only apply changes that are cheap enough to not affect live traffic • More complex changes split into tiny steps: – Create new table – Write to both – Cut over eventually – Drop old table
  • 8. Apply change to standby • Run two DB instances • Apply change to standby • Failover if successfully applied • Might run 3rd db instance for availability
  • 9. Blue/Green Deploy • Run two copies of entire cluster • All databases are replicated • Lets you test, update and rollback both code and schema in one step
  • 10. Schemaless NoSQL • “Schemaless” really means schema is in your source code – Which is great! We can CD schema changes
  • 11. Schemaless-style SQL • Benefits of existing SQL databases while still being pseudoschemaless • (thing_id, key, value) table • (guid, blob) table
  • 12. Recap • Apply only cheap changes • Apply changes to standby • Blue/Green Deploy • Schemaless NoSQL • Schemaless-style SQL
  • 13. Mobile • Users must opt-in to every update • iOS submissions take a week to be approved • Luckily, lots of tools aimed at the space
  • 14. Remember the basics • CI server / automated tests are critical • Can’t fallback on production alarms / rollback • Hosted options: http://cisimple.com
  • 15. Data-drive everything • Build your views / content from data files • Ping server for updates • Hosted: http://appgrok.com/ – Lets you deploy txt, png and xib dynamically!
  • 16. 99% HTML • Entire app is a single UIWebView • Glue native code to allow access to APIs • Clutch.io is awesome (and FOSS now) – Live reloading for local dev – Streamlined deploys – https://github.com/clutchio
  • 17. Hybrid HTML/Native • Core app is native • Sections can be replaced by HTML – i.e. Facebook stream entries fallback to HTML • Infrequently used sections are 100% HTML
  • 18. Recap • Remember the basics • Data-drive everything • 99% HTML • Hybrid HTML/Native
  • 19. Scalability • Maintaining availability, performance and happiness • As a function of # of people • As a function of # of tests
  • 20. Availability • The build must always be green • Set a “green SLA” – 99% green – Never red for > 15m • Measure, track and report on these numbers
  • 21. Performance • Measure (intent to) commit to time of deploy – Goal: < 5 minutes • Measure local development test loop – Goal: < 2s
  • 22. Happiness • CI/CD System is a product • Software Engineers are the customer • Keep your customer happy!
  • 24. How do you make tests fast? • Tests can exercise large amounts of code without being slow • Minimize system calls (no I/O, no disk) • Minimize test data size • Make sure all systems are cheap to instantiate/teardown • No external state makes tests more reliable
  • 25. Run Tests in Parallel • Multiprocess • Multimachine • Multi-VM • Instant multi-VM: http://circleci.com
  • 26. Hardware Scale • CI Cluster will get huge – Function of cumulative engineering man-months – Rule of thumb: 10% of your cluster size • You will need a CI/CD DevOps person – CI cluster monitoring / alerting – Configuration Management critical
  • 27. Scale testing infrastructure recap • Write the right kind of tests • Make those tests as fast as possible • Run those tests in parallel
  • 28. People / Roles • Sheriff – Designated reverter / problem troubleshooter – Common pattern (IMVU, Chromium, Firefox) • CD “Product Owner” – Held accountable for SLA / Performance – Manage infrastructure backlog
  • 29. Single trunk • Do this until it doesn’t work for you • Gets painful in the 16 – 32 developer range • Faster commit->deploy reduces the pain – But effort becomes prohibitive
  • 30. “Try” pipeline • Conceptually, a second tree that “doesn’t matter” but still gets tested for feedback • Buildbot implements a patch-pushing version • Takes a significant amount of pressure off of trunk builds
  • 31. CI Server takes active role • Server automatically reverts red commits • Server merges green commits to trunk
  • 32. Feature branches • All incremental development happens on branches, branches land when feature is “ready” • If “feature” is kept small, can be 2-3 per engineer per week on average • Less continuous, but scales much better – Feature branches tested before merge
  • 33. Merge tree • Tree per team / feature • Trees merged into trunk daily (if green) • Scale up via tree of trees (of trees…) • Again, less continuous
  • 34. Federation • Each team gets their own deploy pipeline • Requires SOA / component architecture • Each team can set their own CD pace • “Enterprise Ready”
  • 35. Recap • Single trunk + Try pipeline / Autorevert • Feature Branches • Merge Tree • Federation

Editor's Notes

  1. About me: IMVU, Canvas, Continuous Deployment)Ground rules: I don’t demand your attention, please tweet / follow links while I’m talking. If you have questions, shoot up a hand. If I don’t see you, yell at me.
  2. ContinousDeployment vs Continuous Delivery (next slide)
  3. Very similarContinuous Delivery is “agile as it should be”Continuous Delivery does NOT advocate for deploy-on-commit (Book and term reaction to Continuous Deployment)Continuous Deployment advocates for MORE THAN deploy-on-commit. (August 2007, 5 years old!)At the end of the day, I don’t care. People seem to be using both terms interchangeably. Whatever you call it, however far you take it, it’s better than not deploying very often.
  4. These are the problems I get asked about all the time.No silver bullets.
  5. In general I assume some sort of LAMP stack.
  6. Seems straightforward, but often means what would’ve been a 1 step process (with downtime) is now a 15 step process. BUT! If you follow it well, you can always step backwards, one small step at a time, in the event that it doesn’t work out / is wrong
  7. Sharding your data and keeping your shards balanced (and measured) means you can quantitatively assess this, especially if you keep an offline copy of a single shard.i.e. keep 100k users per shard, letting you do almost anything to any O(user) table.
  8. Not hitting on master-master / master-standby / etc distinctions, because the exact setup depends on your db / replication / required availability.
  9. http://martinfowler.com/bliki/BlueGreenDeployment.html
  10. Hybrid SQL db and NoSQLdb, use each where appropriate
  11. Most real world scenarios are a hybrid
  12. Experience heavily biased to iOS, would love to hear Android perspective
  13. Don’t just throw up your arms and say “CD can’t be done”Apparently cozying up with Apple can lower your submission turnaround time from 1 week to 1 day.
  14. Trades development speed for performanceFor most apps that’s probably the right trade (users don’t really notice)
  15. Get back the performance you were missing, and native scrollingLose some of the dynamic propertiesMuch more expensive to develop
  16. Commit to deploy&lt; 5 minute: stay in flow5 – 15 minutes: can keep working on feature15 minutes: failures are surprises and require expensive rewindingLocal dev loop&lt; 2s: stay in tight flow2s – 10s: tab away from terminal, looser flow10s – 1min: start thinking or coding on next thing, failures require rewinding1m – 5min: significant rewinding, high distraction, painful testing (rolling chair jousting)
  17. Example: All metrics green, everything looks great, but got to metrics by shaming anyone who breaks build. Culture of tip-toeing through the bulid system leads to reduced happiness (and reduced throughput!)Sidenote: Measure throughput! Deploys per engineer (avg/median/extremes), is it scaling up with org or are you getting less deploys?
  18. GUI Testsi.e. SeleniumIntegration Testsi.e. uses the databaseUnit Testssmall fast tests with no external state
  19. i.e. Don’t reload your country code or ipgeo tables, even into memory from disk, on every test
  20. Highly recommend Buildbot when you hit scale. It’s proven at huge scale (500+ nodes) and growing, and allows way better pipeline customization at that scale
  21. Test time is O(Cumulative man-months)Doubling staff means halving test time, in spite of ever-marching-on increase in tests
  22. Github modelMost FOSS is moving to or using this model
  23. Lots of people/process overheadClassical way of scaling up to extremely large teams
  24. Fortune 500-proof.
  25. Remember: Don’t overinvest up front. Better to do something simple until it doesn’t work, than to overbuild.Don’t underbuild either. Use