The Anatomy of Continuous Deployment
At Scale
John Viner (Development Manager)
CTO Summit 2016
Who is Envato?
We run a two-sided Marketplace for creative assets
that has paid out $450 million to our “Authors” in the
last 10 years
7 million registered users
10 million items for sale
The Business
25,000 requests a minute
160m Page Views a Month
1 sale a second at Peak
120,000 lines of code (Ruby on Rails)
25,000 tests
75 contributors, 10 years old
The Tech
Continuous Deployment vs Continuous Delivery
“Continuous Delivery is a software development
discipline where you build software in such a way
that the software can be released to production at
any time.”
“Continuous Deployment is a software development
practice in which every code change is deployed to
production, automatically.”
Foundational Engineering Capabilities
Visible Operational
Monitoring
Feature Toggle Tooling
Rollback Capability
Automated Build and Deploy Toolset
Comprehensive Automated Test Suite
Quality Code
Continuous Deployment
Foundational Organisational Behaviours
MTTR Focus
No QA / Approval Gates
Access to Production
Blameless Post Incident Reviews
Developers on First Line Support
Pairing and/or Peer Review
Continuous Deployment
The Lifecycle of a Change at Envato
Pull Request (PR) for all changes - Github
Developer Merges PR with at least 1 review
Automated Tests executed (no perf tests) - Buildkite
Automated Deploy executed on passed test - Samson + CodeDeploy
Rolling Deploy of Application with Zero Downtime
Schema Migrations with Zero Downtime
The Thumbs Up
Communication
The Numbers for June 2016
~ 40 developers
3000 total builds (2.5 builds p/dev/day)
500 Pull Requests Closed
400 deployments (20 p/day, 0.5 p/dev/day, peak 25 per day)
4.5 hour cycle time (from first commit to deploy - working hours)
8000 deployments in the last 2 Years
Pull Request Size - 37 lines changed , 3 files changed
Scaling the Team
Myths Dispelled
Continuous Deployment is only for:
• Small Teams
• Small Scale
• Low Commercial Impact of Outage
The Benefits for Envato
1. Reduced Risk.
• Small Changes are Easier to Identify and Rollback (or forward)
2. Increases Speed to Market
• Small Batch Efficiency - Faster Throughput
3. Increases Developer Happiness
• Code in Production makes for happy devs
• Supporting Production increases empathy with customer
How to Get Started
• Start on a Small System
• Encourage a Culture of Developer Ownership
• Invest in Test Automation
• Automate Your Build and Deployment Pipeline
• Invest in Feature Toggles
• Measure Your Cycle Time
• Optimise for Mean Time to Recovery (not Failure)
Thank you!

The Anatomy of Continuous Deployment at Scale

  • 1.
    The Anatomy ofContinuous Deployment At Scale John Viner (Development Manager) CTO Summit 2016
  • 2.
  • 3.
    We run atwo-sided Marketplace for creative assets that has paid out $450 million to our “Authors” in the last 10 years 7 million registered users 10 million items for sale The Business
  • 4.
    25,000 requests aminute 160m Page Views a Month 1 sale a second at Peak 120,000 lines of code (Ruby on Rails) 25,000 tests 75 contributors, 10 years old The Tech
  • 5.
    Continuous Deployment vsContinuous Delivery “Continuous Delivery is a software development discipline where you build software in such a way that the software can be released to production at any time.” “Continuous Deployment is a software development practice in which every code change is deployed to production, automatically.”
  • 6.
    Foundational Engineering Capabilities VisibleOperational Monitoring Feature Toggle Tooling Rollback Capability Automated Build and Deploy Toolset Comprehensive Automated Test Suite Quality Code Continuous Deployment
  • 7.
    Foundational Organisational Behaviours MTTRFocus No QA / Approval Gates Access to Production Blameless Post Incident Reviews Developers on First Line Support Pairing and/or Peer Review Continuous Deployment
  • 8.
    The Lifecycle ofa Change at Envato Pull Request (PR) for all changes - Github Developer Merges PR with at least 1 review Automated Tests executed (no perf tests) - Buildkite Automated Deploy executed on passed test - Samson + CodeDeploy Rolling Deploy of Application with Zero Downtime Schema Migrations with Zero Downtime
  • 9.
  • 10.
  • 11.
    The Numbers forJune 2016 ~ 40 developers 3000 total builds (2.5 builds p/dev/day) 500 Pull Requests Closed 400 deployments (20 p/day, 0.5 p/dev/day, peak 25 per day) 4.5 hour cycle time (from first commit to deploy - working hours) 8000 deployments in the last 2 Years Pull Request Size - 37 lines changed , 3 files changed
  • 12.
  • 13.
    Myths Dispelled Continuous Deploymentis only for: • Small Teams • Small Scale • Low Commercial Impact of Outage
  • 14.
    The Benefits forEnvato 1. Reduced Risk. • Small Changes are Easier to Identify and Rollback (or forward) 2. Increases Speed to Market • Small Batch Efficiency - Faster Throughput 3. Increases Developer Happiness • Code in Production makes for happy devs • Supporting Production increases empathy with customer
  • 15.
    How to GetStarted • Start on a Small System • Encourage a Culture of Developer Ownership • Invest in Test Automation • Automate Your Build and Deployment Pipeline • Invest in Feature Toggles • Measure Your Cycle Time • Optimise for Mean Time to Recovery (not Failure)
  • 16.