Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Scaling UpContinuous Deployment    Timothy Fitz (.com)
“Continuous deployment involves  deploying early and often, so as toavoid the pitfalls of "deployment hell".The practice a...
Scalability• Maintaining availability, performance and  happiness• As a function of # of people• As a function of # of tests
Availability• The build must always be green• Set a “green SLA”  – 99% green  – Never red for > 15m• Measure, track and re...
Performance• Measure (intent to) commit to time of deploy  – Goal: < 5 minutes• Measure local development test loop  – Goa...
Happiness• CI/CD System is a product• Software Engineers are the customer• Keep your customer happy!
Testing Pyramid
How do you make tests fast?• Tests can exercise large amounts of code  without being slow• Minimize system calls (no I/O, ...
Run Tests in Parallel•   Multiprocess•   Multimachine•   Multi-VM•   Instant multi-VM: http://circleci.com
Hardware Scale• CI Cluster will get huge  – Function of cumulative engineering man-months  – Rule of thumb: 10% of your cl...
Scale testing infrastructure recap• Write the right kind of tests• Make those tests as fast as possible• Run those tests i...
People / Roles• Sheriff  – Designated reverter / problem troubleshooter  – Common pattern (IMVU, Chromium, Firefox)• CD “P...
Single trunk• Do this until it doesn’t work for you• Gets painful in the 16 – 32 developer range• Faster commit->deploy re...
“Try” pipeline• Conceptually, a second tree that “doesn’t  matter” but still gets tested for feedback• Buildbot implements...
CI Server takes active role• Server automatically reverts red commits• Server merges green commits to trunk
Feature branches• All incremental development happens on  branches, branches land when feature is  “ready”• If “feature” i...
Merge tree•   Tree per team / feature•   Trees merged into trunk daily (if green)•   Scale up via tree of trees (of trees…...
Federation•   Each team gets their own deploy pipeline•   Requires SOA / component architecture•   Each team can set their...
Recap•   Single trunk + Try pipeline / Autorevert•   Feature Branches•   Merge Tree•   Federation
Questions?Timothy Fitz (.com)
You’ve finished this document.
Download and read it offline.
Upcoming SlideShare
Continuous Delivery in the AWS Cloud
Next
Upcoming SlideShare
Continuous Delivery in the AWS Cloud
Next
Download to read offline and view in fullscreen.

Share

Scaling Up Continuous Deployment

Download to read offline

How do you continue to ship 50 times a day, when you're constantly hiring more engineers? How can you continue, when every day you write more tests that need to be run on every commit? This talk will cover how to scale up Continuous Integration and Continuous Deployment infrastracture, for teams as small as a handful of engineers and as large as hundreds of engineers.

Related Books

Free with a 30 day trial from Scribd

See all

Related Audiobooks

Free with a 30 day trial from Scribd

See all

Scaling Up Continuous Deployment

  1. 1. Scaling UpContinuous Deployment Timothy Fitz (.com)
  2. 2. “Continuous deployment involves deploying early and often, so as toavoid the pitfalls of "deployment hell".The practice aims to reduce timely rework and thus reduce cost and development time.”
  3. 3. Scalability• Maintaining availability, performance and happiness• As a function of # of people• As a function of # of tests
  4. 4. Availability• The build must always be green• Set a “green SLA” – 99% green – Never red for > 15m• Measure, track and report on these numbers
  5. 5. Performance• Measure (intent to) commit to time of deploy – Goal: < 5 minutes• Measure local development test loop – Goal: < 2s
  6. 6. Happiness• CI/CD System is a product• Software Engineers are the customer• Keep your customer happy!
  7. 7. Testing Pyramid
  8. 8. How do you make tests fast?• Tests can exercise large amounts of code without being slow• Minimize system calls (no I/O, no disk)• Minimize test data size• Make sure all systems are cheap to instantiate/teardown• No external state makes tests more reliable
  9. 9. Run Tests in Parallel• Multiprocess• Multimachine• Multi-VM• Instant multi-VM: http://circleci.com
  10. 10. Hardware Scale• CI Cluster will get huge – Function of cumulative engineering man-months – Rule of thumb: 10% of your cluster size• You will need a CI/CD DevOps person – CI cluster monitoring / alerting – Configuration Management critical
  11. 11. Scale testing infrastructure recap• Write the right kind of tests• Make those tests as fast as possible• Run those tests in parallel
  12. 12. People / Roles• Sheriff – Designated reverter / problem troubleshooter – Common pattern (IMVU, Chromium, Firefox)• CD “Product Owner” – Held accountable for SLA / Performance – Manage infrastructure backlog
  13. 13. Single trunk• Do this until it doesn’t work for you• Gets painful in the 16 – 32 developer range• Faster commit->deploy reduces the pain – But effort becomes prohibitive
  14. 14. “Try” pipeline• Conceptually, a second tree that “doesn’t matter” but still gets tested for feedback• Buildbot implements a patch-pushing version• Takes a significant amount of pressure off of trunk builds
  15. 15. CI Server takes active role• Server automatically reverts red commits• Server merges green commits to trunk
  16. 16. Feature branches• All incremental development happens on branches, branches land when feature is “ready”• If “feature” is kept small, can be 2-3 per engineer per week on average• Less continuous, but scales much better – Feature branches tested before merge
  17. 17. Merge tree• Tree per team / feature• Trees merged into trunk daily (if green)• Scale up via tree of trees (of trees…)• Again, less continuous
  18. 18. Federation• Each team gets their own deploy pipeline• Requires SOA / component architecture• Each team can set their own CD pace• “Enterprise Ready”
  19. 19. Recap• Single trunk + Try pipeline / Autorevert• Feature Branches• Merge Tree• Federation
  20. 20. Questions?Timothy Fitz (.com)
  • xbalaji

    Jan. 22, 2015

How do you continue to ship 50 times a day, when you're constantly hiring more engineers? How can you continue, when every day you write more tests that need to be run on every commit? This talk will cover how to scale up Continuous Integration and Continuous Deployment infrastracture, for teams as small as a handful of engineers and as large as hundreds of engineers.

Views

Total views

1,492

On Slideshare

0

From embeds

0

Number of embeds

24

Actions

Downloads

18

Shares

0

Comments

0

Likes

1

×