Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

The Hard Problems of Continuous Deployment


Published on

Published in: Technology
  • Login to see the comments

The Hard Problems of Continuous Deployment

  1. 1. The Hard Problemsof Continuous Deployment Timothy Fitz (.com)
  2. 2. “Continuous deployment involves deploying early and often, so as toavoid the pitfalls of "deployment hell".The practice aims to reduce timely rework and thus reduce cost and development time.”
  3. 3. “Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.”- Principles behind the Agile Manifesto (2001)
  4. 4. The Hard Problems (The ones I’m going to cover)• Data• Mobile• Scale
  5. 5. Data makes CD harder• Updating data is slow• Moving data is slow• Schemas live outside code deploy process• Rollbacks are often hard or impossible
  6. 6. Schema rev. N, N+1 compatible• Add the column in production• Push the code that writes to that column• Optionally, run a data migration to populate the existing rows with data• Push the code that reads from that column.
  7. 7. Apply only cheap changes• Only apply changes that are cheap enough to not affect live traffic• More complex changes split into tiny steps: – Create new table – Write to both – Cut over eventually – Drop old table
  8. 8. Apply change to standby• Run two DB instances• Apply change to standby• Failover if successfully applied• Might run 3rd db instance for availability
  9. 9. Blue/Green Deploy• Run two copies of entire cluster• All databases are replicated• Lets you test, update and rollback both code and schema in one step
  10. 10. Schemaless NoSQL• “Schemaless” really means schema is in your source code – Which is great! We can CD schema changes
  11. 11. Schemaless-style SQL• Benefits of existing SQL databases while still being pseudoschemaless• (thing_id, key, value) table• (guid, blob) table
  12. 12. Recap• Apply only cheap changes• Apply changes to standby• Blue/Green Deploy• Schemaless NoSQL• Schemaless-style SQL
  13. 13. Mobile• Users must opt-in to every update• iOS submissions take a week to be approved• Luckily, lots of tools aimed at the space
  14. 14. Remember the basics• CI server / automated tests are critical• Can’t fallback on production alarms / rollback• Hosted options:
  15. 15. Data-drive everything• Build your views / content from data files• Ping server for updates• Hosted: – Lets you deploy txt, png and xib dynamically!
  16. 16. 99% HTML• Entire app is a single UIWebView• Glue native code to allow access to APIs• is awesome (and FOSS now) – Live reloading for local dev – Streamlined deploys –
  17. 17. Hybrid HTML/Native• Core app is native• Sections can be replaced by HTML – i.e. Facebook stream entries fallback to HTML• Infrequently used sections are 100% HTML
  18. 18. Recap• Remember the basics• Data-drive everything• 99% HTML• Hybrid HTML/Native
  19. 19. Scalability• Maintaining availability, performance and happiness• As a function of # of people• As a function of # of tests
  20. 20. Availability• The build must always be green• Set a “green SLA” – 99% green – Never red for > 15m• Measure, track and report on these numbers
  21. 21. Performance• Measure (intent to) commit to time of deploy – Goal: < 5 minutes• Measure local development test loop – Goal: < 2s
  22. 22. Happiness• CI/CD System is a product• Software Engineers are the customer• Keep your customer happy!
  23. 23. Testing Pyramid
  24. 24. How do you make tests fast?• Tests can exercise large amounts of code without being slow• Minimize system calls (no I/O, no disk)• Minimize test data size• Make sure all systems are cheap to instantiate/teardown• No external state makes tests more reliable
  25. 25. Run Tests in Parallel• Multiprocess• Multimachine• Multi-VM• Instant multi-VM:
  26. 26. Hardware Scale• CI Cluster will get huge – Function of cumulative engineering man-months – Rule of thumb: 10% of your cluster size• You will need a CI/CD DevOps person – CI cluster monitoring / alerting – Configuration Management critical
  27. 27. Scale testing infrastructure recap• Write the right kind of tests• Make those tests as fast as possible• Run those tests in parallel
  28. 28. People / Roles• Sheriff – Designated reverter / problem troubleshooter – Common pattern (IMVU, Chromium, Firefox)• CD “Product Owner” – Held accountable for SLA / Performance – Manage infrastructure backlog
  29. 29. Single trunk• Do this until it doesn’t work for you• Gets painful in the 16 – 32 developer range• Faster commit->deploy reduces the pain – But effort becomes prohibitive
  30. 30. “Try” pipeline• Conceptually, a second tree that “doesn’t matter” but still gets tested for feedback• Buildbot implements a patch-pushing version• Takes a significant amount of pressure off of trunk builds
  31. 31. CI Server takes active role• Server automatically reverts red commits• Server merges green commits to trunk
  32. 32. Feature branches• All incremental development happens on branches, branches land when feature is “ready”• If “feature” is kept small, can be 2-3 per engineer per week on average• Less continuous, but scales much better – Feature branches tested before merge
  33. 33. Merge tree• Tree per team / feature• Trees merged into trunk daily (if green)• Scale up via tree of trees (of trees…)• Again, less continuous
  34. 34. Federation• Each team gets their own deploy pipeline• Requires SOA / component architecture• Each team can set their own CD pace• “Enterprise Ready”
  35. 35. Recap• Single trunk + Try pipeline / Autorevert• Feature Branches• Merge Tree• Federation
  36. 36. Questions?Timothy Fitz (.com)