0
Continuous Deployment<br />Timothy Fitz<br />CTO of Canvas<br />
“Continuous integration involves integrating early and often, so as to avoid the pitfalls of "integration hell". The pract...
“Continuous deployment involves deploying early and often, so as to avoid the pitfalls of "deployment hell". The practice ...
The Vision<br />On every key press<br />Compile<br />Run automated tests<br />Deploy<br /><ul><li>“Live Coding”</li></li><...
The Reality<br />Change has risk<br />Infrastructure isn’t free<br />
At Canvas<br />Small CD shop<br />5 committers<br />Deploy process is “git push”<br />
At Scale: IMVU<br />Profitable<br />MMO + Virtual Economy Etsy<br />50+ Technical Staff<br />
At Scale: Etsy<br />
The Deploy Equation<br />Direct Value (DV)<br />Information Value (IV)<br />Deployment Risk<br />When IV + DV > Risk: Depl...
Increase Information Value<br />Small commits mean more information earlier<br />Implement features implementation-risk-fi...
Increase Direct Value<br />Feature shippable from day 0<br />Never blocked on deploy cycle<br />Higher velocity<br />Lean ...
Risk=Exposure * Probability * Severity<br />
Decrease Exposure<br />Dark launch non-frontend changes<br />Controlled exposure via feature rollout code<br />Expose to s...
Decrease Probability<br />Automated tests<br />Regression / Functional / Integration tests<br />Unit tests<br />Browser te...
Decrease Severity <br />Decrease length of degradation<br />Production Alerts<br /><ul><li>Cluster Immune System
Instant production roll back</li></ul>Decrease effects of degradation<br />Stability through isolation<br />Product level ...
FAQ<br />Whatabout shema changes?<br />Great, how do I get started?<br />
Schema Changes: They hate your code<br />Code and schema move in locked steps<br />Favor schemaless design<br />Minimize c...
Schema Changes: They hate your uptime<br />Did I mention schemalessdatabases yet?<br />Apply updates to standbys<br />Blue...
Great, how do I get started?<br />Nike method: Just do it<br />
tl;dr<br />We’ve come a long way<br />We have a long way to go<br />IV + DV > Exposure * Probability * Severity.<br />Reth...
Upcoming SlideShare
Loading in...5
×

Continuous Deployment

3,729

Published on

Published in: Technology, Business
0 Comments
3 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
3,729
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
26
Comments
0
Likes
3
Embeds 0
No embeds

No notes for slide
  • What? Why? Tailor for unique tradeoffs, focusing on risk mitigation.Trans: what is cd? What is CI?
  • Incremental logical step. Coined the term, didn’t invent the concept.(IMVU took it to the next level, but others were doing very similar things before IMVU existed)
  • Ancient concept:Expect it on local dev boxes (php-style)SqueakGenera : An entire OS, locally deployable instantly. Musical Live CodingTrans: So… why?
  • Story? System Chat. Threw away a week of work! Dark launch via CD.Bad Assumptions: Code does what it says Code doesn’t negatively affect other code Code is scalable Code handles edge cases correctly 3rd party API allows for certain behavior
  • “Continuous Deployment” coined August 2007, 20k google hits (phrase is only 4 years old)Feb 2009: Blog post lands
  • Where we were: 2008Handful of companiesZero documentationNo common languageWhat changed?Pattern got a name.People started talking about it.
  • Talk through commit deploy processGit pushGithubPing JenkinsJenkins runs python ./manage.py test canvasRuns through a couple hundred automated tests
  • Does it scale?IMVU: Profitable MMO + Virtual Economy, 50 person technical staffCommit-to-live in about 15 minutes.Massive cluster, massive parallelism. Extensive stats tracking.
  • “Gregory House Theory”Not a theory of CDWaterfallApply the theory on a per-change basis (schemas vsTrans: How to change that curve?
  • Hypothesis commits: What happens if I X?Pay attention to your S/N ratio. Refactoring in separate commits to pull out noise. (Behavior preserving? Behavior changing?)
  • Bank that IVLots of “Lean Startup” and “Lean Thinking” benefits, covered elsewhere.
  • Flickr switchesChrome dev-channel crashes constantly, but I still love it. (WebGL!)Google Labs
  • Website is down! Write tests.“Install the client” test, fear barrier.
  • Nagios alerts are an extension of test coverage.Cluster Immune is a hedge against the cost of big regressions. Does a subset of Nagios alerts with finely tuned parameters.Instant rollback (&lt;15s) is so critical. Human processes (how do I do it? Who does it? What’s a serious regression?)Just go read “Release It!” – Michael NygardCan’t take out a MySQL instance with a bad queryApp works even if search is downIsolation: Think AppEngine.Schemas: Lock down. Review. Try to avoid. Key-value store (NoSQL or YesSQL)
  • Schema changes are high friction; they’re often slow and expensive to deploy: most data stores fight the natural order of Continuous Deployment: small discrete schema changes.Everyone has established practices, patterns and unique situations given choice of database, Code works with schema v0 and v1 (i.e. adding a new row; code explicitly selects the columns it wants and ignores new row)Means you can always step the code and the database back (potentially multiple times, but that takes many steps)
  • Schema changes are computationally expensive and risky
  • Nike method: Just do itHot tub method: Ease into itNuclear option: quit your day job (amazon: switch teams)(Last one’s mostly a joke, but f you look at how the growth of agile methodologies: successful projects and dev environments attract talent; talent came from somewhere!)
  • Putting it all together, examples:Web Startup: forget riskEstablished Service: velocity  IMVU“Big Business”: CD to opt-in customers, daily deploy of baked functionality.Medical: CD to test-environment.Principles apply everywhere.
  • Transcript of "Continuous Deployment"

    1. 1. Continuous Deployment<br />Timothy Fitz<br />CTO of Canvas<br />
    2. 2. “Continuous integration involves integrating early and often, so as to avoid the pitfalls of "integration hell". The practice aims to reduce timely rework and thus reduce cost and development time.”<br />
    3. 3. “Continuous deployment involves deploying early and often, so as to avoid the pitfalls of "deployment hell". The practice aims to reduce timely rework and thus reduce cost and development time.”<br />
    4. 4. The Vision<br />On every key press<br />Compile<br />Run automated tests<br />Deploy<br /><ul><li>“Live Coding”</li></li></ul><li>Eliminate Waste<br /><ul><li>Deploying code validates assumptions</li></ul>Bad assumptions cause waste<br />Code built on top is waste<br />Designbuilt on top is waste<br />Thought built on top is waste<br />
    5. 5. The Reality<br />Change has risk<br />Infrastructure isn’t free<br />
    6. 6.
    7. 7. At Canvas<br />Small CD shop<br />5 committers<br />Deploy process is “git push”<br />
    8. 8. At Scale: IMVU<br />Profitable<br />MMO + Virtual Economy Etsy<br />50+ Technical Staff<br />
    9. 9. At Scale: Etsy<br />
    10. 10. The Deploy Equation<br />Direct Value (DV)<br />Information Value (IV)<br />Deployment Risk<br />When IV + DV > Risk: Deploy!<br />
    11. 11. Increase Information Value<br />Small commits mean more information earlier<br />Implement features implementation-risk-first<br />Conscious information gathering<br />
    12. 12. Increase Direct Value<br />Feature shippable from day 0<br />Never blocked on deploy cycle<br />Higher velocity<br />Lean Thinking<br />
    13. 13. Risk=Exposure * Probability * Severity<br />
    14. 14. Decrease Exposure<br />Dark launch non-frontend changes<br />Controlled exposure via feature rollout code<br />Expose to staff/QA only<br />Expose to opt-in beta testers<br />Gradually increase exposure from 1-100%<br />Feature-level rollback<br />
    15. 15. Decrease Probability<br />Automated tests<br />Regression / Functional / Integration tests<br />Unit tests<br />Browser tests / Click tests<br />3rd party integration tests<br />Manual QA prior to exposing features<br />Build code in a deploy mindset<br />
    16. 16. Decrease Severity <br />Decrease length of degradation<br />Production Alerts<br /><ul><li>Cluster Immune System
    17. 17. Instant production roll back</li></ul>Decrease effects of degradation<br />Stability through isolation<br />Product level fault tolerance<br />Lock down core infrastructure<br />
    18. 18. FAQ<br />Whatabout shema changes?<br />Great, how do I get started?<br />
    19. 19. Schema Changes: They hate your code<br />Code and schema move in locked steps<br />Favor schemaless design<br />Minimize classical schema changes<br />Offend DBAs with your lack of normalization<br />Lightweight/Schemaless databases(“nosql”)<br />
    20. 20. Schema Changes: They hate your uptime<br />Did I mention schemalessdatabases yet?<br />Apply updates to standbys<br />Blue/Green cluster setup<br />
    21. 21. Great, how do I get started?<br />Nike method: Just do it<br />
    22. 22. tl;dr<br />We’ve come a long way<br />We have a long way to go<br />IV + DV > Exposure * Probability * Severity.<br />Rethink schema changes<br />Continous Deployment: Just do it<br />Questions?<br />
    1. A particular slide catching your eye?

      Clipping is a handy way to collect important slides you want to go back to later.

    ×