Everything you ever wantedto know about deployment  ...but were afraid to ask           Laura Thomson         laura@mozill...
DisclaimersNot about toolsNot prescriptiveRecognize where you are, and where youwant to be                   2
Models  3
Maturity model(after Capability Maturity Model, CMU)1. Initial: “chaotic”, “individual heroics”2. Repeatable: documented3....
InitialStartup phase of many projectsLong termPush code whenever you feel like itDevs push codeNot a lot of tests, automat...
RepeatableOften after a 1.0, first non-beta ship, or firstship with a significant number of usersSome kind of written process...
DefinedProcedural documentationStart of automationOften done by a sysadmin                      7
ManagedAutomationTools: packagingVerification post-pushMeasurement: How often do we push? Howlong does it take? How did tha...
OptimizedTake the drama out of deploymentOften - not essentially - continuousdeploymentTypically a lot of test automationL...
How much do we ship?Start with per-patch pushesMove to featuresThen to releasesThen back to featuresThe back to per-patch ...
Per-patchFeatures               Features           Releases              11
Velocity modelsCritical massSingle hard deadlineTrain modelContinuous deployment                       12
Critical mass“enough stuff to release”MVP                    13
Single hard deadlineSupport for X by date YShipping to a marketing planHard deadlines are hard                    14
Train modelRelease every WednesdayWhatever’s ready to ship, shipsAnything else catches the next train                    15
Continuous         deploymentShip each change as soon as it’s doneContinuous is kind of a misnomer;deployment is discrete ...
Tools and practices         17
Source controlStable vs unstableBranch per bug, branch per feature“git flow” is overkill, but you need a processIf it’s not...
Dev EnvsDev’s laptop is a horrible environmentVMs can be hard to maintainDevelopment databases are hard: fake data, minidb...
Staging EnvsStaging environment MUST REFLECTPRODUCTIONSame versions, same proportions: a scalemodelRealistic traffic and lo...
One Box FailStaging needs to be more than one boxIf you have multiple databases or webheadsor whatever in prod...you need ...
Continuous IntegrationBuild-on-commitVM-per-buildRun all unit tests(Auto) push build to stagingRun more tests (acceptance/...
TestingUnit tests: run locally, run on buildAcceptance/User tests: run against browser(Selenium or whatever)Load tests: ho...
Deployment toolsIt doesn’t really matter what you useAutomate itDo it the same way in staging and productionUse configurati...
QAFeature tests on unstableFull tests on stageFull tests on production (verification)                      25
MeasurementMonitoringPerformance testingInstrument, instrument, instrumentIs it actually possible to have too much data?(H...
PostmortemsWhat went rightWhat went wrongNo finger pointing                    27
When things go wrong         28
Quantum of          deployment(via Erik Kastner)“What’s the smallest number of steps,with the smallest number of peopleand...
ChemspillsEven if you have heavyweight/non-automateddeployments, what does a chemspill looklike?                   30
THIS IS NOT A DRILL        31
Fail forwardFail forward: the premise thatMean Time To Repairis the key measure, not MTBF                     32
FailSometimes you can’t fail forwardExample: intractable/unforeseen performanceproblem, hardware failures, datacentermigra...
RollbackGoing back to the last known goodHaving a known process for rollback is justas important as having a known process...
Decision pointsWhen shipping something new, define somerules and decision pointsIf it passes this test/performance criteria...
Feature switchesA nicer alternative to rollbackTurn a feature on for a subset of users: betausers, developers, n% of users...
ContinuousDeployment    37
What is CD?Total misnomerNot continuous, discreteAutomated not automatic, generallyIntention is push-per-changeUsually dri...
Technical requirementsContinuous integration with build-on-commitTests with good coverage, and a good feel for theholes in...
People and processHigh levels of trustRealistic risk assessment and toleranceExcellent code reviewExcellent source code ma...
Testing vs monitoringRun tests against productionContinuous testing = one kind of monitoringTesting is an important monito...
You should build the capability forcontinuous deploymenteven if you never intend to docontinuous deployment.              ...
The only way to get good atdeployment is to deploy a lot.                 43
Questions? laura@mozilla.com       44
Upcoming SlideShare
Loading in …5
×

Everything you ever wanted to know about deployment but were afraid to ask

1,861 views
1,734 views

Published on

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
1,861
On SlideShare
0
From Embeds
0
Number of Embeds
25
Actions
Shares
0
Downloads
22
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • Everything you ever wanted to know about deployment but were afraid to ask

    1. 1. Everything you ever wantedto know about deployment ...but were afraid to ask Laura Thomson laura@mozilla.com @lxt 1
    2. 2. DisclaimersNot about toolsNot prescriptiveRecognize where you are, and where youwant to be 2
    3. 3. Models 3
    4. 4. Maturity model(after Capability Maturity Model, CMU)1. Initial: “chaotic”, “individual heroics”2. Repeatable: documented3. Defined: standard process, some tuning4. Managed: measured5. Optimizing: continual improvement, innovation 4
    5. 5. InitialStartup phase of many projectsLong termPush code whenever you feel like itDevs push codeNot a lot of tests, automation, or verification 5
    6. 6. RepeatableOften after a 1.0, first non-beta ship, or firstship with a significant number of usersSome kind of written processPush when a feature is done: less often thaninitially, typically 6
    7. 7. DefinedProcedural documentationStart of automationOften done by a sysadmin 7
    8. 8. ManagedAutomationTools: packagingVerification post-pushMeasurement: How often do we push? Howlong does it take? How did that push affectperformance? 8
    9. 9. OptimizedTake the drama out of deploymentOften - not essentially - continuousdeploymentTypically a lot of test automationLightweight 9
    10. 10. How much do we ship?Start with per-patch pushesMove to featuresThen to releasesThen back to featuresThe back to per-patch pushes 10
    11. 11. Per-patchFeatures Features Releases 11
    12. 12. Velocity modelsCritical massSingle hard deadlineTrain modelContinuous deployment 12
    13. 13. Critical mass“enough stuff to release”MVP 13
    14. 14. Single hard deadlineSupport for X by date YShipping to a marketing planHard deadlines are hard 14
    15. 15. Train modelRelease every WednesdayWhatever’s ready to ship, shipsAnything else catches the next train 15
    16. 16. Continuous deploymentShip each change as soon as it’s doneContinuous is kind of a misnomer;deployment is discrete 16
    17. 17. Tools and practices 17
    18. 18. Source controlStable vs unstableBranch per bug, branch per feature“git flow” is overkill, but you need a processIf it’s not per-patch-push, tag what you pushOpen source needs ESRs even if you’re highvelocity 18
    19. 19. Dev EnvsDev’s laptop is a horrible environmentVMs can be hard to maintainDevelopment databases are hard: fake data, minidbsDevelopment API sandboxLightweight set up and tear down VMs“Development” staging server (unstable) 19
    20. 20. Staging EnvsStaging environment MUST REFLECTPRODUCTIONSame versions, same proportions: a scalemodelRealistic traffic and load (scale)Staging must be monitoredStaging must have managed configuration 20
    21. 21. One Box FailStaging needs to be more than one boxIf you have multiple databases or webheadsor whatever in prod...you need that instaging 21
    22. 22. Continuous IntegrationBuild-on-commitVM-per-buildRun all unit tests(Auto) push build to stagingRun more tests (acceptance/UI) 22
    23. 23. TestingUnit tests: run locally, run on buildAcceptance/User tests: run against browser(Selenium or whatever)Load tests: how does it perform under prodload?Smoke tests: what’s the maximum load wecan support with this build? 23
    24. 24. Deployment toolsIt doesn’t really matter what you useAutomate itDo it the same way in staging and productionUse configuration management to deployconfig changes and manage yourplatform...the same way in staging andproduction 24
    25. 25. QAFeature tests on unstableFull tests on stageFull tests on production (verification) 25
    26. 26. MeasurementMonitoringPerformance testingInstrument, instrument, instrumentIs it actually possible to have too much data?(Hint: yes. But only if no insight) 26
    27. 27. PostmortemsWhat went rightWhat went wrongNo finger pointing 27
    28. 28. When things go wrong 28
    29. 29. Quantum of deployment(via Erik Kastner)“What’s the smallest number of steps,with the smallest number of peopleand the smallest amount of ceremony requiredto get new code running on your servers?” http://codeascraft.etsy.com/2010/05/20/quantum-of-deployment/, 29
    30. 30. ChemspillsEven if you have heavyweight/non-automateddeployments, what does a chemspill looklike? 30
    31. 31. THIS IS NOT A DRILL 31
    32. 32. Fail forwardFail forward: the premise thatMean Time To Repairis the key measure, not MTBF 32
    33. 33. FailSometimes you can’t fail forwardExample: intractable/unforeseen performanceproblem, hardware failures, datacentermigrationsSometimes it happens.Upper time limits: failing forward is takingtoo long 33
    34. 34. RollbackGoing back to the last known goodHaving a known process for rollback is justas important as having a known process fordeploymentPractice rollbacks 34
    35. 35. Decision pointsWhen shipping something new, define somerules and decision pointsIf it passes this test/performance criteriawe’ll ship itIf these things go wrong we’ll roll backMake these rules beforehand, while headsare calm 35
    36. 36. Feature switchesA nicer alternative to rollbackTurn a feature on for a subset of users: betausers, developers, n% of usersTurn it on for everybodyTurn it off if you’re having problems orunexpected load: “load shedding” 36
    37. 37. ContinuousDeployment 37
    38. 38. What is CD?Total misnomerNot continuous, discreteAutomated not automatic, generallyIntention is push-per-changeUsually driven by a Big Red Button 38
    39. 39. Technical requirementsContinuous integration with build-on-commitTests with good coverage, and a good feel for theholes in coverageA staging environment that reflects productionManaged configurationScripted single button deployment to a large numberof machines 39
    40. 40. People and processHigh levels of trustRealistic risk assessment and toleranceExcellent code reviewExcellent source code managementTracking, trending, monitoring 40
    41. 41. Testing vs monitoringRun tests against productionContinuous testing = one kind of monitoringTesting is an important monitorYou need other monitorsYou need tests too 41
    42. 42. You should build the capability forcontinuous deploymenteven if you never intend to docontinuous deployment. 42
    43. 43. The only way to get good atdeployment is to deploy a lot. 43
    44. 44. Questions? laura@mozilla.com 44

    ×