Everything you ever wanted to know about deployment but were afraid to ask
Upcoming SlideShare
Loading in...5
×
 

Everything you ever wanted to know about deployment but were afraid to ask

on

  • 1,715 views

 

Statistics

Views

Total Views
1,715
Views on SlideShare
1,691
Embed Views
24

Actions

Likes
0
Downloads
21
Comments
0

2 Embeds 24

http://lanyrd.com 23
https://si0.twimg.com 1

Accessibility

Categories

Upload Details

Uploaded via as Apple Keynote

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n

Everything you ever wanted to know about deployment but were afraid to ask Everything you ever wanted to know about deployment but were afraid to ask Presentation Transcript

  • Everything you ever wantedto know about deployment ...but were afraid to ask Laura Thomson laura@mozilla.com @lxt 1
  • DisclaimersNot about toolsNot prescriptiveRecognize where you are, and where youwant to be 2
  • Models 3
  • Maturity model(after Capability Maturity Model, CMU)1. Initial: “chaotic”, “individual heroics”2. Repeatable: documented3. Defined: standard process, some tuning4. Managed: measured5. Optimizing: continual improvement, innovation 4
  • InitialStartup phase of many projectsLong termPush code whenever you feel like itDevs push codeNot a lot of tests, automation, or verification 5
  • RepeatableOften after a 1.0, first non-beta ship, or firstship with a significant number of usersSome kind of written processPush when a feature is done: less often thaninitially, typically 6
  • DefinedProcedural documentationStart of automationOften done by a sysadmin 7
  • ManagedAutomationTools: packagingVerification post-pushMeasurement: How often do we push? Howlong does it take? How did that push affectperformance? 8
  • OptimizedTake the drama out of deploymentOften - not essentially - continuousdeploymentTypically a lot of test automationLightweight 9
  • How much do we ship?Start with per-patch pushesMove to featuresThen to releasesThen back to featuresThe back to per-patch pushes 10
  • Per-patchFeatures Features Releases 11
  • Velocity modelsCritical massSingle hard deadlineTrain modelContinuous deployment 12
  • Critical mass“enough stuff to release”MVP 13
  • Single hard deadlineSupport for X by date YShipping to a marketing planHard deadlines are hard 14
  • Train modelRelease every WednesdayWhatever’s ready to ship, shipsAnything else catches the next train 15
  • Continuous deploymentShip each change as soon as it’s doneContinuous is kind of a misnomer;deployment is discrete 16
  • Tools and practices 17
  • Source controlStable vs unstableBranch per bug, branch per feature“git flow” is overkill, but you need a processIf it’s not per-patch-push, tag what you pushOpen source needs ESRs even if you’re highvelocity 18
  • Dev EnvsDev’s laptop is a horrible environmentVMs can be hard to maintainDevelopment databases are hard: fake data, minidbsDevelopment API sandboxLightweight set up and tear down VMs“Development” staging server (unstable) 19
  • Staging EnvsStaging environment MUST REFLECTPRODUCTIONSame versions, same proportions: a scalemodelRealistic traffic and load (scale)Staging must be monitoredStaging must have managed configuration 20
  • One Box FailStaging needs to be more than one boxIf you have multiple databases or webheadsor whatever in prod...you need that instaging 21
  • Continuous IntegrationBuild-on-commitVM-per-buildRun all unit tests(Auto) push build to stagingRun more tests (acceptance/UI) 22
  • TestingUnit tests: run locally, run on buildAcceptance/User tests: run against browser(Selenium or whatever)Load tests: how does it perform under prodload?Smoke tests: what’s the maximum load wecan support with this build? 23
  • Deployment toolsIt doesn’t really matter what you useAutomate itDo it the same way in staging and productionUse configuration management to deployconfig changes and manage yourplatform...the same way in staging andproduction 24
  • QAFeature tests on unstableFull tests on stageFull tests on production (verification) 25
  • MeasurementMonitoringPerformance testingInstrument, instrument, instrumentIs it actually possible to have too much data?(Hint: yes. But only if no insight) 26
  • PostmortemsWhat went rightWhat went wrongNo finger pointing 27
  • When things go wrong 28
  • Quantum of deployment(via Erik Kastner)“What’s the smallest number of steps,with the smallest number of peopleand the smallest amount of ceremony requiredto get new code running on your servers?” http://codeascraft.etsy.com/2010/05/20/quantum-of-deployment/, 29
  • ChemspillsEven if you have heavyweight/non-automateddeployments, what does a chemspill looklike? 30
  • THIS IS NOT A DRILL 31
  • Fail forwardFail forward: the premise thatMean Time To Repairis the key measure, not MTBF 32
  • FailSometimes you can’t fail forwardExample: intractable/unforeseen performanceproblem, hardware failures, datacentermigrationsSometimes it happens.Upper time limits: failing forward is takingtoo long 33
  • RollbackGoing back to the last known goodHaving a known process for rollback is justas important as having a known process fordeploymentPractice rollbacks 34
  • Decision pointsWhen shipping something new, define somerules and decision pointsIf it passes this test/performance criteriawe’ll ship itIf these things go wrong we’ll roll backMake these rules beforehand, while headsare calm 35
  • Feature switchesA nicer alternative to rollbackTurn a feature on for a subset of users: betausers, developers, n% of usersTurn it on for everybodyTurn it off if you’re having problems orunexpected load: “load shedding” 36
  • ContinuousDeployment 37
  • What is CD?Total misnomerNot continuous, discreteAutomated not automatic, generallyIntention is push-per-changeUsually driven by a Big Red Button 38
  • Technical requirementsContinuous integration with build-on-commitTests with good coverage, and a good feel for theholes in coverageA staging environment that reflects productionManaged configurationScripted single button deployment to a large numberof machines 39
  • People and processHigh levels of trustRealistic risk assessment and toleranceExcellent code reviewExcellent source code managementTracking, trending, monitoring 40
  • Testing vs monitoringRun tests against productionContinuous testing = one kind of monitoringTesting is an important monitorYou need other monitorsYou need tests too 41
  • You should build the capability forcontinuous deploymenteven if you never intend to docontinuous deployment. 42
  • The only way to get good atdeployment is to deploy a lot. 43
  • Questions? laura@mozilla.com 44