: The Early YearsPros:Early Etsy engineersused this releasecycle to bootstrap themarketplace fromnothing.Forever grateful.
: The Early YearsCons:• Large changesets• Infrequent deploys• Weak conﬁdence in deploy success• Signiﬁcant time spent deploying• Low ability to experiment/iterate/react• Developer stress/unhappiness
: The Early YearsBy late 2008, Etsy is still a startup, but has the deploy process of a much bulkier company.Popularity is on the verge of outpacing capacity.
: Today1. Small changesets, deployed frequently
: Today1. Small changesets 2. Engineers deploy the site
: TodayAnd not just engineers, but also:• Designers• Product Folks• Upper Management• Board Members• Dogs
: Today1. Small changesets2. Engineers deploy 3. Deploys are fast and near-eﬀortless
: Today1. Small changesets2. Engineers deploy3. Deploys are fast 4. Most changes behind conﬁg ﬂags (safer deploys)
: Today1. Small changesets 4. Changes behind ﬂags2. Engineers deploy3. Deploys are fast 5. Graphs/metrics to assess deploy
: Today1. Small changesets 4. Changes behind ﬂags2. Engineers deploy 5. Copious graphs/metrics3. Deploys are fast6. If issues, ﬁx immediately & roll forward
: Today This isn’t license to break stuﬀ, quickly.Engineer-driven QA and solid unit testing are integral parts of the process.
: Today1. Small changesets 4. Changes behind ﬂags2. Engineers deploy 5. Copious graphs/metrics3. Deploys are fast 6. Fix fast & roll forward 7. Repeat 25+ times per day, every day
Then: 1. Weeks writing code 2. Painful merge 3. Hand oﬀ to deployers 4. Deploy, site down 5. Roll back deploy 6. Fix bugs, go to step 2Now: 1. Small changesets 2. Engineers deploy 3. Deploys are fast 4. Changes behind ﬂags 5. Copious graphs/metrics 6. Fix fast & roll forward
Etsy Deploy Stats: 2012• Deployed to production 6,419 times• On average, 535/month, 25/day• Additional 3,851 conﬁg-only deploys• 196 diﬀerent people deployed to prod• Nov/Dec 2012: deployed 752 times
Continuous Deployment Math • N = # of deploys • P = probability of site degradation • S = average severity of degradation • T = time to detect/resolve Expected = N*P*S*T Downtime
Continuous Deployment MathN = # of deploys S = avg. severity of degradationP = prob. of degradation T = time to detect/resolve Before: Now: •N=1 • N = 250 ↑↑↑↑ • P = 0.5 • P = 0.1 ↓ • S = 0.7 • S = 0.05 ↓↓ • T = 100 • T = 5 ↓↓↓ E.D. = 35 E.D. = 6.25 (all numbers completely arbitrary)
Big Takeaway Etsy circa 2013 (400+ employees)acts, in some ways, more like a startup than Etsy circa 2008 (40+ employees).
Continuous Deployment makes possible: “Continuous Experimentation” http://etsy.me/continuous-experimentation
Continuous Experimentation 1. Small changes 2. Run experiment (A/B test) 3. Analyze data 4. Re-examine assumptionsRepeat continuously in pursuit of larger goals.
Heard since 2010:“Neat experiment, but this will never scale.” As of 2013, Etsy has 100+ engineers - still going strong.
Some Etsy CustomizationsDeploying is a ﬁrst-class feature. Inability todeploy is a P1 incident (same as site down).
Some Etsy CustomizationsWe continuously deploy not just the main Etsy website, but as much as possible: • Internal admin site • API • Big data • Search • Blog • Deployinator itself
Some Etsy Customizations In the rare case we can’t continuously deploy, we create alternative tools: • Database schema changes • PCI-DSS environment (credit cards) We do continuously deploy as much of ourpayment processing as is safe & legal (98%).
Some Etsy CustomizationsKeeping deploys fast is paramount and worth the investment in manpower & hardware.
Some Etsy Customizations Continuous deployment is all about movingforward, sometimes at the expense of the past.Our solution: engineering-wide bug rotation,one day a month, every engineer participates.
Fun Fact: Continuous Deployment is a fantasticrecruitment tool for attracting engineerswho like to move fast and get stuﬀ done.
Learn more:http://codeascraft.etsy.com/Etsy open source (Deployinator, StatsD)http://etsy.github.com/Join the fun:http://www.etsy.com/careers