Scaling Deployment at Etsy

Scaling Deployment at Etsy
Daniel Schauenberg
dschauenberg@etsy.com
@mrtazz
Thursday, October 10, 13

August 2013
•1.8 billion page views
•5,483,399 items sold
•$109.1 million of goods sold
•> 30 million members
•> 1 million active shops
http://www.etsy.com/blog/news/2013/etsy-statistics-august-2013-weather-report/ | Items by RockerDollJewellery, ZulamimiLand, codice, 42Things

LAMMP
Item by TheBackPackShoppe

Item by FrankelPhotos
Monolithic App

No Branching
Item by NurseryWallArt

Deploy Frequency

First Day
Item by ﬂowersandﬂeurons

DeveloperVMs
•KVM
•Dev version of full Etsy stack
•Chef
•DevTools

Item by codecards

% review -r dschauenberg

automatically assignedautomatically assigned

Try
Item by CSSDesign

Actually ...

The Bobs
Item by Signz

The Bobs
•LXC containers on buildtests
•Multiple SSDs
•Labels for heavy/any execution
•One heavy executor per disk

CI/Try
•~260 Bobs
•Mostly for try
•Constant monitoring for slow tests

push train
Item by decomodwalls

#push
•IRC channel to organize push trains
•Join a train if you want to deploy changes
•Schedule is planned via the channel topic
•First in the train is the driver (controls the deploy)
•Opening hours: 7am - 10pm NYC time

#push
<prod> kseever* + jameslee | jpaul | DanielConvissor (c)

#push
bateman* + krunal* + enorris* | tristan (c) + jameslee (c)
+ jlaster (c) | dawa + corey + sandosh + jklein + magera
+ seth_home + mpascual + nathan | bateman | russp (c)

pushbot
•.join
•.in
•.good
•.done

pushbot

Item by EsalonPhotography

Deployinator

stale commits

version checks
buttons disabledbuttons disabled

version checks

lock down deploys

https://github.com/etsy/deployinator

Downsides
•Deploys not atomic on the request level
•Limbo during the time of the local rsync
•Common strategy was to split commits into 3 deploys

Item by Geographicsart

Atomic Deploys

Basic Idea
Yin Yang
Active Docroot

Basic Idea
Yin Yang
Active Docroot
rsync

Problems
•Symlink swap during requests
•Code needs to be guaranteed to ﬁnish on the docroot
it started
•Code inclusion mid request

etsy/mod_realdoc
•Apache post_read_request hook
•Whole request works on realpath of docroot
•Caches realpath for 2s

ini_set('include_path', $_SERVER['DOCUMENT_ROOT'].'/../include');

etsy/incpath
•PHP module to set the incpath
•Gets docroot from Apache or realpath() itself
•Looks for a pattern to replace in include_path
•Restores include_path at the end of the request

What did we get?
•Remove functions and call site in same deploy
•No restarts necessary
•Opcode caches stay warm for ﬁles that don’t change
between 2 deploys

Things to watch out for
•Code that uses full path names to scripts
•Atomic symlink swapping with `mv -T`
•Realpath caching to not stress the ﬁlesystem
•Opcode cache needs to ﬁt 2x code size
•Only request atomicity

The Plateau
Item by ﬁnandfancy

The Plateau
•Regular deploys took ~15 mins
•Conﬁg deploys about half
•10am - 6pm => ~ 32 deploys
•Long waiting times

Item by KlaireWarren

Waiting for push queueWaiting for push queue

Split The Queues
Item by KlaireWarren

HELLO SPLIT QUEUESHELLO SPLIT QUEUES

Dashboards
deploy linesdeploy lines

Supergrep

Summary
•Current setup has scaled to ~150 people
•Constantly trying to improve the speed of deployment
•Find weak parts in the process and make them more
robust/faster
•Bring Dev closer to Prod
•Not being able to deploy has the same status as the site
being down

codeascraft.etsy.com
www.etsy.com/codeascraft/talks
etsy.github.com
www.etsy.com/careers

Questions?

Scaling Deployment at Etsy

More Related Content

What's hot

Similar to Scaling Deployment at Etsy

More from Daniel Schauenberg

Recently uploaded

Scaling Deployment at Etsy