Scale
Deployment
continuous
at
@premshree
1.7 million active sellers
26.1 million active buyers
$2.39 billion GMS (2015)
921 employees
Premshree Pillai
Sr. Engineer, Etsy
hello!
principles
q & a
tooling + culture
principles
just ship
enable innovation
enable innovation
(you are creating products, not code)
optimize for purpose
optimize for purpose
optimize for autonomy
optimize for mastery
experimentation
experimentation
A/B testing
quick iteration
quick iteration
(improving products/features)
quick iteration
(fail fast > stagnant code)
continuous improvement
low MTTR
continuous deployment
continuous delivery
/
commit
commit build
commit build tests
commit build tests user tests
commit build tests user tests release
commit build tests user tests release
commit build tests user tests release
tooling
frequent check-ins
frequent check-ins
(to master!)
frequent check-ins
branching in code
Feature
Feature
github.com/etsy/feature
$server_config['my_feature'] = [
'enabled' => 'on'
];
$server_config['my_feature'] = [
'enabled' => 'off'
];
$server_config['my_feature'] = [
'enabled' => 1
];
$server_config['my_feature'] = [
'enabled' => 1,
'bucketing' => 'user'
];
if (Feature::isEnabled('my_feature') {
}
if (Feature::isEnabledFor('my_feature', $user) {
}
Experimentation
Experimentation
A/B testing
$server_config['my_feature'] = [
'enabled' => 'on'
];
$server_config['my_feature'] = [
'enabled' => [
'layout1' => 1,
'layout2' => 3,
'layout3' => 3
]
];
if (Feature::isEnabled('my_feature')) {
switch (Feature::variant('my_feature')) {
case 'layout1':
// layout1 code
break;
case 'layout2':
// layout2 code
break;
}
}
continuous integration
continuous integration
(and automated tests)
keep the build green
ready to release
ready to release
anytime
commit build tests user tests release
try
Try
Try
(before you commit)
TryLib
TryLib
github.com/etsy/TryLib
commit build tests user tests release
try deployinator
Deployinator
Deployinator
github.com/etsy/deployinator
commit build tests user tests release
try deployinator ci/jenkins
commit build tests user tests release
try deployinator ci/jenkins manual tests
commit build tests user tests release
try deployinator ci/jenkins manual tests deploy
commit build tests user tests release
commit build tests user tests princess prod
anybody can push
http://www.flickr.com/photos/zsqr0000/5269002895
be anxious
… but do not fear
dev
/
ops
♥dev
/
ops
dark changes
template changes
css tweaks
unreferenced code
config pushes
$server_config['my_feature'] = [
'enabled' => 1,
'bucketing' => 'user'
];
$server_config['my_feature'] = [
'enabled' => 50,
'bucketing' => 'user'
];
push train
premshree> .join
john
john + premshree
premshree>
john + premshree
premshree> .good
john + premshree
premshree> .bueno
john + premshree*
premshree>
john + premshree*
sally> .join
john + premshree* | sally
sally>
push train
(with PushBot)
push train
github.com/etsy/pushbot
post-deploy
post-deploy
(gaining confidence)
supergrep
supergrep
github.com/etsy/supergrep
dashboards
dashboards
(deploy)
push
dashboards
(app)
StatsD + Graphite
StatsD + Graphite
github.com/etsy/statsd
summary
pre-flight check
ready to push
ready to deploy
testing/verification
confidence
pre-flight check
ready to push
ready to deploy
testing/verification
confidence
try/git hooks
push train/irc
deployinator
user testing
supergrep/dashboards
https://www.flickr.com/photos/saschaaa/152502539/
poka-yoke
culture
assume best intentions
cultivate empathy
open
failure is an option
failure is an option
(but not our intention)
failure is an option
(we strongly want NOT to fail)
failure is an opportunity
post-mortems
blameless post-mortems
+ remediation
morgue
morgue
github.com/etsy/morgue
mixer
mixer
github.com/etsy/mixer
culture + tooling
deploy
fail ok
post-mortem
remediation
stimulii
deploys
30+ deploys
30+ config deploys
codeascraft.com
fin
thank you!
questions?
@premshreepolkadotsocks.net

Continuous Deployment at Scale, PHPConfAsia 2016