Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Scale
Deployment
continuous
at
@premshree
1.7 million active sellers
26.1 million active buyers
$2.39 billion GMS (2015)
921 employees
Premshree Pillai
Sr. Engineer, Etsy
hello!
principles
q & a
tooling + culture
principles
just ship
enable innovation
enable innovation
(you are creating products, not code)
optimize for purpose
optimize for purpose
optimize for autonomy
optimize for mastery
experimentation
experimentation
A/B testing
quick iteration
quick iteration
(improving products/features)
quick iteration
(fail fast > stagnant code)
continuous improvement
low MTTR
continuous deployment
continuous delivery
/
commit
commit build
commit build tests
commit build tests user tests
commit build tests user tests release
commit build tests user tests release
commit build tests user tests release
tooling
frequent check-ins
frequent check-ins
(to master!)
frequent check-ins
branching in code
Feature
Feature
github.com/etsy/feature
$server_config['my_feature'] = [
'enabled' => 'on'
];
$server_config['my_feature'] = [
'enabled' => 'off'
];
$server_config['my_feature'] = [
'enabled' => 1
];
$server_config['my_feature'] = [
'enabled' => 1,
'bucketing' => 'user'
];
if (Feature::isEnabled('my_feature') {
}
if (Feature::isEnabledFor('my_feature', $user) {
}
Experimentation
Experimentation
A/B testing
$server_config['my_feature'] = [
'enabled' => 'on'
];
$server_config['my_feature'] = [
'enabled' => [
'layout1' => 1,
'layout2' => 3,
'layout3' => 3
]
];
if (Feature::isEnabled('my_feature')) {
switch (Feature::variant('my_feature')) {
case 'layout1':
// layout1 code
break;
c...
continuous integration
continuous integration
(and automated tests)
keep the build green
ready to release
ready to release
anytime
commit build tests user tests release
try
Try
Try
(before you commit)
TryLib
TryLib
github.com/etsy/TryLib
commit build tests user tests release
try deployinator
Deployinator
Deployinator
github.com/etsy/deployinator
commit build tests user tests release
try deployinator ci/jenkins
commit build tests user tests release
try deployinator ci/jenkins manual tests
commit build tests user tests release
try deployinator ci/jenkins manual tests deploy
commit build tests user tests release
commit build tests user tests princess prod
anybody can push
http://www.flickr.com/photos/zsqr0000/5269002895
be anxious
… but do not fear
dev
/
ops
♥dev
/
ops
dark changes
template changes
css tweaks
unreferenced code
config pushes
$server_config['my_feature'] = [
'enabled' => 1,
'bucketing' => 'user'
];
$server_config['my_feature'] = [
'enabled' => 50,
'bucketing' => 'user'
];
push train
premshree> .join
john
john + premshree
premshree>
john + premshree
premshree> .good
john + premshree
premshree> .bueno
john + premshree*
premshree>
john + premshree*
sally> .join
john + premshree* | sally
sally>
push train
(with PushBot)
push train
github.com/etsy/pushbot
post-deploy
post-deploy
(gaining confidence)
supergrep
supergrep
github.com/etsy/supergrep
dashboards
dashboards
(deploy)
push
dashboards
(app)
StatsD + Graphite
StatsD + Graphite
github.com/etsy/statsd
summary
pre-flight check
ready to push
ready to deploy
testing/verification
confidence
pre-flight check
ready to push
ready to deploy
testing/verification
confidence
try/git hooks
push train/irc
deployinator
u...
https://www.flickr.com/photos/saschaaa/152502539/
poka-yoke
culture
assume best intentions
cultivate empathy
open
failure is an option
failure is an option
(but not our intention)
failure is an option
(we strongly want NOT to fail)
failure is an opportunity
post-mortems
blameless post-mortems
+ remediation
morgue
morgue
github.com/etsy/morgue
mixer
mixer
github.com/etsy/mixer
culture + tooling
deploy
fail ok
post-mortem
remediation
stimulii
deploys
30+ deploys
30+ config deploys
codeascraft.com
fin
thank you!
questions?
@premshreepolkadotsocks.net
Continuous Deployment at Scale, PHPConfAsia 2016
Continuous Deployment at Scale, PHPConfAsia 2016
Continuous Deployment at Scale, PHPConfAsia 2016
Continuous Deployment at Scale, PHPConfAsia 2016
Continuous Deployment at Scale, PHPConfAsia 2016
Continuous Deployment at Scale, PHPConfAsia 2016
Continuous Deployment at Scale, PHPConfAsia 2016
Continuous Deployment at Scale, PHPConfAsia 2016
Continuous Deployment at Scale, PHPConfAsia 2016
Continuous Deployment at Scale, PHPConfAsia 2016
Continuous Deployment at Scale, PHPConfAsia 2016
Continuous Deployment at Scale, PHPConfAsia 2016
Continuous Deployment at Scale, PHPConfAsia 2016
Continuous Deployment at Scale, PHPConfAsia 2016
Continuous Deployment at Scale, PHPConfAsia 2016
Continuous Deployment at Scale, PHPConfAsia 2016
Continuous Deployment at Scale, PHPConfAsia 2016
Continuous Deployment at Scale, PHPConfAsia 2016
Continuous Deployment at Scale, PHPConfAsia 2016
Continuous Deployment at Scale, PHPConfAsia 2016
Upcoming SlideShare
Loading in …5
×

Continuous Deployment at Scale, PHPConfAsia 2016

908 views

Published on

At Etsy our approach to development is to make small, incremental, continuous changes to the site. Deploying code to production anywhere between 20 to 40 times a day requires not only technical tooling, but also a culture that allows for and encourages innovation, confidence backed by actual data and comprehensive tests, and blameless post-mortems that allow for a feedback loop we can learn from and improve.

Learn about the tools we have, and how we foster an environment where we make this possible.

Published in: Technology
  • Login to see the comments

  • Be the first to like this

Continuous Deployment at Scale, PHPConfAsia 2016

  1. 1. Scale Deployment continuous at @premshree
  2. 2. 1.7 million active sellers 26.1 million active buyers $2.39 billion GMS (2015) 921 employees
  3. 3. Premshree Pillai Sr. Engineer, Etsy
  4. 4. hello!
  5. 5. principles q & a tooling + culture
  6. 6. principles
  7. 7. just ship
  8. 8. enable innovation
  9. 9. enable innovation (you are creating products, not code)
  10. 10. optimize for purpose
  11. 11. optimize for purpose
  12. 12. optimize for autonomy
  13. 13. optimize for mastery
  14. 14. experimentation
  15. 15. experimentation A/B testing
  16. 16. quick iteration
  17. 17. quick iteration (improving products/features)
  18. 18. quick iteration (fail fast > stagnant code)
  19. 19. continuous improvement
  20. 20. low MTTR
  21. 21. continuous deployment continuous delivery /
  22. 22. commit
  23. 23. commit build
  24. 24. commit build tests
  25. 25. commit build tests user tests
  26. 26. commit build tests user tests release
  27. 27. commit build tests user tests release
  28. 28. commit build tests user tests release
  29. 29. tooling
  30. 30. frequent check-ins
  31. 31. frequent check-ins (to master!)
  32. 32. frequent check-ins branching in code
  33. 33. Feature
  34. 34. Feature github.com/etsy/feature
  35. 35. $server_config['my_feature'] = [ 'enabled' => 'on' ];
  36. 36. $server_config['my_feature'] = [ 'enabled' => 'off' ];
  37. 37. $server_config['my_feature'] = [ 'enabled' => 1 ];
  38. 38. $server_config['my_feature'] = [ 'enabled' => 1, 'bucketing' => 'user' ];
  39. 39. if (Feature::isEnabled('my_feature') { }
  40. 40. if (Feature::isEnabledFor('my_feature', $user) { }
  41. 41. Experimentation
  42. 42. Experimentation A/B testing
  43. 43. $server_config['my_feature'] = [ 'enabled' => 'on' ];
  44. 44. $server_config['my_feature'] = [ 'enabled' => [ 'layout1' => 1, 'layout2' => 3, 'layout3' => 3 ] ];
  45. 45. if (Feature::isEnabled('my_feature')) { switch (Feature::variant('my_feature')) { case 'layout1': // layout1 code break; case 'layout2': // layout2 code break; } }
  46. 46. continuous integration
  47. 47. continuous integration (and automated tests)
  48. 48. keep the build green
  49. 49. ready to release
  50. 50. ready to release anytime
  51. 51. commit build tests user tests release try
  52. 52. Try
  53. 53. Try (before you commit)
  54. 54. TryLib
  55. 55. TryLib github.com/etsy/TryLib
  56. 56. commit build tests user tests release try deployinator
  57. 57. Deployinator
  58. 58. Deployinator github.com/etsy/deployinator
  59. 59. commit build tests user tests release try deployinator ci/jenkins
  60. 60. commit build tests user tests release try deployinator ci/jenkins manual tests
  61. 61. commit build tests user tests release try deployinator ci/jenkins manual tests deploy
  62. 62. commit build tests user tests release
  63. 63. commit build tests user tests princess prod
  64. 64. anybody can push
  65. 65. http://www.flickr.com/photos/zsqr0000/5269002895
  66. 66. be anxious
  67. 67. … but do not fear
  68. 68. dev / ops
  69. 69. ♥dev / ops
  70. 70. dark changes
  71. 71. template changes css tweaks unreferenced code
  72. 72. config pushes
  73. 73. $server_config['my_feature'] = [ 'enabled' => 1, 'bucketing' => 'user' ];
  74. 74. $server_config['my_feature'] = [ 'enabled' => 50, 'bucketing' => 'user' ];
  75. 75. push train
  76. 76. premshree> .join john
  77. 77. john + premshree premshree>
  78. 78. john + premshree premshree> .good
  79. 79. john + premshree premshree> .bueno
  80. 80. john + premshree* premshree>
  81. 81. john + premshree* sally> .join
  82. 82. john + premshree* | sally sally>
  83. 83. push train (with PushBot)
  84. 84. push train github.com/etsy/pushbot
  85. 85. post-deploy
  86. 86. post-deploy (gaining confidence)
  87. 87. supergrep
  88. 88. supergrep github.com/etsy/supergrep
  89. 89. dashboards
  90. 90. dashboards (deploy)
  91. 91. push
  92. 92. dashboards (app)
  93. 93. StatsD + Graphite
  94. 94. StatsD + Graphite github.com/etsy/statsd
  95. 95. summary
  96. 96. pre-flight check ready to push ready to deploy testing/verification confidence
  97. 97. pre-flight check ready to push ready to deploy testing/verification confidence try/git hooks push train/irc deployinator user testing supergrep/dashboards
  98. 98. https://www.flickr.com/photos/saschaaa/152502539/
  99. 99. poka-yoke
  100. 100. culture
  101. 101. assume best intentions
  102. 102. cultivate empathy
  103. 103. open
  104. 104. failure is an option
  105. 105. failure is an option (but not our intention)
  106. 106. failure is an option (we strongly want NOT to fail)
  107. 107. failure is an opportunity
  108. 108. post-mortems
  109. 109. blameless post-mortems
  110. 110. + remediation
  111. 111. morgue
  112. 112. morgue github.com/etsy/morgue
  113. 113. mixer
  114. 114. mixer github.com/etsy/mixer
  115. 115. culture + tooling deploy fail ok post-mortem remediation stimulii
  116. 116. deploys
  117. 117. 30+ deploys
  118. 118. 30+ config deploys
  119. 119. codeascraft.com
  120. 120. fin
  121. 121. thank you!
  122. 122. questions? @premshreepolkadotsocks.net

×