Scaling Deployment at Etsy

  • 718 views
Uploaded on

How deployment is tackled at Etsy and how we keep it fast and scale it out with a growing engineering team. …

How deployment is tackled at Etsy and how we keep it fast and scale it out with a growing engineering team.

Reprised version of the talk given at the Continuous Delivery NYC meetup October 2013.

More in: Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
718
On Slideshare
0
From Embeds
0
Number of Embeds
2

Actions

Shares
Downloads
14
Comments
0
Likes
1

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Scaling Deployment at Etsy Daniel Schauenberg dschauenberg@etsy.com @mrtazz Thursday, October 10, 13
  • 2. Thursday, October 10, 13
  • 3. August 2013 •1.8 billion page views •5,483,399 items sold •$109.1 million of goods sold •> 30 million members •> 1 million active shops http://www.etsy.com/blog/news/2013/etsy-statistics-august-2013-weather-report/ | Items by RockerDollJewellery, ZulamimiLand, codice, 42Things Thursday, October 10, 13
  • 4. LAMMP Item by TheBackPackShoppe Thursday, October 10, 13
  • 5. Item by FrankelPhotos Monolithic App Thursday, October 10, 13
  • 6. No Branching Item by NurseryWallArt Thursday, October 10, 13
  • 7. Deploy Frequency Thursday, October 10, 13
  • 8. First Day Item by flowersandfleurons Thursday, October 10, 13
  • 9. Thursday, October 10, 13
  • 10. IRC Thursday, October 10, 13
  • 11. Thursday, October 10, 13
  • 12. DeveloperVMs •KVM •Dev version of full Etsy stack •Chef •DevTools Thursday, October 10, 13
  • 13. Thursday, October 10, 13
  • 14. Item by codecards Thursday, October 10, 13
  • 15. % review -r dschauenberg Thursday, October 10, 13
  • 16. automatically assignedautomatically assigned Thursday, October 10, 13
  • 17. Try Item by CSSDesign Thursday, October 10, 13
  • 18. Actually ... Thursday, October 10, 13
  • 19. The Bobs Item by Signz Thursday, October 10, 13
  • 20. The Bobs •LXC containers on buildtests •Multiple SSDs •Labels for heavy/any execution •One heavy executor per disk Thursday, October 10, 13
  • 21. CI/Try •~260 Bobs •Mostly for try •Constant monitoring for slow tests Thursday, October 10, 13
  • 22. push train Item by decomodwalls Thursday, October 10, 13
  • 23. #push •IRC channel to organize push trains •Join a train if you want to deploy changes •Schedule is planned via the channel topic •First in the train is the driver (controls the deploy) •Opening hours: 7am - 10pm NYC time Thursday, October 10, 13
  • 24. #push <prod> kseever* + jameslee | jpaul | DanielConvissor (c) Thursday, October 10, 13
  • 25. #push bateman* + krunal* + enorris* | tristan (c) + jameslee (c) + jlaster (c) | dawa + corey + sandosh + jklein + magera + seth_home + mpascual + nathan | bateman | russp (c) Thursday, October 10, 13
  • 26. pushbot •.join •.in •.good •.done Thursday, October 10, 13
  • 27. pushbot Thursday, October 10, 13
  • 28. Item by EsalonPhotography Thursday, October 10, 13
  • 29. Deployinator Thursday, October 10, 13
  • 30. Thursday, October 10, 13
  • 31. Thursday, October 10, 13
  • 32. stale commits Thursday, October 10, 13
  • 33. version checks buttons disabledbuttons disabled Thursday, October 10, 13
  • 34. version checks Thursday, October 10, 13
  • 35. lock down deploys Thursday, October 10, 13
  • 36. https://github.com/etsy/deployinator Thursday, October 10, 13
  • 37. Downsides •Deploys not atomic on the request level •Limbo during the time of the local rsync •Common strategy was to split commits into 3 deploys Thursday, October 10, 13
  • 38. Item by Geographicsart Thursday, October 10, 13
  • 39. Atomic Deploys Thursday, October 10, 13
  • 40. Basic Idea Yin Yang Active Docroot Thursday, October 10, 13
  • 41. Basic Idea Yin Yang Active Docroot rsync Thursday, October 10, 13
  • 42. Basic Idea Yin Yang Active Docroot Thursday, October 10, 13
  • 43. Basic Idea Yin Yang Active Docroot Thursday, October 10, 13
  • 44. Problems •Symlink swap during requests •Code needs to be guaranteed to finish on the docroot it started •Code inclusion mid request Thursday, October 10, 13
  • 45. etsy/mod_realdoc •Apache post_read_request hook •Whole request works on realpath of docroot •Caches realpath for 2s Thursday, October 10, 13
  • 46. ini_set('include_path', $_SERVER['DOCUMENT_ROOT'].'/../include'); Thursday, October 10, 13
  • 47. etsy/incpath •PHP module to set the incpath •Gets docroot from Apache or realpath() itself •Looks for a pattern to replace in include_path •Restores include_path at the end of the request Thursday, October 10, 13
  • 48. What did we get? •Remove functions and call site in same deploy •No restarts necessary •Opcode caches stay warm for files that don’t change between 2 deploys Thursday, October 10, 13
  • 49. Things to watch out for •Code that uses full path names to scripts •Atomic symlink swapping with `mv -T` •Realpath caching to not stress the filesystem •Opcode cache needs to fit 2x code size •Only request atomicity Thursday, October 10, 13
  • 50. The Plateau Item by finandfancy Thursday, October 10, 13
  • 51. The Plateau •Regular deploys took ~15 mins •Config deploys about half •10am - 6pm => ~ 32 deploys •Long waiting times Thursday, October 10, 13
  • 52. Item by KlaireWarren Thursday, October 10, 13
  • 53. Waiting for push queueWaiting for push queue Thursday, October 10, 13
  • 54. Split The Queues Item by KlaireWarren Thursday, October 10, 13
  • 55. HELLO SPLIT QUEUESHELLO SPLIT QUEUES Thursday, October 10, 13
  • 56. Dashboards deploy linesdeploy lines Thursday, October 10, 13
  • 57. Supergrep Thursday, October 10, 13
  • 58. Thursday, October 10, 13
  • 59. Summary •Current setup has scaled to ~150 people •Constantly trying to improve the speed of deployment •Find weak parts in the process and make them more robust/faster •Bring Dev closer to Prod •Not being able to deploy has the same status as the site being down Thursday, October 10, 13
  • 60. codeascraft.etsy.com www.etsy.com/codeascraft/talks etsy.github.com www.etsy.com/careers Thursday, October 10, 13
  • 61. Questions? Thursday, October 10, 13
  • 62. Scaling Deployment at Etsy Daniel Schauenberg dschauenberg@etsy.com @mrtazz Thursday, October 10, 13