Your SlideShare is downloading. ×
Mapping Etsy's Front-end
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×

Saving this for later?

Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime - even offline.

Text the download link to your phone

Standard text messaging rates apply

Mapping Etsy's Front-end

489
views

Published on

Video and slides synchronized, mp3 and slide download available at URL http://bit.ly/1smUK82. …

Video and slides synchronized, mp3 and slide download available at URL http://bit.ly/1smUK82.

Daniel Espeset talks about how Etsy built an incremental compiler for the JavaScript modules, and used it to see how static assets are compiled, being deployed, and loaded. Filmed at qconlondon.com.

Daniel Espeset is a Software Engineer working on the Performance team at Etsy. He enjoys hacking on build processes, improving front-end architecture and pursuing moonshot experiments in code instrumentation and static analysis.

Published in: Technology

0 Comments
4 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
489
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
0
Comments
0
Likes
4
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Friday, March 7, 14
  • 2. InfoQ.com: News & Community Site • 750,000 unique visitors/month • Published in 4 languages (English, Chinese, Japanese and Brazilian Portuguese) • Post content from our QCon conferences • News 15-20 / week • Articles 3-4 / week • Presentations (videos) 12-15 / week • Interviews 2-3 / week • Books 1 / month Watch the video with slide synchronization on InfoQ.com! http://www.infoq.com/presentations /javascript-compiler-deploy
  • 3. Presented at QCon London www.qconlondon.com Purpose of QCon - to empower software development by facilitating the spread of knowledge and innovation Strategy - practitioner-driven conference designed for YOU: influencers of change and innovation in your teams - speakers and topics driving the evolution and innovation - connecting and catalyzing the influencers and innovators Highlights - attended by more than 12,000 delegates since 2007 - held in 9 cities worldwide
  • 4. Mapping Etsy’s Frontend Tools and Techniques for Managing Complexity at Scale @danielespeset Friday, March 7, 14
  • 5. @danielespeset Seth Walker Allison McKnight Jonathan Klein @sethwalker @aemcknig @jonathanklein Daniel Espeset @danielespeset Brought to you by Friday, March 7, 14
  • 6. @danielespeset Etsy Friday, March 7, 14
  • 7. @danielespeset A global marketplace for buying and selling unique goods. Etsy Friday, March 7, 14
  • 8. @danielespeset Friday, March 7, 14
  • 9. @danielespeset Friday, March 7, 14
  • 10. @danielespeset roughly 180 engineers Friday, March 7, 14
  • 11. @danielespeset 50 deploys a day Friday, March 7, 14
  • 12. @danielespeset Deployinator IRC Push Queue Friday, March 7, 14
  • 13. @danielespeset Anybody can touch anything. Friday, March 7, 14
  • 14. @danielespeset Anybody can touch anything. try not to break the site Friday, March 7, 14
  • 15. @danielespeset Continuous Experimentation Friday, March 7, 14
  • 16. @danielespeset Engineers own their code through the entire stack. Friday, March 7, 14
  • 17. @danielespeset Bootcamps Friday, March 7, 14
  • 18. @danielespeset Our frontend is made up of 300 different “pages” Friday, March 7, 14
  • 19. @danielespeset At any given time we may be running dozens of experiments per page Friday, March 7, 14
  • 20. @danielespeset 2 12 4,096= Friday, March 7, 14
  • 21. @danielespeset 2 12 300 = 1,228,800x Friday, March 7, 14
  • 22. @danielespeset OMG, how do actually you do anything? Friday, March 7, 14
  • 23. @danielespeset 12 month growth of CSS code last year 992 files 197,812 lines last week 1,653 files (1.67x) 319,152 lines (1.61x) Friday, March 7, 14
  • 24. @danielespeset 12 month growth of JavaScript code last year 1,451 files 425,915 lines last week 2,394 files (1.65x) 639,484 lines (1.5x) Friday, March 7, 14
  • 25. @danielespeset Friday, March 7, 14
  • 26. @danielespeset sup? Friday, March 7, 14
  • 27. @danielespeset html * #buttons button.button { modules/buttons.css Friday, March 7, 14
  • 28. @danielespeset html * #buttons button.button { wtf? modules/buttons.css Friday, March 7, 14
  • 29. @danielespeset $css/> grep “@import modules/buttons” Friday, March 7, 14
  • 30. @danielespeset $css/> grep “@import modules/buttons” a/browse.css a/taxeditor.css a/taxonomies.css a/weddings.css b/modules-responsive.css b/modules.css a/orders/refund.css a/scout/scout.css c/account/credit-card.css c/bill/autobilling.css c/bill/index.css c/bill/payments.css c/shop/registration/billing.css Friday, March 7, 14
  • 31. @danielespeset $css/> for ... grep “@import $1” ... a/browse.css a/taxeditor.css a/taxonomies.css a/weddings.css b/modules-responsive.css b/modules.css a/orders/refund.css a/scout/scout.css c/account/credit-card.css c/bill/autobilling.css c/bill/index.css c/bill/payments.css c/shop/registration/billing.css base-responsive base base_after_inline Friday, March 7, 14
  • 32. @danielespeset $css/> for ... grep “@import $1” ... a/browse.css a/taxeditor.css a/taxonomies.css a/weddings.css b/modules-responsive.css b/modules.css a/orders/refund.css a/scout/scout.css c/account/credit-card.css c/bill/autobilling.css c/bill/index.css c/bill/payments.css c/shop/registration/billing.css /base-responsive /base /base_after_inline c/template_editor c/mini a/scout/dashboard Friday, March 7, 14
  • 33. @danielespeset $templates/> for ... grep “{% css=$1” ... a/browse.css a/taxeditor.css a/taxonomies.css a/weddings.css b/modules-responsive.css b/modules.css a/orders/refund.css a/scout/scout.css c/account/credit-card.css c/bill/autobilling.css c/bill/index.css c/bill/payments.css c/shop/registration/billing.css /base-responsive /base /base_after_inline c/template_editor c/mini a/scout/dashboard wholesale/modules/vendor_avatar wholesale/modules/address_from_array wholesale/pages/purchase_order/modules/uneditable_fields/payment_method wholesale/pages/purchase_order/modules/uneditable_fields/payment_terms wholesale/pages/purchase_order/modules/uneditable_fields/ship_date wholesale/pages/purchase_order/modules/uneditable_fields/billing_address wholesale/pages/purchase_order/modules/uneditable_fields/shipping_address wholesale/pages/purchase_order/modules/product_grid modules/time_utils base Friday, March 7, 14
  • 34. @danielespeset $templates/> for ... grep “{% css=$1” ... a/browse.css a/taxeditor.css a/taxonomies.css a/weddings.css b/modules-responsive.css b/modules.css a/orders/refund.css a/scout/scout.css c/account/credit-card.css c/bill/autobilling.css c/bill/index.css c/bill/payments.css c/shop/registration/billing.css /base-responsive /base /base_after_inline c/template_editor c/mini a/scout/dashboard wholesale/modules/vendor_avatar wholesale/modules/address_from_array wholesale/pages/purchase_order/modules/uneditable_fields/payment_method wholesale/pages/purchase_order/modules/uneditable_fields/payment_terms wholesale/pages/purchase_order/modules/uneditable_fields/ship_date wholesale/pages/purchase_order/modules/uneditable_fields/billing_address wholesale/pages/purchase_order/modules/uneditable_fields/shipping_address wholesale/pages/purchase_order/modules/product_grid modules/time_utils base modules/deferred_clientlogger modules/jquery modules/async_analytics_init notice/experiment_notice modules/locale/dismissible_link pages/recommendations/friends/v2/modules/gift-ideas-reminder header pages/translator/_modules/md5_overlay modules/breadcrumbs pages/browse/_modules/footer footer Friday, March 7, 14
  • 35. @danielespeset $templates/> for ... grep “{% css=$1” ... a/browse.css a/taxeditor.css a/taxonomies.css a/weddings.css b/modules-responsive.css b/modules.css a/orders/refund.css a/scout/scout.css c/account/credit-card.css c/bill/autobilling.css c/bill/index.css c/bill/payments.css c/shop/registration/billing.css /base-responsive /base /base_after_inline c/template_editor c/mini a/scout/dashboard wholesale/modules/vendor_avatar wholesale/modules/address_from_array wholesale/pages/purchase_order/modules/uneditable_fields/payment_method wholesale/pages/purchase_order/modules/uneditable_fields/payment_terms wholesale/pages/purchase_order/modules/uneditable_fields/ship_date wholesale/pages/purchase_order/modules/uneditable_fields/billing_address wholesale/pages/purchase_order/modules/uneditable_fields/shipping_address wholesale/pages/purchase_order/modules/product_grid modules/time_utils base modules/deferred_clientlogger modules/jquery modules/async_analytics_init notice/experiment_notice modules/locale/dismissible_link pages/recommendations/friends/v2/modules/gift-ideas-reminder header pages/translator/_modules/md5_overlay modules/breadcrumbs pages/browse/_modules/footer footer Some/Controller/Class.php Other/Controller/Class.php Foo/Controller/Class.php Bar/Controller/Class.php Baz/Controller/Class.php Batman/Controller/Class.php ... Friday, March 7, 14
  • 36. @danielespeset $templates/> for ... grep “{% css=$1” ... a/browse.css a/taxeditor.css a/taxonomies.css a/weddings.css b/modules-responsive.css b/modules.css a/orders/refund.css a/scout/scout.css c/account/credit-card.css c/bill/autobilling.css c/bill/index.css c/bill/payments.css c/shop/registration/billing.css /base-responsive /base /base_after_inline c/template_editor c/mini a/scout/dashboard wholesale/modules/vendor_avatar wholesale/modules/address_from_array wholesale/pages/purchase_order/modules/uneditable_fields/payment_method wholesale/pages/purchase_order/modules/uneditable_fields/payment_terms wholesale/pages/purchase_order/modules/uneditable_fields/ship_date wholesale/pages/purchase_order/modules/uneditable_fields/billing_address wholesale/pages/purchase_order/modules/uneditable_fields/shipping_address wholesale/pages/purchase_order/modules/product_grid modules/time_utils base modules/deferred_clientlogger modules/jquery modules/async_analytics_init notice/experiment_notice modules/locale/dismissible_link pages/recommendations/friends/v2/modules/gift-ideas-reminder header pages/translator/_modules/md5_overlay modules/breadcrumbs pages/browse/_modules/footer footer Some/Controller/Class.php Other/Controller/Class.php Foo/Controller/Class.php Bar/Controller/Class.php Baz/Controller/Class.php Batman/Controller/Class.php ... htdocs/my_endpoint.php htdocs/foo.php htdocs/other_endpoint.php htdocs/baz/index.php htdocs/index.php Friday, March 7, 14
  • 37. @danielespeset $templates/> for ... grep “{% css=$1” ... a/browse.css a/taxeditor.css a/taxonomies.css a/weddings.css b/modules-responsive.css b/modules.css a/orders/refund.css a/scout/scout.css c/account/credit-card.css c/bill/autobilling.css c/bill/index.css c/bill/payments.css c/shop/registration/billing.css /base-responsive /base /base_after_inline c/template_editor c/mini a/scout/dashboard wholesale/modules/vendor_avatar wholesale/modules/address_from_array wholesale/pages/purchase_order/modules/uneditable_fields/payment_method wholesale/pages/purchase_order/modules/uneditable_fields/payment_terms wholesale/pages/purchase_order/modules/uneditable_fields/ship_date wholesale/pages/purchase_order/modules/uneditable_fields/billing_address wholesale/pages/purchase_order/modules/uneditable_fields/shipping_address wholesale/pages/purchase_order/modules/product_grid modules/time_utils base modules/deferred_clientlogger modules/jquery modules/async_analytics_init notice/experiment_notice modules/locale/dismissible_link pages/recommendations/friends/v2/modules/gift-ideas-reminder header pages/translator/_modules/md5_overlay modules/breadcrumbs pages/browse/_modules/footer footer Some/Controller/Class.php Other/Controller/Class.php Foo/Controller/Class.php Bar/Controller/Class.php Baz/Controller/Class.php Batman/Controller/Class.php ... htdocs/my_endpoint.php htdocs/foo.php htdocs/other_endpoint.php htdocs/baz/index.php htdocs/index.php RewriteRule ^(.*)/ve/(.*)$ $1/voluntary-employee/$2 [L,R=301] RewriteRule ^(.*)/hsa/(.*)$ $1/health-saving-account/$2 [L,R=301]   RewriteCond %{REQUEST_FILENAME} !-f # Existing File RewriteCond %{REQUEST_FILENAME} !-d # Existing Directory RewriteRule . /index.php [L]   RewriteCond %{HTTP_REFERER} !^$ RewriteCond %{HTTP_REFERER} !^http://(subdomain.)?domain.tld/.*$ [NC] RewriteRule ^.*.(bmp|tif|gif|jpg|jpeg|jpe|png)$ - [F] Friday, March 7, 14
  • 38. @danielespeset CC Image courtesy of Mark Skipper (bitterjug) on Flickr Halp! Friday, March 7, 14
  • 39. @danielespeset Reasoning about our frontend is hard. Friday, March 7, 14
  • 40. @danielespeset Deleting code is scary. Friday, March 7, 14
  • 41. @danielespeset Testing changes is difficult. Friday, March 7, 14
  • 42. @danielespeset Mental overhead is a performance problem. Friday, March 7, 14
  • 43. @danielespeset What are our options? • Static Analysis • Synthetic Testing • Access Logs • Dynamic Analysis Friday, March 7, 14
  • 44. @danielespeset What are our options? • Static Analysis • Synthetic Testing • Access Logs • Dynamic Analysis Friday, March 7, 14
  • 45. @danielespeset What are our options? • Static Analysis • Synthetic Testing • Access Logs • Dynamic Analysis Friday, March 7, 14
  • 46. @danielespeset What are our options? • Static Analysis • Synthetic Testing • Access Logs • Dynamic Analysis Friday, March 7, 14
  • 47. @danielespeset What are our options? • Static Analysis • Synthetic Testing • Access Logs • Dynamic Analysis Friday, March 7, 14
  • 48. @danielespeset What are our options? • Static Analysis • Synthetic Testing • Access Logs • Dynamic Analysis Friday, March 7, 14
  • 49. Builda Friday, March 7, 14
  • 50. @danielespeset A brief history of JS loading at Etsy • <script> Friday, March 7, 14
  • 51. @danielespeset A brief history of JS loading at Etsy • <script> • Etsy.loader Friday, March 7, 14
  • 52. @danielespeset A brief history of JS loading at Etsy • <script> • Etsy.loader • //= require Friday, March 7, 14
  • 53. @danielespeset Builda Loading JS in development assets/js <script src="foo.js"> builda.php?name=foo.js processing Etsy.loader // require Friday, March 7, 14
  • 54. @danielespeset Building JS in deploys bin/builda Builda Friday, March 7, 14
  • 55. @danielespeset Builda Building JS in deploys bin/builda assets/js dist/js processing Etsy.loader // require compiled output Friday, March 7, 14
  • 56. @danielespeset Builda dist/js Building JS in deploys bin/builda assets/js processing Etsy.loader // require compiled output Friday, March 7, 14
  • 57. @danielespeset Builda dist/js Building JS in deploys bin/builda assets/js processing Etsy.loader // require compiled output Friday, March 7, 14
  • 58. @danielespeset Builda Building JS in deploys bin/builda assets/js dist/js Minification Friday, March 7, 14
  • 59. @danielespeset bin/builda assets/jsBuilda Building JS in deploys dist/js Minification Friday, March 7, 14
  • 60. @danielespeset Building JS in deploys bin/builda assets/js dist/js WWW Builda Friday, March 7, 14
  • 61. @danielespeset A brief history of JS loading at Etsy • <script> • Etsy.loader • //= require • AMD (requirejs) Friday, March 7, 14
  • 62. @danielespeset Asyncronous Module Definition Friday, March 7, 14
  • 63. define(function() { return function Lightsaber(color) { this.color = color; }; }); lightsaber.js Friday, March 7, 14
  • 64. define(function() { return function Lightsaber(color) { this.color = color; }; }); lightsaber.js define(['lightsaber'], function(Lightsaber) { return function Luke() { this.lightsaber = new Lightsaber('green'); }; }); luke.js Friday, March 7, 14
  • 65. define(function() { return function Lightsaber(color) { this.color = color; }; }); lightsaber.js define(['lightsaber'], function(Lightsaber) { return function Luke() { this.lightsaber = new Lightsaber('green'); }; }); luke.js require(['plot','luke','vader'], function(plot, Luke, Vader) { var luke = new Luke(), vader = new Vader(); plot.exploreFatherIssues(luke, vader); }); starwars.js Friday, March 7, 14
  • 66. @danielespeset define(function() { return function Lightsaber(color) { this.color = color; }; }); lightsaber.js define(['lightsaber'], function(Lightsaber) { return function Luke() { this.lightsaber = new Lightsaber('green'); }; }); luke.js require(['plot','luke','vader'], function(plot, Luke, Vader) { var luke = new Luke(), vader = new Vader(); plot.exploreFatherIssues(luke, vader); }); starwars.js <script type="text/javascript" src="starwars.js"></script> Builda HTTP GET starwars.js Async loading AMD in development Friday, March 7, 14
  • 67. @danielespeset require(['plot','luke','vader'], function(plot, Luke, Vader) { define(function() { return function Lightsaber(color) { this.color = color; }; }); lightsaber.js define(['lightsaber'], function(Lightsaber) { return function Luke() { this.lightsaber = new Lightsaber('green'); }; }); luke.js Async loading AMD in development Builda uilda HTTP GET luke.js Friday, March 7, 14
  • 68. @danielespeset define(function() { return function Lightsaber(color) { this.color = color; }; }); lightsaber.js define(['lightsaber'], function(Lightsaber) { Builda HTTP GET lightsaber.js Async loading AMD in development Friday, March 7, 14
  • 69. @danielespeset Async loading AMD in development Friday, March 7, 14
  • 70. Incorporating AMD into Builda, phase 1 Friday, March 7, 14
  • 71. @danielespeset Building JS with AMD in deploys bin/builda Builda assets/js dist/js r.jsr.jsr.jsr.js Friday, March 7, 14
  • 72. define(function() { return function Lightsaber(color) { this.color = color; }; }); lightsaber.js define(['lightsaber'], function(Lightsaber) { return function Luke() { this.lightsaber = new Lightsaber('green'); }; }); luke.js require(['plot','luke','vader'], function(plot, Luke, Vader) { var luke = new Luke(), vader = new Vader(); plot.exploreFatherIssues(luke, vader); }); starwars.js Friday, March 7, 14
  • 73. // more modules... starwars.12345678901234.js define('lightsaber', function() { return function Lightsaber(color) { this.color = color; }; }); define('luke', ['lightsaber'], function(Lightsaber) { return function Luke() { this.lightsaber = new Lightsaber('green'); }; }); require(['plot','luke','vader'], function(plot, Luke, Vader) { var luke = new Luke(), vader = new Vader(); plot.exploreFatherIssues(luke, vader); }); // more modules... Friday, March 7, 14
  • 74. This broke dev-prod environment parity Friday, March 7, 14
  • 75. Also it was crazy slow Friday, March 7, 14
  • 76. Builda Revised Friday, March 7, 14
  • 77. @danielespeset AKA taking the A out of AMD Friday, March 7, 14
  • 78. @danielespeset assets/js dist/js Building JS in development Builda Friday, March 7, 14
  • 79. @danielespeset assets/js Save foo.js Building JS in development Buildadist/js inotify IN_CLOSE_WRITE foo.js "foo": { "dependency_of": ["bar", "baz"] } Friday, March 7, 14
  • 80. @danielespeset Builda assets/js dist/js AMD (r.js) Etsy.loader // require foo.js Building JS in development Friday, March 7, 14
  • 81. @danielespeset Builda assets/js dist/js AMD (r.js) Etsy.loader // require bar.js Building JS in development Friday, March 7, 14
  • 82. @danielespeset Builda assets/js dist/js AMD (r.js) Etsy.loader // require baz.js Building JS in development Friday, March 7, 14
  • 83. @danielespeset assets/js dist/js Building JS in development Builda HTTP GET foo.js Friday, March 7, 14
  • 84. @danielespeset assets/js dist/js Building JS in development Builda Friday, March 7, 14
  • 85. Ranger Friday, March 7, 14
  • 86. @danielespeset Jason Huff Aaron Moodie @jsnhff @aaronmoodie Friday, March 7, 14
  • 87. @danielespeset Friday, March 7, 14
  • 88. @danielespeset CSS Friday, March 7, 14
  • 89. @danielespeset Friday, March 7, 14
  • 90. @danielespeset Friday, March 7, 14
  • 91. @danielespeset Friday, March 7, 14
  • 92. @danielespeset Pages Friday, March 7, 14
  • 93. @danielespeset Friday, March 7, 14
  • 94. @danielespeset Friday, March 7, 14
  • 95. @danielespeset JavaScript Friday, March 7, 14
  • 96. @danielespeset Friday, March 7, 14
  • 97. @danielespeset Friday, March 7, 14
  • 98. @danielespeset Ranger is an API Lodge is a client Friday, March 7, 14
  • 99. @danielespeset Ranger Friday, March 7, 14
  • 100. @danielespeset Ranger Friday, March 7, 14
  • 101. @danielespeset log host Ranger WWW Friday, March 7, 14
  • 102. @danielespeset Ranger log host WWW Friday, March 7, 14
  • 103. @danielespeset Ranger rollup linter hinter escomplex scp stuff log host WWW Friday, March 7, 14
  • 104. @danielespeset Ranger rollup linter hinter escomplex scp stuff log host WWW Friday, March 7, 14
  • 105. @danielespeset Now we know enough to surface dead files with confidence Friday, March 7, 14
  • 106. @danielespeset Finding dead code paths is still hard Friday, March 7, 14
  • 107. Shrinkray Friday, March 7, 14
  • 108. @danielespeset Justin Donato Kelly Norton @justindo @kellegous Friday, March 7, 14
  • 109. @danielespeset Friday, March 7, 14
  • 110. @danielespeset What’s next? • Adding Templates as a resource Friday, March 7, 14
  • 111. @danielespeset What’s next? • Adding Templates as a resource • Shrinkray powered delete button Friday, March 7, 14
  • 112. @danielespeset What’s next? • Adding Templates as a resource • Shrinkray powered delete button • Detecting inefficiently grouped code Friday, March 7, 14
  • 113. @danielespeset What’s next? • Adding Templates as a resource • Shrinkray powered delete button • Detecting inefficiently grouped code • Feature / User-state asset signatures Friday, March 7, 14
  • 114. @danielespeset What’s next? • Adding Templates as a resource • Shrinkray powered delete button • Detecting inefficiently grouped code • Feature / User-state asset signatures • Shrinkray for JavaScript Friday, March 7, 14
  • 115. @danielespeset The frontend is getting more complex everywhere. Friday, March 7, 14
  • 116. @danielespeset We need new strategies to manage and understand it. Friday, March 7, 14
  • 117. @danielespeset The browser is part of your distributed system, not just a client you support. Friday, March 7, 14
  • 118. @danielespeset The more extensible a tool the more useful it will be. Friday, March 7, 14
  • 119. @danielespeset Thanks! codeascraft.com @codeascraft etsy.github.io etsy.com/codeascraft/talks etsy.com/careers Friday, March 7, 14
  • 120. Friday, March 7, 14
  • 121. Watch the video with slide synchronization on InfoQ.com! http://www.infoq.com/presentations/javascript -compiler-deploy