Video and slides synchronized, mp3 and slide download available at URL http://bit.ly/1smUK82. Daniel Espeset talks about how Etsy built an incremental compiler for the JavaScript modules, and used it to see how static assets are compiled, being deployed, and loaded. Filmed at qconlondon.com. Daniel Espeset is a Software Engineer working on the Performance team at Etsy. He enjoys hacking on build processes, improving front-end architecture and pursuing moonshot experiments in code instrumentation and static analysis.

1. Friday, March 7, 14

2. InfoQ.com: News & Community Site
• 750,000 unique visitors/month
• Published in 4 languages (English, Chinese, Japanese and Brazilian
Portuguese)
• Post content from our QCon conferences
• News 15-20 / week
• Articles 3-4 / week
• Presentations (videos) 12-15 / week
• Interviews 2-3 / week
• Books 1 / month
Watch the video with slide
synchronization on InfoQ.com!
http://www.infoq.com/presentations
/javascript-compiler-deploy

3. Presented at QCon London
www.qconlondon.com
Purpose of QCon
- to empower software development by facilitating the spread of
knowledge and innovation
Strategy
- practitioner-driven conference designed for YOU: influencers of
change and innovation in your teams
- speakers and topics driving the evolution and innovation
- connecting and catalyzing the influencers and innovators
Highlights
- attended by more than 12,000 delegates since 2007
- held in 9 cities worldwide

4. Mapping Etsy’s Frontend
Tools and Techniques for
Managing Complexity at Scale
@danielespeset
Friday, March 7, 14

5. @danielespeset
Seth Walker
Allison McKnight Jonathan Klein
@sethwalker
@aemcknig @jonathanklein
Daniel Espeset
@danielespeset
Brought to you by
Friday, March 7, 14

6. @danielespeset
Etsy
Friday, March 7, 14

7. @danielespeset
A global marketplace for
buying and selling unique goods.
Etsy
Friday, March 7, 14

8. @danielespeset
Friday, March 7, 14

9. @danielespeset
Friday, March 7, 14

10. @danielespeset
roughly
180 engineers
Friday, March 7, 14

11. @danielespeset
50 deploys a day
Friday, March 7, 14

12. @danielespeset
Deployinator
IRC Push Queue
Friday, March 7, 14

13. @danielespeset
Anybody can touch anything.
Friday, March 7, 14

14. @danielespeset
Anybody can touch anything.
try not to break the site
Friday, March 7, 14

15. @danielespeset
Continuous
Experimentation
Friday, March 7, 14

16. @danielespeset
Engineers own their code
through the entire stack.
Friday, March 7, 14

17. @danielespeset
Bootcamps
Friday, March 7, 14

18. @danielespeset
Our frontend is made up of
300
different “pages”
Friday, March 7, 14

19. @danielespeset
At any given time we
may be running dozens
of experiments per page
Friday, March 7, 14

20. @danielespeset
2
12
4,096=
Friday, March 7, 14

21. @danielespeset
2
12
300 = 1,228,800x
Friday, March 7, 14

22. @danielespeset
OMG, how do actually
you do anything?
Friday, March 7, 14

23. @danielespeset
12 month growth of CSS code
last year
992 files
197,812 lines
last week
1,653 files (1.67x)
319,152 lines (1.61x)
Friday, March 7, 14

24. @danielespeset
12 month growth of JavaScript code
last year
1,451 files
425,915 lines
last week
2,394 files (1.65x)
639,484 lines (1.5x)
Friday, March 7, 14

25. @danielespeset
Friday, March 7, 14

26. @danielespeset
sup?
Friday, March 7, 14

27. @danielespeset
html * #buttons button.button {
modules/buttons.css
Friday, March 7, 14

28. @danielespeset
html * #buttons button.button {
wtf?
modules/buttons.css
Friday, March 7, 14

29. @danielespeset
$css/> grep “@import modules/buttons”
Friday, March 7, 14

30. @danielespeset
$css/> grep “@import modules/buttons”
a/browse.css
a/taxeditor.css
a/taxonomies.css
a/weddings.css
b/modules-responsive.css
b/modules.css
a/orders/refund.css
a/scout/scout.css
c/account/credit-card.css
c/bill/autobilling.css
c/bill/index.css
c/bill/payments.css
c/shop/registration/billing.css
Friday, March 7, 14

31. @danielespeset
$css/> for ... grep “@import $1” ...
a/browse.css
a/taxeditor.css
a/taxonomies.css
a/weddings.css
b/modules-responsive.css
b/modules.css
a/orders/refund.css
a/scout/scout.css
c/account/credit-card.css
c/bill/autobilling.css
c/bill/index.css
c/bill/payments.css
c/shop/registration/billing.css
base-responsive
base
base_after_inline
Friday, March 7, 14

32. @danielespeset
$css/> for ... grep “@import $1” ...
a/browse.css
a/taxeditor.css
a/taxonomies.css
a/weddings.css
b/modules-responsive.css
b/modules.css
a/orders/refund.css
a/scout/scout.css
c/account/credit-card.css
c/bill/autobilling.css
c/bill/index.css
c/bill/payments.css
c/shop/registration/billing.css
/base-responsive
/base
/base_after_inline
c/template_editor
c/mini
a/scout/dashboard
Friday, March 7, 14

33. @danielespeset
$templates/> for ... grep “{% css=$1” ...
a/browse.css
a/taxeditor.css
a/taxonomies.css
a/weddings.css
b/modules-responsive.css
b/modules.css
a/orders/refund.css
a/scout/scout.css
c/account/credit-card.css
c/bill/autobilling.css
c/bill/index.css
c/bill/payments.css
c/shop/registration/billing.css
/base-responsive
/base
/base_after_inline
c/template_editor
c/mini
a/scout/dashboard
wholesale/modules/vendor_avatar
wholesale/modules/address_from_array
wholesale/pages/purchase_order/modules/uneditable_fields/payment_method
wholesale/pages/purchase_order/modules/uneditable_fields/payment_terms
wholesale/pages/purchase_order/modules/uneditable_fields/ship_date
wholesale/pages/purchase_order/modules/uneditable_fields/billing_address
wholesale/pages/purchase_order/modules/uneditable_fields/shipping_address
wholesale/pages/purchase_order/modules/product_grid
modules/time_utils
base
Friday, March 7, 14

34. @danielespeset
$templates/> for ... grep “{% css=$1” ...
a/browse.css
a/taxeditor.css
a/taxonomies.css
a/weddings.css
b/modules-responsive.css
b/modules.css
a/orders/refund.css
a/scout/scout.css
c/account/credit-card.css
c/bill/autobilling.css
c/bill/index.css
c/bill/payments.css
c/shop/registration/billing.css
/base-responsive
/base
/base_after_inline
c/template_editor
c/mini
a/scout/dashboard
wholesale/modules/vendor_avatar
wholesale/modules/address_from_array
wholesale/pages/purchase_order/modules/uneditable_fields/payment_method
wholesale/pages/purchase_order/modules/uneditable_fields/payment_terms
wholesale/pages/purchase_order/modules/uneditable_fields/ship_date
wholesale/pages/purchase_order/modules/uneditable_fields/billing_address
wholesale/pages/purchase_order/modules/uneditable_fields/shipping_address
wholesale/pages/purchase_order/modules/product_grid
modules/time_utils
base
modules/deferred_clientlogger
modules/jquery
modules/async_analytics_init
notice/experiment_notice
modules/locale/dismissible_link
pages/recommendations/friends/v2/modules/gift-ideas-reminder
header
pages/translator/_modules/md5_overlay
modules/breadcrumbs
pages/browse/_modules/footer
footer
Friday, March 7, 14

35. @danielespeset
$templates/> for ... grep “{% css=$1” ...
a/browse.css
a/taxeditor.css
a/taxonomies.css
a/weddings.css
b/modules-responsive.css
b/modules.css
a/orders/refund.css
a/scout/scout.css
c/account/credit-card.css
c/bill/autobilling.css
c/bill/index.css
c/bill/payments.css
c/shop/registration/billing.css
/base-responsive
/base
/base_after_inline
c/template_editor
c/mini
a/scout/dashboard
wholesale/modules/vendor_avatar
wholesale/modules/address_from_array
wholesale/pages/purchase_order/modules/uneditable_fields/payment_method
wholesale/pages/purchase_order/modules/uneditable_fields/payment_terms
wholesale/pages/purchase_order/modules/uneditable_fields/ship_date
wholesale/pages/purchase_order/modules/uneditable_fields/billing_address
wholesale/pages/purchase_order/modules/uneditable_fields/shipping_address
wholesale/pages/purchase_order/modules/product_grid
modules/time_utils
base
modules/deferred_clientlogger
modules/jquery
modules/async_analytics_init
notice/experiment_notice
modules/locale/dismissible_link
pages/recommendations/friends/v2/modules/gift-ideas-reminder
header
pages/translator/_modules/md5_overlay
modules/breadcrumbs
pages/browse/_modules/footer
footer
Some/Controller/Class.php
Other/Controller/Class.php
Foo/Controller/Class.php
Bar/Controller/Class.php
Baz/Controller/Class.php
Batman/Controller/Class.php
...
Friday, March 7, 14

36. @danielespeset
$templates/> for ... grep “{% css=$1” ...
a/browse.css
a/taxeditor.css
a/taxonomies.css
a/weddings.css
b/modules-responsive.css
b/modules.css
a/orders/refund.css
a/scout/scout.css
c/account/credit-card.css
c/bill/autobilling.css
c/bill/index.css
c/bill/payments.css
c/shop/registration/billing.css
/base-responsive
/base
/base_after_inline
c/template_editor
c/mini
a/scout/dashboard
wholesale/modules/vendor_avatar
wholesale/modules/address_from_array
wholesale/pages/purchase_order/modules/uneditable_fields/payment_method
wholesale/pages/purchase_order/modules/uneditable_fields/payment_terms
wholesale/pages/purchase_order/modules/uneditable_fields/ship_date
wholesale/pages/purchase_order/modules/uneditable_fields/billing_address
wholesale/pages/purchase_order/modules/uneditable_fields/shipping_address
wholesale/pages/purchase_order/modules/product_grid
modules/time_utils
base
modules/deferred_clientlogger
modules/jquery
modules/async_analytics_init
notice/experiment_notice
modules/locale/dismissible_link
pages/recommendations/friends/v2/modules/gift-ideas-reminder
header
pages/translator/_modules/md5_overlay
modules/breadcrumbs
pages/browse/_modules/footer
footer
Some/Controller/Class.php
Other/Controller/Class.php
Foo/Controller/Class.php
Bar/Controller/Class.php
Baz/Controller/Class.php
Batman/Controller/Class.php
...
htdocs/my_endpoint.php
htdocs/foo.php
htdocs/other_endpoint.php
htdocs/baz/index.php
htdocs/index.php
Friday, March 7, 14

37. @danielespeset
$templates/> for ... grep “{% css=$1” ...
a/browse.css
a/taxeditor.css
a/taxonomies.css
a/weddings.css
b/modules-responsive.css
b/modules.css
a/orders/refund.css
a/scout/scout.css
c/account/credit-card.css
c/bill/autobilling.css
c/bill/index.css
c/bill/payments.css
c/shop/registration/billing.css
/base-responsive
/base
/base_after_inline
c/template_editor
c/mini
a/scout/dashboard
wholesale/modules/vendor_avatar
wholesale/modules/address_from_array
wholesale/pages/purchase_order/modules/uneditable_fields/payment_method
wholesale/pages/purchase_order/modules/uneditable_fields/payment_terms
wholesale/pages/purchase_order/modules/uneditable_fields/ship_date
wholesale/pages/purchase_order/modules/uneditable_fields/billing_address
wholesale/pages/purchase_order/modules/uneditable_fields/shipping_address
wholesale/pages/purchase_order/modules/product_grid
modules/time_utils
base
modules/deferred_clientlogger
modules/jquery
modules/async_analytics_init
notice/experiment_notice
modules/locale/dismissible_link
pages/recommendations/friends/v2/modules/gift-ideas-reminder
header
pages/translator/_modules/md5_overlay
modules/breadcrumbs
pages/browse/_modules/footer
footer
Some/Controller/Class.php
Other/Controller/Class.php
Foo/Controller/Class.php
Bar/Controller/Class.php
Baz/Controller/Class.php
Batman/Controller/Class.php
...
htdocs/my_endpoint.php
htdocs/foo.php
htdocs/other_endpoint.php
htdocs/baz/index.php
htdocs/index.php
RewriteRule ^(.*)/ve/(.*)$ $1/voluntary-employee/$2 [L,R=301]
RewriteRule ^(.*)/hsa/(.*)$ $1/health-saving-account/$2 [L,R=301]
RewriteCond %{REQUEST_FILENAME} !-f # Existing File
RewriteCond %{REQUEST_FILENAME} !-d # Existing Directory
RewriteRule . /index.php [L]
RewriteCond %{HTTP_REFERER} !^$
RewriteCond %{HTTP_REFERER} !^http://(subdomain.)?domain.tld/.*$ [NC]
RewriteRule ^.*.(bmp|tif|gif|jpg|jpeg|jpe|png)$ - [F]
Friday, March 7, 14

38. @danielespeset
CC Image courtesy of Mark Skipper (bitterjug) on Flickr
Halp!
Friday, March 7, 14

39. @danielespeset
Reasoning about our
frontend is hard.
Friday, March 7, 14

40. @danielespeset
Deleting code is scary.
Friday, March 7, 14

41. @danielespeset
Testing changes is difficult.
Friday, March 7, 14

42. @danielespeset
Mental overhead is a
performance problem.
Friday, March 7, 14

43. @danielespeset
What are our options?
• Static Analysis
• Synthetic Testing
• Access Logs
• Dynamic Analysis
Friday, March 7, 14

44. @danielespeset
What are our options?
• Static Analysis
• Synthetic Testing
• Access Logs
• Dynamic Analysis
Friday, March 7, 14

45. @danielespeset
What are our options?
• Static Analysis
• Synthetic Testing
• Access Logs
• Dynamic Analysis
Friday, March 7, 14

46. @danielespeset
What are our options?
• Static Analysis
• Synthetic Testing
• Access Logs
• Dynamic Analysis
Friday, March 7, 14

47. @danielespeset
What are our options?
• Static Analysis
• Synthetic Testing
• Access Logs
• Dynamic Analysis
Friday, March 7, 14

48. @danielespeset
What are our options?
• Static Analysis
• Synthetic Testing
• Access Logs
• Dynamic Analysis
Friday, March 7, 14

49. Builda
Friday, March 7, 14

50. @danielespeset
A brief history of JS loading at Etsy
• <script>
Friday, March 7, 14

51. @danielespeset
A brief history of JS loading at Etsy
• <script>
• Etsy.loader
Friday, March 7, 14

52. @danielespeset
A brief history of JS loading at Etsy
• <script>
• Etsy.loader
• //= require
Friday, March 7, 14

53. @danielespeset
Builda
Loading JS in development
assets/js
<script src="foo.js">
builda.php?name=foo.js
processing
Etsy.loader
// require
Friday, March 7, 14

54. @danielespeset
Building JS in deploys
bin/builda
Builda
Friday, March 7, 14

55. @danielespeset
Builda
Building JS in deploys
bin/builda
assets/js
dist/js
processing
Etsy.loader
// require
compiled output
Friday, March 7, 14

56. @danielespeset
Builda
dist/js
Building JS in deploys
bin/builda
assets/js
processing
Etsy.loader
// require
compiled output
Friday, March 7, 14

57. @danielespeset
Builda
dist/js
Building JS in deploys
bin/builda
assets/js
processing
Etsy.loader
// require
compiled output
Friday, March 7, 14

58. @danielespeset
Builda
Building JS in deploys
bin/builda
assets/js
dist/js
Minification
Friday, March 7, 14

59. @danielespeset
bin/builda
assets/jsBuilda
Building JS in deploys
dist/js
Minification
Friday, March 7, 14

60. @danielespeset
Building JS in deploys
bin/builda
assets/js
dist/js WWW
Builda
Friday, March 7, 14

61. @danielespeset
A brief history of JS loading at Etsy
• <script>
• Etsy.loader
• //= require
• AMD (requirejs)
Friday, March 7, 14

62. @danielespeset
Asyncronous
Module
Definition
Friday, March 7, 14

63. define(function() {
return function Lightsaber(color) {
this.color = color;
};
});
lightsaber.js
Friday, March 7, 14

64. define(function() {
return function Lightsaber(color) {
this.color = color;
};
});
lightsaber.js
define(['lightsaber'], function(Lightsaber) {
return function Luke() {
this.lightsaber = new Lightsaber('green');
};
});
luke.js
Friday, March 7, 14

65. define(function() {
return function Lightsaber(color) {
this.color = color;
};
});
lightsaber.js
define(['lightsaber'], function(Lightsaber) {
return function Luke() {
this.lightsaber = new Lightsaber('green');
};
});
luke.js
require(['plot','luke','vader'], function(plot, Luke,
Vader) {
var luke = new Luke(),
vader = new Vader();
plot.exploreFatherIssues(luke, vader);
});
starwars.js
Friday, March 7, 14

66. @danielespeset
define(function() {
return function Lightsaber(color) {
this.color = color;
};
});
lightsaber.js
define(['lightsaber'], function(Lightsaber) {
return function Luke() {
this.lightsaber = new Lightsaber('green');
};
});
luke.js
require(['plot','luke','vader'], function(plot, Luke, Vader) {
var luke = new Luke(),
vader = new Vader();
plot.exploreFatherIssues(luke, vader);
});
starwars.js
<script type="text/javascript" src="starwars.js"></script>
Builda
HTTP GET starwars.js
Async loading AMD in development
Friday, March 7, 14

67. @danielespeset
require(['plot','luke','vader'], function(plot, Luke, Vader) {
define(function() {
return function Lightsaber(color) {
this.color = color;
};
});
lightsaber.js
define(['lightsaber'], function(Lightsaber) {
return function Luke() {
this.lightsaber = new Lightsaber('green');
};
});
luke.js
Async loading AMD in development
Builda
uilda
HTTP GET luke.js
Friday, March 7, 14

68. @danielespeset
define(function() {
return function Lightsaber(color) {
this.color = color;
};
});
lightsaber.js
define(['lightsaber'], function(Lightsaber) {
Builda
HTTP GET lightsaber.js
Async loading AMD in development
Friday, March 7, 14

69. @danielespeset
Async loading AMD in development
Friday, March 7, 14

70. Incorporating AMD into
Builda, phase 1
Friday, March 7, 14

71. @danielespeset
Building JS with AMD in deploys
bin/builda
Builda assets/js
dist/js
r.jsr.jsr.jsr.js
Friday, March 7, 14

72. define(function() {
return function Lightsaber(color) {
this.color = color;
};
});
lightsaber.js
define(['lightsaber'], function(Lightsaber) {
return function Luke() {
this.lightsaber = new Lightsaber('green');
};
});
luke.js
require(['plot','luke','vader'], function(plot, Luke,
Vader) {
var luke = new Luke(),
vader = new Vader();
plot.exploreFatherIssues(luke, vader);
});
starwars.js
Friday, March 7, 14

73. // more modules...
starwars.12345678901234.js
define('lightsaber', function() {
return function Lightsaber(color) {
this.color = color;
};
});
define('luke', ['lightsaber'], function(Lightsaber) {
return function Luke() {
this.lightsaber = new Lightsaber('green');
};
});
require(['plot','luke','vader'], function(plot, Luke,
Vader) {
var luke = new Luke(),
vader = new Vader();
plot.exploreFatherIssues(luke, vader);
});
// more modules...
Friday, March 7, 14

74. This broke dev-prod
environment parity
Friday, March 7, 14

75. Also it was crazy slow
Friday, March 7, 14

76. Builda Revised
Friday, March 7, 14

77. @danielespeset
AKA taking the
A out of AMD
Friday, March 7, 14

78. @danielespeset
assets/js
dist/js
Building JS in development
Builda
Friday, March 7, 14

79. @danielespeset
assets/js
Save foo.js
Building JS in development
Buildadist/js
inotify
IN_CLOSE_WRITE
foo.js
"foo": {
"dependency_of": ["bar", "baz"]
}
Friday, March 7, 14

80. @danielespeset
Builda
assets/js
dist/js
AMD (r.js)
Etsy.loader
// require
foo.js
Building JS in development
Friday, March 7, 14

81. @danielespeset
Builda
assets/js
dist/js
AMD (r.js)
Etsy.loader
// require
bar.js
Building JS in development
Friday, March 7, 14

82. @danielespeset
Builda
assets/js
dist/js
AMD (r.js)
Etsy.loader
// require
baz.js
Building JS in development
Friday, March 7, 14

83. @danielespeset
assets/js
dist/js
Building JS in development
Builda
HTTP GET foo.js
Friday, March 7, 14

84. @danielespeset
assets/js
dist/js
Building JS in development
Builda
Friday, March 7, 14

85. Ranger
Friday, March 7, 14

86. @danielespeset
Jason Huff Aaron Moodie
@jsnhff @aaronmoodie
Friday, March 7, 14

87. @danielespeset
Friday, March 7, 14

88. @danielespeset
CSS
Friday, March 7, 14

89. @danielespeset
Friday, March 7, 14

90. @danielespeset
Friday, March 7, 14

91. @danielespeset
Friday, March 7, 14

92. @danielespeset
Pages
Friday, March 7, 14

93. @danielespeset
Friday, March 7, 14

94. @danielespeset
Friday, March 7, 14

95. @danielespeset
JavaScript
Friday, March 7, 14

96. @danielespeset
Friday, March 7, 14

97. @danielespeset
Friday, March 7, 14

98. @danielespeset
Ranger is an API
Lodge is a client
Friday, March 7, 14

99. @danielespeset
Ranger
Friday, March 7, 14

100. @danielespeset
Ranger
Friday, March 7, 14

101. @danielespeset
log host
Ranger
WWW
Friday, March 7, 14

102. @danielespeset
Ranger
log host
WWW
Friday, March 7, 14

103. @danielespeset
Ranger
rollup
linter
hinter
escomplex
scp stuff
log host
WWW
Friday, March 7, 14

104. @danielespeset
Ranger
rollup
linter
hinter
escomplex
scp stuff
log host
WWW
Friday, March 7, 14

105. @danielespeset
Now we know enough
to surface dead files
with confidence
Friday, March 7, 14

106. @danielespeset
Finding dead code paths
is still hard
Friday, March 7, 14

107. Shrinkray
Friday, March 7, 14

108. @danielespeset
Justin Donato Kelly Norton
@justindo @kellegous
Friday, March 7, 14

109. @danielespeset
Friday, March 7, 14

110. @danielespeset
What’s next?
• Adding Templates as a resource
Friday, March 7, 14

111. @danielespeset
What’s next?
• Adding Templates as a resource
• Shrinkray powered delete button
Friday, March 7, 14

112. @danielespeset
What’s next?
• Adding Templates as a resource
• Shrinkray powered delete button
• Detecting inefficiently grouped code
Friday, March 7, 14

113. @danielespeset
What’s next?
• Adding Templates as a resource
• Shrinkray powered delete button
• Detecting inefficiently grouped code
• Feature / User-state asset signatures
Friday, March 7, 14

114. @danielespeset
What’s next?
• Adding Templates as a resource
• Shrinkray powered delete button
• Detecting inefficiently grouped code
• Feature / User-state asset signatures
• Shrinkray for JavaScript
Friday, March 7, 14

115. @danielespeset
The frontend is getting more
complex everywhere.
Friday, March 7, 14

116. @danielespeset
We need new strategies to
manage and understand it.
Friday, March 7, 14

117. @danielespeset
The browser is part of your
distributed system, not
just a client you support.
Friday, March 7, 14

118. @danielespeset
The more extensible a tool
the more useful it will be.
Friday, March 7, 14

119. @danielespeset
Thanks!
codeascraft.com
@codeascraft
etsy.github.io
etsy.com/codeascraft/talks
etsy.com/careers
Friday, March 7, 14

120. Friday, March 7, 14

121. Watch the video with slide synchronization on
InfoQ.com!
http://www.infoq.com/presentations/javascript
-compiler-deploy