Netflix runs hundreds of multivariate AB tests a year, many of which help personalize the experience in the UI. This causes an exponential growth in the number of user experiences we serve to members, with each unique experience resulting in a unique JS/CSS bundle. Pre-publishing million of permutations to the CDN for each build of each UI simply does not work at Netflix scale. Instead, we've taken a novel approach by standing up a brand new Node.js service: Codex. Codex's sole responsibility is to build personalized JS/CSS bundles on the fly for our members as they move through the Netflix user experience. This frees up our UI teams to innovate rapidly on the UI itself, without having to worry about the costs of infrastructure and the complexity of pre-publishing to the CDN. As we stood up Codex, we learned a ton about building a horizontally scalable Node.js microservice. This talk is the story of how we built, designed, and scaled that service to meet the needs of our 80 million customers.

1. Alex Liu
A N E T F L I X E N G I N E E R I N G O R I G I N A L
@stinkydofualiu@netflix.com
codex
conditionaL
ModuleS
Strike bacK

4. #netflixeverywhere

8. contents
I. The Problem
II. Codex: Conditional Bundling
III. Scaling for Netflix
IV. Looking to the Future

9. i. the Problem

10. It's all about building
JS bundles.

11. // bundle.js
var a = require('a');
var b = require('b');
bundle.js

12. // bundle.js
var a = require('a');
var b = require('b');
bundle.js

13. var bundles = [
'bundle.js'
];
bundle.js
home.jsprofile.jsfeed.js album.js signup.js

14. home.js profile.js feed.js
signup.js account.js setting.js
album.js photo.js login.js
var bundles = [
'home.js',
'profile.js',
'feed.js',
'signup.js'
'account.js',
'settings.js',
'album.js',
'photo.js',
'login.js'
];

15. var bundles = [
'home.js',
'profile.js',
'feed.js',
'signup.js'
'account.js',
'settings.js',
'album.js',
'photo.js',
'login.js'
];
/home /profile /feed

16. var bundles = [
'home.js',
'profile.js',
'feed.js',
'signup.js'
'account.js',
'settings.js',
'album.js',
'photo.js',
'login.js'
];
/home /profile /feed
home.js profile.js feed.js

18. html5shiv
es5-shim

19. var bundles = [
'home.js',
'homeIE.js',
'profile',
'profileIE',
'feed.js',
'feedIE.js'
'signup.js',
'signupIE.js',
...
];

20. var bundles = [
'home.js',
'homeIE.js',
'profile',
'profileIE',
'feed.js',
'feedIE.js'
'signup.js',
'signupIE.js',
...
];

21. var bundles = [
'home.js',
'homeIE.js',
'profile',
'profileIE',
'feed.js',
'feedIE.js'
'signup.js',
'signupIE.js',
...
];
/home

22. var bundles = [
'home.js',
'homeIE.js',
'profile',
'profileIE',
'feed.js',
'feedIE.js'
'signup.js',
'signupIE.js',
...
];
/home

23. var bundles = [
'home.js',
'homeIE.js',
'profile',
'profileIE',
'feed.js',
'feedIE.js'
'signup.js',
'signupIE.js',
...
];
/home
home.js
NO

24. var bundles = [
'home.js',
'homeIE.js',
'profile',
'profileIE',
'feed.js',
'feedIE.js'
'signup.js',
'signupIE.js',
...
];
/home
home.js homeIE.js
YESNO

25. Netflix
AB testing!

26. AB Test w/ multiple cells
Cells Control (Cell 1) Cell 2 Cell 3
Movie
Cover
Art

27. AB Test w/ multiple cells
Cells Control (Cell 1) Cell 2 Cell 3
Movie
Cover
Art
14% 6%

28. Netflix
AB testing!

29. Netflix
AB testing!

30. home.js

31. home.js
newSearch.jsoldSearch.js

32. home.js
oldSearch.js

33. home.js
oldSearch.js

34. home.js
newSearch.js

35. home.js
newSearch.jsoldSearch.js

36. home.js
newSearch.jsoldSearch.js
jQuery React~80KB ~120KB

37. larger bundle size:
• file sizes
• time to download
• memory usage
• time to interactive (TTI)

38. old school Lego bricks
were generic

39. new Lego is about
specialization

40. hard to reuse
specialized bricks

41. home.js
newSearch.jsoldSearch.js

42. home.js
newSearch.jsoldSearch.js

43. home.js
newSearch.jsoldSearch.js

44. // starting to look like a
// lot of bundles...
var bundles = [
'homeNewSearch.js',
'homeNewSearchIE.js',
'homeOldSearch.js',
'homeOldSearchIE.js',
...
];

45. // starting to look like a
// lot of bundles...
var bundles = [
'homeNewSearch.js',
'homeNewSearchIE.js',
'homeOldSearch.js',
'homeOldSearchIE.js',
...
];
4x variations
already!

46. Netflix runs hundreds of
AB tests

47. Netflix runs hundreds of
AB tests
but we personalize on many
other dimensions too

50. |S1| ⋅ |S2| ⋅⋅⋅ |Sn| = |S1 × S2 × ⋅⋅⋅ × Sn|

51. |S1| ⋅ |S2| ⋅⋅⋅ |Sn| = |S1 × S2 × ⋅⋅⋅ × Sn|
3100
= 5.1537752e47

53. 40,000,000,000 bricks to reach the

54. 40,000,000,000 bricks to reach the
7,600,000,000,000,000 bricks to reach

55. 40,000,000,000 bricks to reach the
7,600,000,000,000,000 bricks to reach
Enough bricks to reach 6.7812832e32 times.

56. that’s a #$*%^ ton of
bundles!

57. https://xkcd.com/303/

58. Website's full
bundle is 10MB+

59. how do we deal with
conditional modules?

60. ii. codex conditional
Bundling
:

61. what if we generate
on-demand?

62. what if we generate
on-demand?
1. identify the UI variation
2. generate the bundle

63. how do we identify the
UI variation?

64. AB Tests

65. AB Tests

66. AB Tests

67. truthsnoun, plural [trooth z, trooths]
a bucket of boolean
flags used to build a
personalized Netflix
experience

68. {
"webfonts": false,
"instantSearch": true,
"socialFeatures": false,
"motionBanner": true,
"html5video": true,
"customScrollbar": true
}

69. {
"webfonts": false,
"instantSearch": true,
"socialFeatures": false,
"motionBanner": true,
"html5video": true,
"customScrollbar": true
}
inputs and outputs
are NOT 1:1

72. how do we generate
the bundle?

73. home.js
newSearch.jsoldSearch.js

74. // home.js
if (truths.isNewSearch === true) {
require('./newSearch');
} else {
require('./oldSearch');
}

75. home.js
newSearch.jsoldSearch.js

76. home.js
newSearch.jsoldSearch.js
conditioncondition

77. home.js
newSearch.jsoldSearch.js
conditioncondition

78. conditioncondition

79. isNewSearch!isNewSearch

80. // home.js
if (truths.isNewSearch === true) {
require('./newSearch');
} else {
require('./oldSearch');
}

81. home.js
newSearch.jsoldSearch.js
isNewSearch!isNewSearch

82. home.js
newSearch.js
newEntryPoint.js
oldSearch.js
!isNewSearch isNewSearch

83. git

84. git

85. git
codex(node.js module)

86. git
codex
artifact
(node.js module)

87. artifact

88. artifact

89. artifact

90. {
"home.js": {
"deps": [
"dep1.js",
"dep2.js",
"dep3.js",
],
"conditionalDeps": {
"newSearch.js": {
"name": "isNewSearch",
"value": true

91. ],
"conditionalDeps": {
"newSearch.js": {
"name": "isNewSearch",
"value": true
},
"oldSearch.js": {
"name": "isNewSearch",
"value": false
}
}
}

92. ],
"conditionalDeps": {
"newSearch.js": {
"name": "isNewSearch",
"value": true
},
"oldSearch.js": {
"name": "isNewSearch",
"value": false
}
}
}

93. it's a conditional map!

94. web/v1 web/v2 web/v3 web/v4

95. artifacttruths

98. <html/>
http://codex.nflxext.com/web/v1/83af

99. http://codex.nflxext.com/web/v1/83af

100. <script/>
http://codex.nflxext.com/web/v1/83af

101. <script/>
http://codex.nflxext.com/web/v1/83af

102. <script/> <script/>
codex
? i got this!
http://codex.nflxext.com/web/v1/83af

103. codex
http://codex.nflxext.com/web/v1/83af
http://codex.nflxext.com/{team}/{version}/{truths}

104. codex
web/v1
http://codex.nflxext.com/web/v1/83af
http://codex.nflxext.com/{team}/{version}/{truths}

105. codex {
83: newSearchTest,
af: isChrome
}
web/v1
http://codex.nflxext.com/web/v1/83af
http://codex.nflxext.com/{team}/{version}/{truths}

106. home.js

107. oldSearch.js
home.js

108. oldSearch.js
home.js

109. oldSearch.js
home.js

110. newSearch.jsoldSearch.js
home.js

111. home.js
newSearch.jsoldSearch.js
response times <= 80ms

112. codex
http://codex.nflxext.com/web/v1/83af

113. <script/>
codex
here
you go
http://codex.nflxext.com/web/v1/83af

114. <script/> <script/>
codex
cached!
here
you go
http://codex.nflxext.com/web/v1/83af

115. Recap
• Build Time: build conditional graph (artifact)
• Run Time: apply truths to artifact
• Conditional bundling is transparent, universal,
configuration free!

116. iii. Scaling For Netflix

117. web/v1 web/v2 tv/v5 tv/v7

118. Storage Metadata

119. Amazon S3
Amazon
DynamoDB
Storage Metadata

120. Build Time: Codex Artifact Management

121. web/v1
Build Time: Codex Artifact Management

122. web/v1
web/v1
Build Time: Codex Artifact Management
saved!

123. SAVED!
web/v1
web/v1
Build Time: Codex Artifact Management
saved!

124. Build Time: Codex Artifact Management

125. activate
web/v1 web/v1
activated!
Build Time: Codex Artifact Management

126. Run Time: Codex Bundler

127. web/v1
here are the
active build ids
Run Time: Codex Bundler

128. here are the
artifacts
web/v1
here are the
active build ids
Run Time: Codex Bundler

129. here are the
artifacts
web/v1
here are the
active build ids
Run Time: Codex Bundler

130. prod prod-new canary

131. 16GB ought to be
enough for us!__🙂

132. codex

133. codex

134. codex

135. 400+ artifacts!

136. 400+ artifacts!
…and we ran out of memory

137. 32GB ought to be
enough for us!__🤔

138. 800+ artifacts!

139. 800+ artifacts!
…and we ran out of memory

140. 64GB ought to be
enough for us…?__😭

141. 1600+ artifacts!

142. 1600+ artifacts!
…and we ran out of memory. again.

143. 1600+ artifacts!
Our teams will use as
much as we give them.
…and we ran out of memory. again.

145. Q: What's cheap,
plentiful, and fast
enough?

146. Q: What's cheap,
plentiful, and fast
enough?
A: Disk.

147. codex
LevelDB

148. codex
LevelDB

149. 100% CPU usage.

150. what's the problem?

151. ~68%

152. v8::internal::Runtime_ParseJson

153. codex
LevelDB

154. codex
LevelDB
JSON.parse

155. JSON.parse is slow.
And blocks the CPU!

156. codex
LevelDB
JSON.parse
LRU Cache: Saving the CPU

157. codex
LevelDB
JSON.parse
LRU Cache: Saving the CPU

158. codex
LevelDB
JSON.parse
LRU Cache: Saving the CPU

159. codex
LevelDB
JSON.parse
LRU Cache: Saving the CPU

160. codex
LevelDB
JSON.parse
LRU Cache: Saving the CPU

164. Breaking change to
conditional graph
traversal algorithm!

165. http://codex.nflxext.com/web/v1/83af

166. http://codex.nflxext.com/web/v1/83af
old algorithm?
new algorithm?

167. old algorithm?
new algorithm?
http://codex.nflxext.com/1.0.0/web/v1/83af
http://codex.nflxext.com/2.0.0/web/v1/83af

168. 1.0.0
2.0.0
zuul

169. zuul 1.0.0
2.0.0

170. zuul 1.0.0
2.0.0

172. Good for now.

173. Good for now.
Continue to look for
engineering wins.

174. What about
operational resiliency?

175. eu-west-1
us-west-2
us-east-1

177. eu-west-1
us-west-2
us-east-1

178. eu-west-1
us-west-2
???

179. eu-west-1
us-west-2
???

180. eu-west-1
us-west-2
???

181. ???
???
???

182. <script/> <script/>
codex

183. <script/>
codex
???

184. codex
<script/>
???

185. Recap
• Management plane necessary at scale
• Performance is critical (TTI)
• Redundancy across 3 AWS zones
• Resilient against CDN failure

186. iv. Looking To The Future

187. why not {bundler}?

188. how do we
support tree
shaking?

189. don't be afraid to
challenge common
convention.

190. don't make
assumptions about
the upper limits.

191. don't optimize before you
understand the system.

192. use the scientific method:
1. gather data
2. formulate hypothesis
3. test hypothesis
4. repeat

193. engineer for fault tolerance

194. Netflix scale is
challenging.

195. https://www.flickr.com/clement127/
https://www.flickr.com/jose_antonio_hidalgo_jimenez/
https://www.flickr.com/reiterlied/
Lego Photo Credits

196. Image Credits

197. Image Credits

198. Image Credits
Artist: alecive (Alessandro Roncone)
Iconset Homepage: https://github.com/alecive/FlatWoken

199. Alex Liu aliu@netflix.com @stinkydofu
fin