Your SlideShare is downloading. ×
Netflix JavaScript Talks - Scaling A/B Testing on Netflix.com with Node.js
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Netflix JavaScript Talks - Scaling A/B Testing on Netflix.com with Node.js

64,794
views

Published on

At Netflix we run hundreds of A/B tests every year. Maintaining multivariate experiences quickly adds strain to any UI engineering team. In this talk, Alex Liu and Micah Ransdell explore the patterns …

At Netflix we run hundreds of A/B tests every year. Maintaining multivariate experiences quickly adds strain to any UI engineering team. In this talk, Alex Liu and Micah Ransdell explore the patterns we’ve built in Node.js to tame this beast - ultimately enabling quick feature development and rapid test iteration on our service used by over 50 million people around the world.

Published in: Engineering

0 Comments
58 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
64,794
On Slideshare
0
From Embeds
0
Number of Embeds
68
Actions
Shares
0
Downloads
0
Comments
0
Likes
58
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Scaling A/B testing on Netflix.com with
  • 2. welcome Alex Liu @stinkydofu @mjr578 Micah Ransdell
  • 3. a/b testing data-driven product development
  • 4. Test 1 A B
  • 5. Test 1 A B C D E F G
  • 6. Test 1 A B C D E F G Test 2 A B C D E F G Test 3 A B C D E F G Test 4 A B C D E F G Test 5 A B C D E F G Test 6 A B C D E F G Test 7 A B C D E F G
  • 7. Test 1 A B C D E F G Test 2 A B C D E F G Test 3 A B C D E F G Test 4 A B C D E F G Test 5 A B C D E F G Test 6 A B C D E F G Test 7 A B C D E F G
  • 8. 2,097,152 unique experiences across seven tests
  • 9. hundreds of new A/B tests per year
  • 10. 984545041164535 824477853104281 030346308437926 36618855...
  • 11. 2,105 566 685 CSS JSTemplates
  • 12. 2.5 million unique css/js packages per push cycle
  • 13. <html/> <link/> <script/>
  • 14. problem: conditional dependencies
  • 15. ➔ Iterate Quickly ➔ Complex Concepts, Minimal Code ➔ Give the Power to the People Why Node?
  • 16. ➔Templating Packaging Bonus Round
  • 17. Templating
  • 18. Payment Test Testing the addition of Direct Debit
  • 19. Payment Test Testing the addition of Direct Debit
  • 20. Payment Test Testing the addition of Direct Debit
  • 21. Payment Test Testing the addition of Direct Debit
  • 22. Payment Test Testing the addition of Direct Debit
  • 23. payment template Control Cell 2 Cell 3 Cell 4 Cell 5 if if ifif if
  • 24. payment templatepayment template Control Cell 2 Cell 3 Cell 4 Cell 5 if if ifif if
  • 25. payment template Control Cell 2 Cell 3 Cell 4 Cell 5 if if ifif if payment_method_cc payment_method_dd
  • 26. payment templatepayment template Control Cell 2 Cell 3 Cell 4 Cell 5 if if ifif if payment_method_cc payment_method_dd
  • 27. payment template Control Cell 2 Cell 3 Cell 4 Cell 5 payment_method_cc payment_method_dd if if if if if
  • 28. payment template Control Cell 2 Cell 3 Cell 4 Cell 5 payment_method_cc payment_method_dd if if if if if
  • 29. payment template Control Cell 2 Cell 3 Cell 4 Cell 5 payment_method_cc payment_method_dd ?
  • 30. payment template Control Cell 2 Cell 3 Cell 4 Cell 5 payment_method_cc payment_method_dd payment_method.json
  • 31. payment template Control Cell 2 Cell 3 Cell 4 Cell 5 payment_method_cc payment_method_dd payment_method.json
  • 32. require(‘nf-template-resolver’)
  • 33. Resolver Rules Template Resolver Mappings
  • 34. require(‘nf-template-loader’)
  • 35. Resolver Template Inclusion Template Loader Template Cache
  • 36. ➔ Developers can combine rules ➔ Improves template legibility ➔ Increases template reuse Benefits
  • 37. Templating ➔Packaging Bonus Round
  • 38. Packaging
  • 39. everything is a module
  • 40. app.js oldSearch newSearch dep1 dep2 dep3 dep4 dep5 dep6 sub-dep sub-dep sub-dep sub-dep sub-dep sub-dep sub-depsub-dep
  • 41. app.js oldSearch newSearch dep1 dep2 dep3 dep4 dep5 dep6 sub-dep sub-dep sub-dep sub-dep sub-dep sub-dep sub-depsub-dep
  • 42. oldSearch newSearch dep1 dep2 dep3 dep4 dep5 dep6 sub-dep sub-dep sub-dep sub-dep sub-dep sub-dep sub-depsub-dep app.js
  • 43. app.js
  • 44. oldSearch newSearch dep1 dep2 dep3 dep4 dep5 dep6 sub-dep sub-dep sub-dep sub-dep sub-dep sub-dep sub-depsub-dep app.js
  • 45. oldSearch newSearch dep1 dep2 dep3 dep4 dep5 dep6 sub-dep sub-dep sub-dep sub-dep sub-dep sub-dep sub-depsub-dep app.js
  • 46. 685 files...?
  • 47. oldSearch newSearch dep1 dep2 dep3 dep4 dep5 dep6 sub-dep sub-dep sub-dep sub-dep sub-dep sub-dep sub-depsub-dep app.js
  • 48. oldSearch newSearch dep1 dep2 dep3 dep4 dep5 dep6 sub-dep sub-dep sub-dep sub-dep sub-dep sub-dep sub-depsub-dep app.js
  • 49. oldSearch newSearch dep1 dep2 dep3 dep4 dep5 dep6 sub-dep sub-dep sub-dep sub-dep sub-dep sub-dep sub-depsub-dep app.js
  • 50. oldSearch newSearch dep1 dep2 dep3 dep4 dep5 dep6 sub-dep sub-dep sub-dep sub-dep sub-dep sub-dep sub-depsub-dep app.js
  • 51. oldSearch newSearch dep1 dep2 dep3 dep4 dep5 dep6 sub-dep sub-dep sub-dep sub-dep sub-dep sub-dep sub-depsub-dep app.js
  • 52. problem: conditional dependencies
  • 53. RequestBuild
  • 54. require(‘nf-include-when’)
  • 55. oldSearch.js
  • 56. newSearch.js
  • 57. anatomy of a rule
  • 58. require(‘nf-asset-registry’)
  • 59. app.js
  • 60. app.js oldSearch.js newSearch.js jquery registry
  • 61. what’s in the registry?
  • 62. nf-include-when
  • 63. require(‘nf-packager’)
  • 64. Step 1: Get the full dependency tree for the requested package from the registry.
  • 65. Step 2: Run the rules.
  • 66. Step 3: Filter out all deps that resolved false.
  • 67. Step 3: Filter out all deps that resolved false.
  • 68. Step 3: Filter out all deps that resolved false. ✓
  • 69. ✓ Step 4: Filter out all extraneous sub-deps.
  • 70. Step 5: Concatenate the files.
  • 71. registry javascript / css Build
  • 72. Request registry rulespackage
  • 73. ➔ Leverage build time tools ➔ Leverage the server when you can ➔ Divide and conquer with modules Take aways
  • 74. Templating Packaging ➔Bonus Round
  • 75. Bonus Round
  • 76. be creative with the registry
  • 77. dependency counting dependency pruning file sizes
  • 78. CSS modules @import
  • 79. CSS analysis
  • 80. and now, the coolest part...
  • 81. templates mappings css javascript templates mappings css javascript
  • 82. templates mappings css javascript UI Bundle
  • 83. UI BundleUI Bundle UI Bundle UI Bundle UI BundleUI Bundle UI BundleUI Bundle UI Bundle
  • 84. anytime deploy UI bundles
  • 85. we never touch the file system
  • 86. < 5ms average response time
  • 87. ➔ Static analysis FTW ➔ Independent UI deployments ➔ Requests never touch the file system ➔ Fast package response times Wins
  • 88. final thoughts
  • 89. learn by doing
  • 90. move faster fail fast
  • 91. I have not failed. I’ve just found 10,000 ways that won’t work.” Thomas Edison “
  • 92. simplify
  • 93. thank you
  • 94. questions? Alex Liu @stinkydofu @mjr578 Micah Ransdell

×