Netflix JavaScript Talks - Scaling A/B Testing on Netflix.com with Node.js

97,478 views
104,783 views

Published on

At Netflix we run hundreds of A/B tests every year. Maintaining multivariate experiences quickly adds strain to any UI engineering team. In this talk, Alex Liu and Micah Ransdell explore the patterns we’ve built in Node.js to tame this beast - ultimately enabling quick feature development and rapid test iteration on our service used by over 50 million people around the world.

Published in: Engineering

Netflix JavaScript Talks - Scaling A/B Testing on Netflix.com with Node.js

  1. 1. Scaling A/B testing on Netflix.com with
  2. 2. welcome Alex Liu @stinkydofu @mjr578 Micah Ransdell
  3. 3. a/b testing data-driven product development
  4. 4. Test 1 A B
  5. 5. Test 1 A B C D E F G
  6. 6. Test 1 A B C D E F G Test 2 A B C D E F G Test 3 A B C D E F G Test 4 A B C D E F G Test 5 A B C D E F G Test 6 A B C D E F G Test 7 A B C D E F G
  7. 7. Test 1 A B C D E F G Test 2 A B C D E F G Test 3 A B C D E F G Test 4 A B C D E F G Test 5 A B C D E F G Test 6 A B C D E F G Test 7 A B C D E F G
  8. 8. 2,097,152 unique experiences across seven tests
  9. 9. hundreds of new A/B tests per year
  10. 10. 984545041164535 824477853104281 030346308437926 36618855...
  11. 11. 2,105 566 685 CSS JSTemplates
  12. 12. 2.5 million unique css/js packages per push cycle
  13. 13. <html/> <link/> <script/>
  14. 14. problem: conditional dependencies
  15. 15. ➔ Iterate Quickly ➔ Complex Concepts, Minimal Code ➔ Give the Power to the People Why Node?
  16. 16. ➔Templating Packaging Bonus Round
  17. 17. Templating
  18. 18. Payment Test Testing the addition of Direct Debit
  19. 19. Payment Test Testing the addition of Direct Debit
  20. 20. Payment Test Testing the addition of Direct Debit
  21. 21. Payment Test Testing the addition of Direct Debit
  22. 22. Payment Test Testing the addition of Direct Debit
  23. 23. payment template Control Cell 2 Cell 3 Cell 4 Cell 5 if if ifif if
  24. 24. payment templatepayment template Control Cell 2 Cell 3 Cell 4 Cell 5 if if ifif if
  25. 25. payment template Control Cell 2 Cell 3 Cell 4 Cell 5 if if ifif if payment_method_cc payment_method_dd
  26. 26. payment templatepayment template Control Cell 2 Cell 3 Cell 4 Cell 5 if if ifif if payment_method_cc payment_method_dd
  27. 27. payment template Control Cell 2 Cell 3 Cell 4 Cell 5 payment_method_cc payment_method_dd if if if if if
  28. 28. payment template Control Cell 2 Cell 3 Cell 4 Cell 5 payment_method_cc payment_method_dd if if if if if
  29. 29. payment template Control Cell 2 Cell 3 Cell 4 Cell 5 payment_method_cc payment_method_dd ?
  30. 30. payment template Control Cell 2 Cell 3 Cell 4 Cell 5 payment_method_cc payment_method_dd payment_method.json
  31. 31. payment template Control Cell 2 Cell 3 Cell 4 Cell 5 payment_method_cc payment_method_dd payment_method.json
  32. 32. require(‘nf-template-resolver’)
  33. 33. Resolver Rules Template Resolver Mappings
  34. 34. require(‘nf-template-loader’)
  35. 35. Resolver Template Inclusion Template Loader Template Cache
  36. 36. ➔ Developers can combine rules ➔ Improves template legibility ➔ Increases template reuse Benefits
  37. 37. Templating ➔Packaging Bonus Round
  38. 38. Packaging
  39. 39. everything is a module
  40. 40. app.js oldSearch newSearch dep1 dep2 dep3 dep4 dep5 dep6 sub-dep sub-dep sub-dep sub-dep sub-dep sub-dep sub-depsub-dep
  41. 41. app.js oldSearch newSearch dep1 dep2 dep3 dep4 dep5 dep6 sub-dep sub-dep sub-dep sub-dep sub-dep sub-dep sub-depsub-dep
  42. 42. oldSearch newSearch dep1 dep2 dep3 dep4 dep5 dep6 sub-dep sub-dep sub-dep sub-dep sub-dep sub-dep sub-depsub-dep app.js
  43. 43. app.js
  44. 44. oldSearch newSearch dep1 dep2 dep3 dep4 dep5 dep6 sub-dep sub-dep sub-dep sub-dep sub-dep sub-dep sub-depsub-dep app.js
  45. 45. oldSearch newSearch dep1 dep2 dep3 dep4 dep5 dep6 sub-dep sub-dep sub-dep sub-dep sub-dep sub-dep sub-depsub-dep app.js
  46. 46. 685 files...?
  47. 47. oldSearch newSearch dep1 dep2 dep3 dep4 dep5 dep6 sub-dep sub-dep sub-dep sub-dep sub-dep sub-dep sub-depsub-dep app.js
  48. 48. oldSearch newSearch dep1 dep2 dep3 dep4 dep5 dep6 sub-dep sub-dep sub-dep sub-dep sub-dep sub-dep sub-depsub-dep app.js
  49. 49. oldSearch newSearch dep1 dep2 dep3 dep4 dep5 dep6 sub-dep sub-dep sub-dep sub-dep sub-dep sub-dep sub-depsub-dep app.js
  50. 50. oldSearch newSearch dep1 dep2 dep3 dep4 dep5 dep6 sub-dep sub-dep sub-dep sub-dep sub-dep sub-dep sub-depsub-dep app.js
  51. 51. oldSearch newSearch dep1 dep2 dep3 dep4 dep5 dep6 sub-dep sub-dep sub-dep sub-dep sub-dep sub-dep sub-depsub-dep app.js
  52. 52. problem: conditional dependencies
  53. 53. RequestBuild
  54. 54. require(‘nf-include-when’)
  55. 55. oldSearch.js
  56. 56. newSearch.js
  57. 57. anatomy of a rule
  58. 58. require(‘nf-asset-registry’)
  59. 59. app.js
  60. 60. app.js oldSearch.js newSearch.js jquery registry
  61. 61. what’s in the registry?
  62. 62. nf-include-when
  63. 63. require(‘nf-packager’)
  64. 64. Step 1: Get the full dependency tree for the requested package from the registry.
  65. 65. Step 2: Run the rules.
  66. 66. Step 3: Filter out all deps that resolved false.
  67. 67. Step 3: Filter out all deps that resolved false.
  68. 68. Step 3: Filter out all deps that resolved false. ✓
  69. 69. ✓ Step 4: Filter out all extraneous sub-deps.
  70. 70. Step 5: Concatenate the files.
  71. 71. registry javascript / css Build
  72. 72. Request registry rulespackage
  73. 73. ➔ Leverage build time tools ➔ Leverage the server when you can ➔ Divide and conquer with modules Take aways
  74. 74. Templating Packaging ➔Bonus Round
  75. 75. Bonus Round
  76. 76. be creative with the registry
  77. 77. dependency counting dependency pruning file sizes
  78. 78. CSS modules @import
  79. 79. CSS analysis
  80. 80. and now, the coolest part...
  81. 81. templates mappings css javascript templates mappings css javascript
  82. 82. templates mappings css javascript UI Bundle
  83. 83. UI BundleUI Bundle UI Bundle UI Bundle UI BundleUI Bundle UI BundleUI Bundle UI Bundle
  84. 84. anytime deploy UI bundles
  85. 85. we never touch the file system
  86. 86. < 5ms average response time
  87. 87. ➔ Static analysis FTW ➔ Independent UI deployments ➔ Requests never touch the file system ➔ Fast package response times Wins
  88. 88. final thoughts
  89. 89. learn by doing
  90. 90. move faster fail fast
  91. 91. I have not failed. I’ve just found 10,000 ways that won’t work.” Thomas Edison “
  92. 92. simplify
  93. 93. thank you
  94. 94. questions? Alex Liu @stinkydofu @mjr578 Micah Ransdell

×