Advertisement

A/B Testing at Scale: Minimizing UI Complexity (SXSW 2015)

Director, UI Engineering at Netflix
Mar. 16, 2015
Advertisement

More Related Content

Advertisement
Advertisement

A/B Testing at Scale: Minimizing UI Complexity (SXSW 2015)

  1. #abtesting@csaintamant #abtesting@csaintamant
  2. #abtesting@csaintamant we can do better
  3. #abtesting@csaintamant brainstorm. test.brainstorm. test. measure.
  4. #abtesting@csaintamant data driven product development
  5. #abtesting@csaintamant continuous improvement
  6. vs ba
  7. a b
  8. A B C D E F G A B C D E F G A B C D E F G A B C D E F G A B C D E F G A B C D E F G A B C D E F G Test 1 Test 2 Test 3 Test 4 Test 5 Test 6 Test 7
  9. #abtesting@csaintamant it gets messy
  10. #abtesting@csaintamant ▶ design ▶ define ▶ deliver #abtesting@csaintamant
  11. #abtesting@csaintamant (testable) hypothesis
  12. #abtesting@csaintamant individual profiles in a multi- user household will create a more personalized experience
 and result in higher retention
  13. #abtesting@csaintamant test variables
  14. #abtesting@csaintamant test cells
  15. Variables Gate Gate w/ Remember Simple Setup Detailed Setup Cell 1 (control) Cell 2 ✓ Cell 3 ✓ ✓ Cell 4 ✓ ✓ Cell 5 ✓ ✓
  16. #abtesting@csaintamant be mindful
  17. #abtesting@csaintamant iterate later
  18. #abtesting@csaintamant ▶ design ▶ define ▶ deliver #abtesting@csaintamant
  19. #abtesting@csaintamant test identifiers
  20. #abtesting@csaintamant test # 1234 test cells 1 - 5
  21. var tests = user.getTests(); if (tests.in(1234, 2) || tests.in(1234, 3) || tests.in(1234, 4) || tests.in(1234, 5)) { // profiles enabled? if (tests.in(1234, 3) || tests.in(1234, 4) || tests.in(1234, 5)) { // gate? if (tests.in(1234, 4)) { // gate with remember? } } } else { // control behavior }
  22. #abtesting@csaintamant externalize config
  23. { meta: { id: 1234, name: 'profiles', }, cells: { '1': { control: true, profilesEnabled: false }, '2': { profilesEnabled: true, simpleSetup: true }, '3': { profilesEnabled: true, simpleSetup: true, gate: true }, '4': { profilesEnabled: true, simpleSetup: true, gate: true, gateRemember: true }, '5': { profilesEnabled: true, detailedSetup: true, gate: true } } }
  24. var profilesTest = user.getTests().get('profiles'); if (profilesTest.attr('profilesEnabled')) { // profiles enabled? if (profilesTest.attr('gate')) { // gate? if (profilesTest.attr('gateRemember')) { // gate with remember? } } } else { // control behavior }
  25. #abtesting@csaintamant feature flags? yep.
  26. profilesEnabled = new Rule('profilesEnabled', function(context, params) { // check test flag var profilesTest = context.getTests().get('profiles'); return profilesTest.attr('profilesEnabled'); }); rules
  27. profilesEnabled = new Rule('profilesEnabled', function(context, params) { // check app config return context.getAppConfig().profilesEnabled; }); rules
  28. var tests = user.getTests(); if (tests.in(1234, 3) || tests.in(1234, 4) || tests.in(1234, 5)) { // gate? } before after if (rules.profilesGate) { // gate! }
  29. #abtesting@csaintamant ▶ design ▶ define ▶ deliver #abtesting@csaintamant
  30. #abtesting@csaintamant UI assembly / delivery HTML + (CSS + JS)
  31. #abtesting@csaintamant templating
  32. profiles-setup.template <div id="profiles-form"> <h1>Who will be watching Netflix?</h1> <input id="profile1" /><label for-"profile1">That's you!</label> <input id="profile2" /> <input id="profile3" /> <input id="profile4" /> <input id="profile5" /> <button>Continue</button> </div> <div id="profiles-promo"> <ul> <li>Up to 5 people</li> <li>No extra fees</li> <li>Each person gets suggestions based on their own viewing and tastes</li> <li>Great for kids</li> </ul> </div>
  33. <div id="profiles-form"> <h1>Who will be watching Netflix?</h1> <input id="profile1" /><label for-"profile1">That's you!</label> <input id="profile2" /> <input id="profile3" /> <input id="profile4" /> <input id="profile5" /> <button>Continue</button> </div> <div id="profiles-promo"> {#control} <ul> <li>Up to 5 people</li> <li>No extra fees</li> <li>Each person gets suggestions based on their own viewing and tastes</li> <li>Great for kids</li> </ul> {/control} {#promo1} <ul> <li>Up to 5 people</li> <li>No extra fees</li> <li>Each person gets suggestions based on their own viewing and tastes</li> <li>Kids under 12 get a safe area with kid-friendly movies and TV shows</li> </ul> {/promo1} {#promo2} <ul> <li>Up to 5 people</li> <li>No extra fees</li> <li>Each person gets suggestions based on their own viewing and tastes</li> <li>Kids under 12 get a safe area with kid-friendly movies and TV shows</li> </ul> {/promo2} {#promo3} <img src="images/profiles-setup.png" /> <h2>Each person added will get suggestions based on what they like to watch</h2> <ul> <li>Up to 5 people</li> <li>No extra fees</li> </ul> {/promo3} {#promo4} <img src="images/profiles-setup.png" /> <h2>Have up to 5 profiles at no extra cost</h2> <ul> <li>Each person will get suggestions based on their own viewing and tastes</li> <li>Great for kids</li> </ul> {/promo4} </div>
  34. profiles-setup.tmpl if ifif if if Control Promo 1 Promo 2 Promo 3 Promo 4
  35. profiles-setup.tmpl promo.tmpl promo2.tmplpromo1.tmpl promo3.tmpl promo4.tmpl if ifif if if
  36. <div id="profiles-form"> <h1>Who will be watching Netflix?</h1> [...] </div> <div id="profiles-promo"> {> promo /} </div>
  37. promo.json profiles-setup.tmpl ? promo.tmpl promo2.tmplpromo1.tmpl promo3.tmpl promo4.tmpl
  38. promo.json { "rules": [], "templateName": "promo" }, { "rules": ["profilesPromo(1)"], "templateName": "promo1" }, { "rules": ["profilesPromo(2)"], "templateName": "promo2" }, { "rules": ["profilesPromo(3)"], "templateName": "promo3" }, { "rules": ["profilesPromo(4)"], "templateName": "promo4" }
  39. profilesPromo = new Rule('profilesPromo', function(context, params) { // check test membership var test = context.getTests().get('profilesSetup'); return test && test.cell(params.id-1); }); rules
  40. #abtesting@csaintamant how does it work?
  41. profiles-setup.tmpl template engine partial
  42. resolver promo.json (mappings) rules
  43. rules promo.tmpl promo1.tmpl promo2.tmpl
  44. profiles-setup.tmpl template engine resolver
  45. #abtesting@csaintamant template loading simplified
  46. #abtesting@csaintamant client-side packaging
  47. app.js import backbone from 'backbone'; import search from 'search'; import profiles from 'profiles'; export ...
  48. #abtesting@csaintamant search app.js profiles dep1 dep2 dep3 dep4 dep5 sub-dep sub-depsub-dep sub-dep sub-dep sub-dep
  49. #abtesting@csaintamant search app.js profiles dep1 dep2 dep3 dep4 dep5 sub-dep sub-depsub-dep sub-dep sub-dep sub-dep
  50. search app.js profiles dep1 dep2 dep3 dep4 dep5 sub-dep sub-depsub-dep sub-dep sub-dep sub-dep
  51. #abtesting@csaintamant 100s of files..? Millions of combinations..?
  52. #abtesting@csaintamant conditional dependencies
  53. search app.js profiles dep1 dep2 dep3 dep4 dep5 sub-dep sub-depsub-dep sub-dep sub-dep sub-dep
  54. rules again! profilesEnabled = new Rule('profilesEnabled', function(context, params) { // check test flag var profilesTest = context.getTests().get('profiles'); return profilesTest.attr('profilesEnabled'); });
  55. /* * @includewhen rule.profilesEnabled */ profiles.js
  56. /* * @includewhen rule.searchEnabled */ search.js
  57. #abtesting@csaintamant build time: static analysis
  58. #abtesting@csaintamant minify / compress build time:
  59. profiles.js backbone search.js app.js registry
  60. "app.js": { "deps": [ "backbone", "search.js", "profiles.js", ], "depsFull": [ "react", "searchDep2.js", "searchDep1.js", "search.js", "profilesDep2.js", "profilesDep1.js", "profiles.js" ] }
  61. "profiles.js": { "rule": "profilesEnabled", "deps": [ "backbone", "profilesDep2.js", "profilesDep1.js", ], "depsFull": [ "backbone", "profilesSubDep3.js", "profilesSubDep2.js" "profilesSubDep1.js" "profilesDep2.js", "profilesDep1.js" ] } rule!
  62. javascript registry build
  63. #abtesting@csaintamant request time: evaluate rules
  64. #abtesting@csaintamant assemble package request time:
  65. registry rulespackager request
  66. #abtesting@csaintamant packaging simplified
  67. #abtesting@csaintamant everything is a module
  68. #abtesting@csaintamant define dependencies
  69. #abtesting@csaintamant only load
 what you need …per user
  70. #abtesting@csaintamant isolate complexity
  71. #abtesting@csaintamant ▶ design ▶ define ▶ deliver #abtesting@csaintamant
  72. #abtesting@csaintamant multivariate testing is powerful…
  73. #abtesting@csaintamant …but complex
  74. #abtesting@csaintamant plan for complexity
  75. “Simplicity is the ultimate sophistication” - Leonardo da Vinci
  76. go experiment!
  77. questions? Chris Saint-Amant @csaintamant #abtesting
Advertisement