Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

A/B Testing at Scale: Minimizing UI Complexity (SXSW 2015)

5,684 views

Published on

Slides from my talk at SXSW Interactive, March 16, 2015.
http://schedule.sxsw.com/2015/events/event_IAP40922

*Image credits*
Cables photo: https://www.flickr.com/photos/mikecogh/8678676687/in/photostream/
Blueprint photo: https://www.flickr.com/photos/wscullin/3770015203
Lego clones photo: https://www.flickr.com/photos/adactio/2856121959
Shipping containers photo: https://www.flickr.com/photos/dahlstroms/3144199355
Jet engine photo: https://www.flickr.com/photos/kingair42/3696473641
Cockpit photo: https://www.flickr.com/photos/jurvetson/6912974136
Whiteboard photo: https://www.flickr.com/photos/juhansonin/344714358
Raised hand photo: https://www.flickr.com/photos/simpleskye/9193218766
Worker icon designed by Bart Laugs from the http://thenounproject.com

Published in: Technology

A/B Testing at Scale: Minimizing UI Complexity (SXSW 2015)

  1. 1. #abtesting@csaintamant #abtesting@csaintamant
  2. 2. #abtesting@csaintamant we can do better
  3. 3. #abtesting@csaintamant brainstorm. test.brainstorm. test. measure.
  4. 4. #abtesting@csaintamant data driven product development
  5. 5. #abtesting@csaintamant continuous improvement
  6. 6. vs ba
  7. 7. a b
  8. 8. A B C D E F G A B C D E F G A B C D E F G A B C D E F G A B C D E F G A B C D E F G A B C D E F G Test 1 Test 2 Test 3 Test 4 Test 5 Test 6 Test 7
  9. 9. #abtesting@csaintamant it gets messy
  10. 10. #abtesting@csaintamant ▶ design ▶ define ▶ deliver #abtesting@csaintamant
  11. 11. #abtesting@csaintamant (testable) hypothesis
  12. 12. #abtesting@csaintamant individual profiles in a multi- user household will create a more personalized experience
 and result in higher retention
  13. 13. #abtesting@csaintamant test variables
  14. 14. #abtesting@csaintamant test cells
  15. 15. Variables Gate Gate w/ Remember Simple Setup Detailed Setup Cell 1 (control) Cell 2 ✓ Cell 3 ✓ ✓ Cell 4 ✓ ✓ Cell 5 ✓ ✓
  16. 16. #abtesting@csaintamant be mindful
  17. 17. #abtesting@csaintamant iterate later
  18. 18. #abtesting@csaintamant ▶ design ▶ define ▶ deliver #abtesting@csaintamant
  19. 19. #abtesting@csaintamant test identifiers
  20. 20. #abtesting@csaintamant test # 1234 test cells 1 - 5
  21. 21. var tests = user.getTests(); if (tests.in(1234, 2) || tests.in(1234, 3) || tests.in(1234, 4) || tests.in(1234, 5)) { // profiles enabled? if (tests.in(1234, 3) || tests.in(1234, 4) || tests.in(1234, 5)) { // gate? if (tests.in(1234, 4)) { // gate with remember? } } } else { // control behavior }
  22. 22. #abtesting@csaintamant externalize config
  23. 23. { meta: { id: 1234, name: 'profiles', }, cells: { '1': { control: true, profilesEnabled: false }, '2': { profilesEnabled: true, simpleSetup: true }, '3': { profilesEnabled: true, simpleSetup: true, gate: true }, '4': { profilesEnabled: true, simpleSetup: true, gate: true, gateRemember: true }, '5': { profilesEnabled: true, detailedSetup: true, gate: true } } }
  24. 24. var profilesTest = user.getTests().get('profiles'); if (profilesTest.attr('profilesEnabled')) { // profiles enabled? if (profilesTest.attr('gate')) { // gate? if (profilesTest.attr('gateRemember')) { // gate with remember? } } } else { // control behavior }
  25. 25. #abtesting@csaintamant feature flags? yep.
  26. 26. profilesEnabled = new Rule('profilesEnabled', function(context, params) { // check test flag var profilesTest = context.getTests().get('profiles'); return profilesTest.attr('profilesEnabled'); }); rules
  27. 27. profilesEnabled = new Rule('profilesEnabled', function(context, params) { // check app config return context.getAppConfig().profilesEnabled; }); rules
  28. 28. var tests = user.getTests(); if (tests.in(1234, 3) || tests.in(1234, 4) || tests.in(1234, 5)) { // gate? } before after if (rules.profilesGate) { // gate! }
  29. 29. #abtesting@csaintamant ▶ design ▶ define ▶ deliver #abtesting@csaintamant
  30. 30. #abtesting@csaintamant UI assembly / delivery HTML + (CSS + JS)
  31. 31. #abtesting@csaintamant templating
  32. 32. profiles-setup.template <div id="profiles-form"> <h1>Who will be watching Netflix?</h1> <input id="profile1" /><label for-"profile1">That's you!</label> <input id="profile2" /> <input id="profile3" /> <input id="profile4" /> <input id="profile5" /> <button>Continue</button> </div> <div id="profiles-promo"> <ul> <li>Up to 5 people</li> <li>No extra fees</li> <li>Each person gets suggestions based on their own viewing and tastes</li> <li>Great for kids</li> </ul> </div>
  33. 33. <div id="profiles-form"> <h1>Who will be watching Netflix?</h1> <input id="profile1" /><label for-"profile1">That's you!</label> <input id="profile2" /> <input id="profile3" /> <input id="profile4" /> <input id="profile5" /> <button>Continue</button> </div> <div id="profiles-promo"> {#control} <ul> <li>Up to 5 people</li> <li>No extra fees</li> <li>Each person gets suggestions based on their own viewing and tastes</li> <li>Great for kids</li> </ul> {/control} {#promo1} <ul> <li>Up to 5 people</li> <li>No extra fees</li> <li>Each person gets suggestions based on their own viewing and tastes</li> <li>Kids under 12 get a safe area with kid-friendly movies and TV shows</li> </ul> {/promo1} {#promo2} <ul> <li>Up to 5 people</li> <li>No extra fees</li> <li>Each person gets suggestions based on their own viewing and tastes</li> <li>Kids under 12 get a safe area with kid-friendly movies and TV shows</li> </ul> {/promo2} {#promo3} <img src="images/profiles-setup.png" /> <h2>Each person added will get suggestions based on what they like to watch</h2> <ul> <li>Up to 5 people</li> <li>No extra fees</li> </ul> {/promo3} {#promo4} <img src="images/profiles-setup.png" /> <h2>Have up to 5 profiles at no extra cost</h2> <ul> <li>Each person will get suggestions based on their own viewing and tastes</li> <li>Great for kids</li> </ul> {/promo4} </div>
  34. 34. profiles-setup.tmpl if ifif if if Control Promo 1 Promo 2 Promo 3 Promo 4
  35. 35. profiles-setup.tmpl promo.tmpl promo2.tmplpromo1.tmpl promo3.tmpl promo4.tmpl if ifif if if
  36. 36. <div id="profiles-form"> <h1>Who will be watching Netflix?</h1> [...] </div> <div id="profiles-promo"> {> promo /} </div>
  37. 37. promo.json profiles-setup.tmpl ? promo.tmpl promo2.tmplpromo1.tmpl promo3.tmpl promo4.tmpl
  38. 38. promo.json { "rules": [], "templateName": "promo" }, { "rules": ["profilesPromo(1)"], "templateName": "promo1" }, { "rules": ["profilesPromo(2)"], "templateName": "promo2" }, { "rules": ["profilesPromo(3)"], "templateName": "promo3" }, { "rules": ["profilesPromo(4)"], "templateName": "promo4" }
  39. 39. profilesPromo = new Rule('profilesPromo', function(context, params) { // check test membership var test = context.getTests().get('profilesSetup'); return test && test.cell(params.id-1); }); rules
  40. 40. #abtesting@csaintamant how does it work?
  41. 41. profiles-setup.tmpl template engine partial
  42. 42. resolver promo.json (mappings) rules
  43. 43. rules promo.tmpl promo1.tmpl promo2.tmpl
  44. 44. profiles-setup.tmpl template engine resolver
  45. 45. #abtesting@csaintamant template loading simplified
  46. 46. #abtesting@csaintamant client-side packaging
  47. 47. app.js import backbone from 'backbone'; import search from 'search'; import profiles from 'profiles'; export ...
  48. 48. #abtesting@csaintamant search app.js profiles dep1 dep2 dep3 dep4 dep5 sub-dep sub-depsub-dep sub-dep sub-dep sub-dep
  49. 49. #abtesting@csaintamant search app.js profiles dep1 dep2 dep3 dep4 dep5 sub-dep sub-depsub-dep sub-dep sub-dep sub-dep
  50. 50. search app.js profiles dep1 dep2 dep3 dep4 dep5 sub-dep sub-depsub-dep sub-dep sub-dep sub-dep
  51. 51. #abtesting@csaintamant 100s of files..? Millions of combinations..?
  52. 52. #abtesting@csaintamant conditional dependencies
  53. 53. search app.js profiles dep1 dep2 dep3 dep4 dep5 sub-dep sub-depsub-dep sub-dep sub-dep sub-dep
  54. 54. rules again! profilesEnabled = new Rule('profilesEnabled', function(context, params) { // check test flag var profilesTest = context.getTests().get('profiles'); return profilesTest.attr('profilesEnabled'); });
  55. 55. /* * @includewhen rule.profilesEnabled */ profiles.js
  56. 56. /* * @includewhen rule.searchEnabled */ search.js
  57. 57. #abtesting@csaintamant build time: static analysis
  58. 58. #abtesting@csaintamant minify / compress build time:
  59. 59. profiles.js backbone search.js app.js registry
  60. 60. "app.js": { "deps": [ "backbone", "search.js", "profiles.js", ], "depsFull": [ "react", "searchDep2.js", "searchDep1.js", "search.js", "profilesDep2.js", "profilesDep1.js", "profiles.js" ] }
  61. 61. "profiles.js": { "rule": "profilesEnabled", "deps": [ "backbone", "profilesDep2.js", "profilesDep1.js", ], "depsFull": [ "backbone", "profilesSubDep3.js", "profilesSubDep2.js" "profilesSubDep1.js" "profilesDep2.js", "profilesDep1.js" ] } rule!
  62. 62. javascript registry build
  63. 63. #abtesting@csaintamant request time: evaluate rules
  64. 64. #abtesting@csaintamant assemble package request time:
  65. 65. registry rulespackager request
  66. 66. #abtesting@csaintamant packaging simplified
  67. 67. #abtesting@csaintamant everything is a module
  68. 68. #abtesting@csaintamant define dependencies
  69. 69. #abtesting@csaintamant only load
 what you need …per user
  70. 70. #abtesting@csaintamant isolate complexity
  71. 71. #abtesting@csaintamant ▶ design ▶ define ▶ deliver #abtesting@csaintamant
  72. 72. #abtesting@csaintamant multivariate testing is powerful…
  73. 73. #abtesting@csaintamant …but complex
  74. 74. #abtesting@csaintamant plan for complexity
  75. 75. “Simplicity is the ultimate sophistication” - Leonardo da Vinci
  76. 76. go experiment!
  77. 77. questions? Chris Saint-Amant @csaintamant #abtesting

×