Slides from my talk at SXSW Interactive, March 16, 2015. http://schedule.sxsw.com/2015/events/event_IAP40922 *Image credits* Cables photo: https://www.flickr.com/photos/mikecogh/8678676687/in/photostream/ Blueprint photo: https://www.flickr.com/photos/wscullin/3770015203 Lego clones photo: https://www.flickr.com/photos/adactio/2856121959 Shipping containers photo: https://www.flickr.com/photos/dahlstroms/3144199355 Jet engine photo: https://www.flickr.com/photos/kingair42/3696473641 Cockpit photo: https://www.flickr.com/photos/jurvetson/6912974136 Whiteboard photo: https://www.flickr.com/photos/juhansonin/344714358 Raised hand photo: https://www.flickr.com/photos/simpleskye/9193218766 Worker icon designed by Bart Laugs from the http://thenounproject.com

1. #abtesting@csaintamant #abtesting@csaintamant

7. #abtesting@csaintamant
we can do better

9. #abtesting@csaintamant
brainstorm. test.brainstorm. test. measure.

10. #abtesting@csaintamant
data driven
product development

11. #abtesting@csaintamant
continuous
improvement

12. vs ba

13. a b

14. A
B
C
D
E
F
G
A
B
C
D
E
F
G
A
B
C
D
E
F
G
A
B
C
D
E
F
G
A
B
C
D
E
F
G
A
B
C
D
E
F
G
A
B
C
D
E
F
G
Test 1 Test 2 Test 3 Test 4 Test 5 Test 6 Test 7

15. #abtesting@csaintamant
it gets
messy

16. #abtesting@csaintamant
▶ design
▶ define
▶ deliver
#abtesting@csaintamant

17. #abtesting@csaintamant
(testable)
hypothesis

18. #abtesting@csaintamant
individual profiles in a multi-
user household will create a
more personalized experience
and result in higher retention

22. #abtesting@csaintamant
test
variables

25. #abtesting@csaintamant
test cells

26. Variables Gate
Gate w/
Remember
Simple Setup Detailed Setup
Cell 1 (control)
Cell 2 ✓
Cell 3 ✓ ✓
Cell 4 ✓ ✓
Cell 5 ✓ ✓

27. #abtesting@csaintamant
be mindful

28. #abtesting@csaintamant
iterate later

29. #abtesting@csaintamant
▶ design
▶ define
▶ deliver
#abtesting@csaintamant

30. #abtesting@csaintamant
test identifiers

31. #abtesting@csaintamant
test # 1234
test cells 1 - 5

32. var tests = user.getTests();
if (tests.in(1234, 2) || tests.in(1234, 3) ||
tests.in(1234, 4) || tests.in(1234, 5)) {
// profiles enabled?
if (tests.in(1234, 3) || tests.in(1234, 4) ||
tests.in(1234, 5)) {
// gate?
if (tests.in(1234, 4)) {
// gate with remember?
}
}
}
else {
// control behavior
}

33. #abtesting@csaintamant
externalize config

34. {
meta: {
id: 1234,
name: 'profiles',
},
cells: {
'1': { control: true, profilesEnabled: false },
'2': { profilesEnabled: true, simpleSetup: true },
'3': { profilesEnabled: true, simpleSetup: true, gate: true },
'4': { profilesEnabled: true, simpleSetup: true, gate: true,
gateRemember: true },
'5': { profilesEnabled: true, detailedSetup: true, gate: true }
}
}

35. var profilesTest = user.getTests().get('profiles');
if (profilesTest.attr('profilesEnabled')) {
// profiles enabled?
if (profilesTest.attr('gate')) {
// gate?
if (profilesTest.attr('gateRemember')) {
// gate with remember?
}
}
}
else {
// control behavior
}

36. #abtesting@csaintamant
feature flags?
yep.

37. profilesEnabled =
new Rule('profilesEnabled', function(context, params) {
// check test flag
var profilesTest = context.getTests().get('profiles');
return profilesTest.attr('profilesEnabled');
});
rules

38. profilesEnabled =
new Rule('profilesEnabled', function(context, params) {
// check app config
return context.getAppConfig().profilesEnabled;
});
rules

39. var tests = user.getTests();
if (tests.in(1234, 3) ||
tests.in(1234, 4) ||
tests.in(1234, 5)) {
// gate?
}
before after
if (rules.profilesGate) {
// gate!
}

40. #abtesting@csaintamant
▶ design
▶ define
▶ deliver
#abtesting@csaintamant

41. #abtesting@csaintamant
UI assembly / delivery
HTML + (CSS + JS)

42. #abtesting@csaintamant
templating

48. profiles-setup.template
<div id="profiles-form">
<h1>Who will be watching Netflix?</h1>
<input id="profile1" /><label for-"profile1">That's you!</label>
<input id="profile2" />
<input id="profile3" />
<input id="profile4" />
<input id="profile5" />
<button>Continue</button>
</div>
<div id="profiles-promo">
<ul>
<li>Up to 5 people</li>
<li>No extra fees</li>
<li>Each person gets suggestions based on their own viewing and tastes</li>
<li>Great for kids</li>
</ul>
</div>

49. <div id="profiles-form">
<h1>Who will be watching Netflix?</h1>
<input id="profile1" /><label for-"profile1">That's you!</label>
<input id="profile2" />
<input id="profile3" />
<input id="profile4" />
<input id="profile5" />
<button>Continue</button>
</div>
<div id="profiles-promo">
{#control}
<ul>
<li>Up to 5 people</li>
<li>No extra fees</li>
<li>Each person gets suggestions based on their own viewing and tastes</li>
<li>Great for kids</li>
</ul>
{/control}
{#promo1}
<ul>
<li>Up to 5 people</li>
<li>No extra fees</li>
<li>Each person gets suggestions based on their own viewing and tastes</li>
<li>Kids under 12 get a safe area with kid-friendly movies and TV shows</li>
</ul>
{/promo1}
{#promo2}
<ul>
<li>Up to 5 people</li>
<li>No extra fees</li>
<li>Each person gets suggestions based on their own viewing and tastes</li>
<li>Kids under 12 get a safe area with kid-friendly movies and TV shows</li>
</ul>
{/promo2}
{#promo3}
<img src="images/profiles-setup.png" />
<h2>Each person added will get suggestions based on what they like to watch</h2>
<ul>
<li>Up to 5 people</li>
<li>No extra fees</li>
</ul>
{/promo3}
{#promo4}
<img src="images/profiles-setup.png" />
<h2>Have up to 5 profiles at no extra cost</h2>
<ul>
<li>Each person will get suggestions based on their own viewing and tastes</li>
<li>Great for kids</li>
</ul>
{/promo4}
</div>

50. profiles-setup.tmpl
if ifif if if
Control Promo 1 Promo 2 Promo 3 Promo 4

51. profiles-setup.tmpl
promo.tmpl promo2.tmplpromo1.tmpl promo3.tmpl promo4.tmpl
if ifif if if

52. <div id="profiles-form">
<h1>Who will be watching Netflix?</h1>
[...]
</div>
<div id="profiles-promo">
{> promo /}
</div>

53. promo.json
profiles-setup.tmpl
?
promo.tmpl promo2.tmplpromo1.tmpl promo3.tmpl promo4.tmpl

54. promo.json
{
"rules": [],
"templateName": "promo"
},
{
"rules": ["profilesPromo(1)"],
"templateName": "promo1"
},
{
"rules": ["profilesPromo(2)"],
"templateName": "promo2"
},
{
"rules": ["profilesPromo(3)"],
"templateName": "promo3"
},
{
"rules": ["profilesPromo(4)"],
"templateName": "promo4"
}

55. profilesPromo =
new Rule('profilesPromo', function(context, params) {
// check test membership
var test = context.getTests().get('profilesSetup');
return test && test.cell(params.id-1);
});
rules

56. #abtesting@csaintamant
how does it work?

57. profiles-setup.tmpl template engine partial

58. resolver promo.json
(mappings)
rules

59. rules
promo.tmpl
promo1.tmpl
promo2.tmpl

60. profiles-setup.tmpl template engine resolver

61. #abtesting@csaintamant
template loading
simplified

62. #abtesting@csaintamant
client-side
packaging

64. app.js
import backbone from 'backbone';
import search from 'search';
import profiles from 'profiles';
export ...

65. #abtesting@csaintamant
search
app.js
profiles
dep1 dep2 dep3 dep4 dep5
sub-dep sub-depsub-dep sub-dep sub-dep sub-dep

66. #abtesting@csaintamant
search
app.js
profiles
dep1 dep2 dep3 dep4 dep5
sub-dep sub-depsub-dep sub-dep sub-dep sub-dep

67. search
app.js
profiles
dep1 dep2 dep3 dep4 dep5
sub-dep sub-depsub-dep sub-dep sub-dep sub-dep

69. #abtesting@csaintamant
100s of files..?
Millions of combinations..?

70. #abtesting@csaintamant
conditional
dependencies

71. search
app.js
profiles
dep1 dep2 dep3 dep4 dep5
sub-dep sub-depsub-dep sub-dep sub-dep sub-dep

72. rules again!
profilesEnabled =
new Rule('profilesEnabled', function(context, params) {
// check test flag
var profilesTest = context.getTests().get('profiles');
return profilesTest.attr('profilesEnabled');
});

73. /*
* @includewhen rule.profilesEnabled
*/
profiles.js

74. /*
* @includewhen rule.searchEnabled
*/
search.js

75. #abtesting@csaintamant
build time:
static analysis

76. #abtesting@csaintamant
minify / compress
build time:

77. profiles.js
backbone
search.js
app.js
registry

78. "app.js": {
"deps": [
"backbone",
"search.js",
"profiles.js",
],
"depsFull": [
"react",
"searchDep2.js",
"searchDep1.js",
"search.js",
"profilesDep2.js",
"profilesDep1.js",
"profiles.js"
]
}

79. "profiles.js": {
"rule": "profilesEnabled",
"deps": [
"backbone",
"profilesDep2.js",
"profilesDep1.js",
],
"depsFull": [
"backbone",
"profilesSubDep3.js",
"profilesSubDep2.js"
"profilesSubDep1.js"
"profilesDep2.js",
"profilesDep1.js"
]
}
rule!

80. javascript
registry
build

81. #abtesting@csaintamant
request time:
evaluate rules

82. #abtesting@csaintamant
assemble package
request time:

83. registry
rulespackager
request

84. #abtesting@csaintamant
packaging
simplified

85. #abtesting@csaintamant
everything is
a module

86. #abtesting@csaintamant
define dependencies

87. #abtesting@csaintamant
only load
what you need
…per user

88. #abtesting@csaintamant
isolate complexity

89. #abtesting@csaintamant
▶ design
▶ define
▶ deliver
#abtesting@csaintamant

92. #abtesting@csaintamant
multivariate testing
is powerful…

93. #abtesting@csaintamant
…but complex

94. #abtesting@csaintamant
plan for complexity

95. “Simplicity is the
ultimate sophistication”
- Leonardo da Vinci

96. go experiment!

97. questions?
Chris Saint-Amant
@csaintamant
#abtesting