Operationalizing experimentation at companies with established product practices can be painful. Bill Hinderman has built experimentation platforms and practices at five different companies. He will share how to develop a high-performing experimentation program that empowers your team to build more impactful product while managing the transition for product and engineering.

2. The Sky is Not Falling:
Scaling Experimentation for Product Development
Bill Hinderman
Web and Experimentation Engineering Lead, Vivid Seats

3. Hi.

4. Hi.

6. Direct
Organic
Search
Paid
Search
Ad
Placements
Email

7. Only I didn’t
say fudge.

8. falling
sky
The
is not
Opticon 8
@billhinderman
billhinderman.com
bill@billhinderman.com
18

9. falling
sky
The
is not

10. In brief.
What haven’t I done this?
How do I start?
How do I grow?

11. What size of team needs to scale experimentation?

13. Every
techincal change
is a
political change.
Every
technical change
is a political change.

14. How
do I
start?.
Pick your tools.
Build a roadmap based off of learning.

15. fig. 1: “Whoops!”

17. $

18. Build tools that fit
testing into your own
workflow.

19. Layer of abstraction allows us to skip call
Rapid QA cycle
Ability to share recipes with stakeholders
Easy debugging
Built into our own API
?experiment_key=recipe_key

20. private Variation setForcedVariationFromRequest(..., String uuid, String experimentKey) {
String parameter = OPTIMIZELY_FORCE_COOKIE_PREFIX + experimentKey;
String variationKey = request.getParameter(parameter);
if(variationKey == null) {
variationKey = cookieService.getCookieValue(request, parameter);
}
return new Variation(uuid, variationKey);
}
?experiment_key=recipe_key

21. private Variation setForcedVariationFromRequest(..., String uuid, String experimentKey) {
String parameter = OPTIMIZELY_FORCE_COOKIE_PREFIX + experimentKey;
String variationKey = request.getParameter(parameter);
if(variationKey == null) {
variationKey = cookieService.getCookieValue(request, parameter);
}
return new Variation(uuid, variationKey);
}
?experiment_key=recipe_key

22. private Variation setForcedVariationFromRequest(..., String uuid, String experimentKey) {
String parameter = OPTIMIZELY_FORCE_COOKIE_PREFIX + experimentKey;
String variationKey = request.getParameter(parameter);
if(variationKey == null) {
variationKey = cookieService.getCookieValue(request, parameter);
}
return new Variation(uuid, variationKey);
}
?experiment_key=recipe_key

23. Align experiment environments
with development environments.

26. When
am I
done
starting?.
Testing is part of product & engineering KPIs.
Team structure supports efficient experimentation.
Experimentation tools become their own product.

27. Decide upon a team
structure.

28. Experimentation Search App Checkout UI Components Supplier Integration
Experimentation Team
Experimentation
Feature / Platform

29. Experimentation Search App Checkout UI Components Supplier Integration
Decentralized experimentation
Experimentation
Feature / Platform

30. Experimentation Search App Checkout UI Components Supplier Integration
Experimentation
Feature / Platform

31. Experimentation Search App Checkout UI Components Supplier Integration
Experimentation Consultancy
Experimentation
Feature / Platform

32. Black box your testing
engine.

36. public Variation getVariation(HttpServletRequest request, HttpServletResponse response, String uuid, String experimentKey, Map<String, String> attributes) {
try {
if (requestUtils.isCustomer(request)) {
Cookie optimizelyCookie = cookieService.getCookie(request, OPTIMIZELY_COOKIE_PREFIX + experimentKey);
Variation variation;
if (attributes == null) {
attributes = new HashMap<>();
}
if (!attributes.containsKey("device_type")) {
attributes.putAll(getBaseAttributes(request));
}
String forceParameter = OPTIMIZELY_FORCE_COOKIE_PREFIX + experimentKey;
if (request.getParameter(forceParameter) != null || cookieService.doesCookieExist(request, forceParameter)) {
variation = setForcedVariationFromRequest(request, response, uuid, experimentKey);
} else if (optimizelyCookie != null) {
Optimizely optimizelyClient = getOptimizelyClient();
if (optimizelyClient == null) {
return null;
}
String optimizelyUUID = initOptimizelyUUID(request, response, uuid);
variation = optimizelyClient.getVariation(experimentKey, optimizelyUUID);
if (variation == null) {
cookieService.deleteCookie(response, optimizelyCookie.getName(), false);
variation = activateExperiment(request, response, uuid, experimentKey, attributes);
}

37. public Variation getVariation(HttpServletRequest request, HttpServletResponse response, String uuid, String experimentKey, Map<String, String> attributes) {
try {
if (requestUtils.isCustomer(request)) {
Cookie optimizelyCookie = cookieService.getCookie(request, OPTIMIZELY_COOKIE_PREFIX + experimentKey);
Variation variation;
if (attributes == null) {
attributes = new HashMap<>();
}
if (!attributes.containsKey("device_type")) {
attributes.putAll(getBaseAttributes(request));
}
String forceParameter = OPTIMIZELY_FORCE_COOKIE_PREFIX + experimentKey;
if (request.getParameter(forceParameter) != null || cookieService.doesCookieExist(request, forceParameter)) {
variation = setForcedVariationFromRequest(request, response, uuid, experimentKey);
} else if (optimizelyCookie != null) {
Optimizely optimizelyClient = getOptimizelyClient();
if (optimizelyClient == null) {
return null;
}
String optimizelyUUID = initOptimizelyUUID(request, response, uuid);
variation = optimizelyClient.getVariation(experimentKey, optimizelyUUID);
if (variation == null) {
cookieService.deleteCookie(response, optimizelyCookie.getName(), false);
variation = activateExperiment(request, response, uuid, experimentKey, attributes);
}

38. private Map<String, String> getBaseAttributes(HttpServletRequest request) {
Map<String, String> baseAttributes = new HashMap<>();
try {
String userAgentString = request.getHeader("User-Agent") == null ? "" : request.getHeader("User-Agent");
UserAgent userAgent = UserAgent.parseUserAgentString(userAgentString);
String browserType = userAgent.getBrowser().getName().toLowerCase();
if (browserType.contains("chrome") || browserType.contains("safari")) {
browserType = browserType.replaceAll("[^a-z]","");
}
String browserVersion = userAgent.getBrowserVersion().getMajorVersion();
String deviceType = userAgent.getOperatingSystem().getDeviceType() == eu.bitwalker.useragentutils.DeviceType.COMPUTER ? "desktop" :
userAgent.getOperatingSystem().getDeviceType().getName().toLowerCase();
String platform = userAgent.getOperatingSystem().getName().toLowerCase();
OrderChannel orderChannel = channelService.getActiveChannel(request);
String channel = "direct";
if (orderChannel != null && orderChannel.getH() != null) {
channel = orderChannel.getH().toLowerCase();
}
baseAttributes.put("browser_type", browserType);
baseAttributes.put("browser_version", browserVersion);
baseAttributes.put("channel", channel);
baseAttributes.put("device_type", deviceType);
baseAttributes.put("platform", platform);
} catch (Exception e) {
LOGGER.warn(e.getMessage(), e);
}
return baseAttributes;
}

39. public Variation getVariation(HttpServletRequest request, HttpServletResponse response, String uuid, String experimentKey, Map<String, String> attributes) {
try {
if (requestUtils.isCustomer(request)) {
Cookie optimizelyCookie = cookieService.getCookie(request, OPTIMIZELY_COOKIE_PREFIX + experimentKey);
Variation variation;
if (attributes == null) {
attributes = new HashMap<>();
}
if (!attributes.containsKey("device_type")) {
attributes.putAll(getBaseAttributes(request));
}
String forceParameter = OPTIMIZELY_FORCE_COOKIE_PREFIX + experimentKey;
if (request.getParameter(forceParameter) != null || cookieService.doesCookieExist(request, forceParameter)) {
variation = setForcedVariationFromRequest(request, response, uuid, experimentKey);
} else if (optimizelyCookie != null) {
Optimizely optimizelyClient = getOptimizelyClient();
if (optimizelyClient == null) {
return null;
}
String optimizelyUUID = initOptimizelyUUID(request, response, uuid);
variation = optimizelyClient.getVariation(experimentKey, optimizelyUUID);
if (variation == null) {
cookieService.deleteCookie(response, optimizelyCookie.getName(), false);
variation = activateExperiment(request, response, uuid, experimentKey, attributes);
}

40. public Variation getVariation(HttpServletRequest request, HttpServletResponse response, String uuid, String experimentKey, Map<String, String> attributes) {
try {
if (requestUtils.isCustomer(request)) {
Cookie optimizelyCookie = cookieService.getCookie(request, OPTIMIZELY_COOKIE_PREFIX + experimentKey);
Variation variation;
if (attributes == null) {
attributes = new HashMap<>();
}
if (!attributes.containsKey("device_type")) {
attributes.putAll(getBaseAttributes(request));
}
String forceParameter = OPTIMIZELY_FORCE_COOKIE_PREFIX + experimentKey;
if (request.getParameter(forceParameter) != null || cookieService.doesCookieExist(request, forceParameter)) {
variation = setForcedVariationFromRequest(request, response, uuid, experimentKey);
} else if (optimizelyCookie != null) {
Optimizely optimizelyClient = getOptimizelyClient();
if (optimizelyClient == null) {
return null;
}
String optimizelyUUID = initOptimizelyUUID(request, response, uuid);
variation = optimizelyClient.getVariation(experimentKey, optimizelyUUID);
if (variation == null) {
cookieService.deleteCookie(response, optimizelyCookie.getName(), false);
variation = activateExperiment(request, response, uuid, experimentKey, attributes);
}

41. String optimizelyUUID = initOptimizelyUUID(request, response, uuid);
private String initOptimizelyUUID(HttpServletRequest request, HttpServletResponse response, String uuid) {
String uuidCookie = cookieService.getCookieValue(request, OPTIMIZELY_COOKIE_PREFIX + "uuid");
if (StringUtils.isBlank(uuidCookie)) {
cookieService.setCookie(response, OPTIMIZELY_COOKIE_PREFIX + "uuid", uuid, SECONDS_5_YEARS);
return uuid;
}
return uuidCookie;
}

42. String forceParameter = OPTIMIZELY_FORCE_COOKIE_PREFIX + experimentKey;
if (request.getParameter(forceParameter) != null || cookieService.doesCookieExist(request, forceParameter)) {
variation = setForcedVariationFromRequest(request, response, uuid, experimentKey);
} else if (optimizelyCookie != null) {
Optimizely optimizelyClient = getOptimizelyClient();
if (optimizelyClient == null) {
return null;
}
String optimizelyUUID = initOptimizelyUUID(request, response, uuid);
variation = optimizelyClient.getVariation(experimentKey, optimizelyUUID);
if (variation == null) {
cookieService.deleteCookie(response, optimizelyCookie.getName(), false);
variation = activateExperiment(request, response, uuid, experimentKey, attributes);
}
} else {
variation = activateExperiment(request, response, uuid, experimentKey, attributes);
}
if (variation != null) {
setLogContextData(request);
return variation;
}
}
} catch (Exception e) {
LOGGER.warn("Unable get variation for experiment key: {}", experimentKey, e);
}
return null;
}

43. private Variation activateExperiment(HttpServletRequest request, HttpServletResponse response, String uuid, String experimentKey) {
return activateExperiment(request, response, uuid, experimentKey, new HashMap<String, String>());
}
private Variation activateExperiment(HttpServletRequest request, HttpServletResponse response, String uuid, String experimentKey, Map<String, String> attributes) {
Optimizely optimizelyClient = getOptimizelyClient();
if (optimizelyClient == null) {
return null;
}
String optimizelyUUID = initOptimizelyUUID(request, response, uuid);
Variation variation = optimizelyClient.activate(experimentKey, optimizelyUUID, attributes);
if (variation != null) {
LOGGER.info("Optimizely SDK: activateExperiment (Java)");
cookieService.setCookie(response, OPTIMIZELY_COOKIE_PREFIX + experimentKey, variation.getKey(), 2592000);
}
return variation;
}

44. private Variation activateExperiment(HttpServletRequest request, HttpServletResponse response, String uuid, String experimentKey) {
return activateExperiment(request, response, uuid, experimentKey, new HashMap<String, String>());
}
private Variation activateExperiment(HttpServletRequest request, HttpServletResponse response, String uuid, String experimentKey, Map<String, String> attributes) {
Optimizely optimizelyClient = getOptimizelyClient();
if (optimizelyClient == null) {
return null;
}
String optimizelyUUID = initOptimizelyUUID(request, response, uuid);
Variation variation = optimizelyClient.activate(experimentKey, optimizelyUUID, attributes);
if (variation != null) {
LOGGER.info("Optimizely SDK: activateExperiment (Java)");
cookieService.setCookie(response, OPTIMIZELY_COOKIE_PREFIX + experimentKey, variation.getKey(), 2592000);
}
return variation;
}

45. Web server-side
Web client-side
iOS client
Android client
Partner tools
Java full-stack
Java full-stack
iOS full-stack
Android full-stack
Partner tools full-stack

46. Web server-side
Web client-side
iOS client
Android client
Partner tools
The experiment stuff

47. How
do I
grow?.
Grow your appetite.
Improve your efficiency.

48. We don’t have the
appetite for more
experimentation.

49. Create a rubric for good
experiment hypothesis.

50. M
V
M
R
C
Adding a ticket details screen between
the tickets list and checkout screen will
aid users to find a more desirable ticket
which will ultimately improve conversion.

51. M
V
M
R
C

52. M
V
M
R
C

55. Measurable
Visible
Meaningful
Realistic
Cheap
Adding a ticket details screen between
the tickets list and checkout screen will
aid users to find a more desirable ticket
which will ultimately improve conversion.

56. Measurable
Visible
Meaningful
Realistic
Cheap
Adding a ticket details screen between
the tickets list and checkout screen will
aid users to find a more desirable ticket
which will ultimately enjoy their event more.

57. Measurable
Visible
Meaningful
Realistic
Cheap
Adding a ticket details screen between
the tickets list and checkout screen will
aid users to find a more desirable ticket
which will ultimately improve conversion.

58. Measurable
Visible
Meaningful
Realistic
Cheap
Adding a ticket details link in the
post-checkout receipt email will aid users
to find a more desirable ticket which will
ultimately improve conversion.

59. Measurable
Visible
Meaningful
Realistic
Cheap
Adding a ticket details screen between
the tickets list and checkout screen will
aid users to find a more desirable ticket
which will ultimately improve conversion.

60. Measurable
Visible
Meaningful
Realistic
Cheap
Adding a ticket details screen between
the tickets list and checkout screen will
aid users to view the venue’s code
of conduct which will ultimately
improve conversion.

61. Measurable
Visible
Meaningful
Realistic
Cheap
Adding a ticket details screen between
the tickets list and checkout screen will
aid users to find a more desirable ticket
which will ultimately improve conversion.

62. Measurable
Visible
Meaningful
Realistic
Cheap
Adding a ticket details screen that offers
free tickets will aid users to find a more
desirable ticket which will ultimately
improve conversion.

63. Measurable
Visible
Meaningful
Realistic
Cheap
Adding a ticket details screen between
the tickets list and checkout screen will
aid users to find a more desirable ticket
which will ultimately improve conversion.

64. Measurable
Visible
Meaningful
Realistic
Cheap
Building a ticket details app will aid users
to view find a more desirable ticket which will
ultimately improve conversion.

65. Measurable
Visible
Meaningful
Realistic
Cheap
Adding a ticket details screen between
the tickets list and checkout screen will
aid users to find a more desirable ticket
which will ultimately improve conversion.

66. We don’t have the
bandwidth for more
experimentation.

68. Build a hyperanalytical
platform.

75. Get QA involved.

76. Ideate
Build Disable
Analyze
Rollout
&
Regress
Run

77. Ideate
Build Disable
Analyze
Rollout
&
Regress
Run
Ideate
Build
Run

78. private Resolution getTicketPageView(UserAgent userAgent) {
try {
getContext().getResponse().addHeader("Vary", "User-Agent");
String web4801VariationKey = initVariation(WEB_4801_EXPERIMENT_KEY);
boolean isWEB4801Control = !"variant_1".equals(web4801VariationKey);
boolean showMobileView = userAgent.getDeviceType() == DeviceType.MOBILE || userAgent.getDeviceType() == DeviceType.TABLET;
if (!isMobileViewOverride() && showMobileView) {
if(!isWEB4801Control) {
initVariation(WEB_4304_EXPERIMENT_KEY);
return new ForwardResolution(JSP_PATH + RESPONSIVE_TICKET_PAGE_VIEW);
}
return new RedirectResolution(MobileProductionAction.class).addParameter(RequestAttributeConstants.PRODUCTION_PRODUCTION_ID,
production.getProductionId());
…
String web4456Variation = initVariation(WEB_4456_EXPERIMENT_KEY);
boolean isWEB4456Control = !"variant_1".equals(web4456Variation) && !"variant_2".equals(web4456Variation);
if (!isWEB4456Control) {
initVariation(WEB_4304_EXPERIMENT_KEY);
return new ForwardResolution(JSP_PATH + RESPONSIVE_TICKET_PAGE_VIEW);
}

79. Empower & trust your
center of excellence.

81. Every
techincal change
is a
political change.
Every
technical change
is a political change.

82. Let’s review.
Define your team’s optimization goal
Couple technology and process
Mandate measurable hypotheses
By hyperanalytical
Trust your center of excellence

83. You’re okay.

84. @billhinderman
billhinderman.com
bill@billhinderman.com

85. vividseats.com goshortwave.com podcastcampfire.com
mar-
tini
limi
-tedEst. 2018
martini.limited

86. vividseats.com goshortwave.com podcastcampfire.com
mar-
tini
limi
-tedEst. 2018
martini.limited

87. Questions?.
Opticon 8
@billhinderman
billhinderman.com
bill@billhinderman.com
18

88. Victor Forman
linkedin.com/in/victor-forman/
Andy Caples
linkedin.com/in/andy-caples-970b35/
John Banta
linkedin.com/in/bantaj/
Shanu Pant
linkedin.com/in/shanu-pant-093a807/
Smart people.

89. www.optimizely.com
www.shorpy.com
www.quora.com
www.caracaschronicles.com
www.tenor.com
Image and
story credit.

90. Thank you!
#opticon18