The document discusses the importance of building a culture of product experimentation and highlights common pitfalls to avoid, such as optimizing the wrong metrics and thinking too small during A/B testing. It emphasizes the value of frequent experimentation, iterating based on data, and scaling experimentation programs within organizations. Key insights include the necessity to innovate bottom-up, embrace failure, and validate decisions through data-driven strategies.

1. Product Experimentation Pitfalls and
How to Avoid Them
JON NOROHNA
Director, Product Management
OPTIMIZELY

2. D A N S I R O K E R
C O - F O U N D E R & E X E C U T I V E C H A I R M A N
Building a Culture of
Product Experimentation
P E T E K O O M E N
C O - F O U N D E R & C H I E F T E C H N O L O G Y O F F I C E R

3. J O N N O R O N H A
D I R E C T O R , P R O D U C T M A N A G E M E N T
Building a Culture of Experimentation
Avoiding the Pitfalls
T W I T T E R : @ T H A T S J O N S E N S E E M A I L : J O N @ O P T I M I Z E L Y . C O M

4. LAUNCH
BUILD
DESIGN PRAY
EXPERIMENT
ITERATE

5. LAUNCH
BUILDDESIGN PRAY
EXPERIMENT
ITERATE

6. LAUNCHBUILDDESIGN
PRAY
EXPERIMENT
ITERATE

7. LAUNCHBUILDDESIGN PRAY
EXPERIMENT
ITERATE

9. LAUNCHBUILDDESIGN PRAY
EXPERIMENT
ITERATE

10. DESIGN BUILD EXPERIMENT ITERATE
LAUNCH
PRAY

11. Product Experimentation
Think hypothetically
• Painted doors
• Validating MVPs
• Qualitative research
• Exploratory A/B testing
Mitigate development risk
• Frequent, small releases
• Feature flagging
• Staged rollouts
• Safe rollbacks
Quantify product impact
• Feature validation
• Multivariate tests
• Iterative adjustments
• Measuring business impact
Experimentation

13. Old Reality Culture of Experimentation
Top-Down Innovation
Embrace Success
Make Decisions
Follow Orders
Bottom-Up Innovation
Embrace Failure
Validate Decisions
Follow Data

14. A/B
TESTING
CULTURE OF
EXPERIMENTATION

15. CULTURE OF
EXPERIMENTATION
CULTURE OF
EXPERIMENTATION

16. 0
10
100
1000
10000
VELOCITY
M AT U R I T Y
EXPERIMENTATION
HERO

17. 0
10
100
1000
10000
VELOCITY
M AT U R I T Y
EXPERIMENTATION
PROGRAM
EXPERIMENTATION
HERO

18. 0
10
100
1000
10000
VELOCITY
M AT U R I T Y
EXPERIMENTATION
PROGRAM
CULTURE OF
EXPERIMENTATION
EXPERIMENTATION
HERO

19. 0
10
100
1000
10000
VELOCITY
M AT U R I T Y
EXPERIMENTATION
HERO
EXPERIMENTATION
PROGRAM
CULTURE OF
EXPERIMENTATION

26. The World’s Largest Digital Laboratory
1,000,000+ Experiments and Counting

27. 0
10
100
1000
10000
VELOCITY
M AT U R I T Y
EXPERIMENTATION
HERO
EXPERIMENTATION
PROGRAM
CULTURE OF
EXPERIMENTATION

28. Experimentation Pitfalls
#1
Optimizing the wrong metrics

29. Bookings per Session Repeat Bookings per Visitor

30. Queries per Unique User

33. clicks that trigger a new query
clicks that don’t count as a new search

34. “We’re the only website on earth that
tries to get rid of our users as quickly as
possible”

35. Queries per Unique User
Queries per Session
Sessions per User

36. Tips for choosing the right metrics
Put yourself in the user’s shoes
Ask: what if this went up and nothing else?
Constantly re-evaluate
Trust your gut!

38. Experimentation Pitfalls
#2
Thinking too small

39. Theproblemwith“A/Btesting”

40. Theproblemwith“A/Btesting”

41. 2 Variations
A B
3 Variations
C
4 Variations
D
5 Variations
E
>6 Variations
FA B

42. Three-fourthsofallexperimentsonlyhave2variations
2 Variations
3 Variations
4 Variations
5 Variations
>6 Variations
77%
14%
5%
2%
1%

43. Testing5ormorevariationscanimproveyourwinrateby75%
+71%
+75%
+48%
+32%
25%
33%
37%
44%
43%
+75%
2 Variations
3 Variations
4 Variations
5 Variations
>6 Variations
Significant uplift
Significant reduction
Inconclusive

44. “Every day, we run a thousand concurrent experiments
to quickly validate new ideas. These experiments run
across all our products, from mobile apps and tools
used by hoteliers to customer service phone lines and
internal systems. Experimentation has become so
ingrained in Booking.com culture that every change,
from entire redesigns and infrastructure changes to
bug fixes, is wrapped in an experiment.”

46. Experimental Treatment
Server-Side Experimentation
Optimizely Full Stack
Client-Side Experimentation
Optimizely Web
Client
Snippet Experiment
Dashboard
Default Treatment
Server
Experimental Treatment
Experimental Treatment
Client
Server SDK Experiment
Dashboard
User User

47. Experimentation Pitfalls
#3
Hoarding insights

48. HYPOTHESIS
CREATIVE
DEVELOPMENT
SETUP & QA
TESTING
ANALYSIS
SHARE

49. Remove all
contact
buttons for
EMEA
+1
Pop-Up
Promoting
the live
stream
Add NYC +
Atlanta
User Groups
to Log-in
10/17
Navigation
Experiment
(oct release)
AB test
EXO
handbook LP
Test new
hero
nav on
Blog
Scroll PopupBlog Posts
? Personalize
Promo cards
based on
behavior
August
Product
Release
Butterbar
DONE?
Test making
"try it free"
button green
Add PhoneTrackingfor
Leads
Test old
homepage vs.new homepage

51. Atlanta
User Groups
to Log-in
10/17
gation
eriment
(oct release)
st
dbook LP
Test new
hero
nav on
Blog
Scroll PopupBlog Posts
? Personalize
Promo cards
based on
behavior
Test making
"try it free"
button green
Add PhoneTrackingfor
Leads

52. Experimentation Hero
10’s experiments / year
Experimentation Program
100’s experiments / year
Culture of Experimentation
1000’s experiments / year
Scaling the Experimentation Program

58. 0
10
100
1000
10000
VELOCITY
M AT U R I T Y
EXPERIMENTATION
HERO
EXPERIMENTATION
PROGRAM
CULTURE OF
EXPERIMENTATION

59. Product Experimentation
Think hypothetically
• Painted doors
• Validating MVPs
• Qualitative research
• Exploratory A/B testing
Mitigate development risk
• Frequent, small releases
• Feature flagging
• Staged rollouts
• Safe rollbacks
Quantify product impact
• Feature validation
• Multivariate tests
• Iterative adjustments
• Measuring business impact
Experimentation

60. J O N N O R O N H A
D I R E C T O R , P R O D U C T M A N A G E M E N T
Thank you!
Questions?
T W I T T E R : @ T H A T S J O N S E N S E E M A I L : J O N @ O P T I M I Z E L Y . C O M

61. THANK YOU
#Modernize2018

64. Algorithms
Optimize your search results,
recommendations, promotions, or any other
algorithms to deliver the most relevant content
to your customers
Redesigns
Test entirely new site or app designs before
you launch them, without any performance
impact to your customers
Feature Rollouts
Test features as you roll them out, to validate
your hypothesis and monitor your key business
metrics
Price Testing
Experiment with different prices, plans, or fees
personalized to your customer segments to
drive revenue
Backend Performance
Don’t just test the customer experience: test
APIs, microservices, clusters, and architecture
designs to improve performance & reliability
Omnichannel
Test experiences on mobile devices, over-the-
top apps, IoT, conversational interfaces, and
any other experience that runs in code, and
test across devices.
Server-side Experimentation Use Cases
A
B