More and more teams are looking for ways to iterate gradually and validate their ideas with data. Techniques like A/B testing, feature flagging, and gradual rollouts are quickly going from niche to mainstream. More experimentation means faster development and better products.
But like any trend, product experimentation is a good idea that can easily go wrong. For every game-changing A/B test, there’s a trail of testing mistakes that led well-meaning teams down the wrong path.
In this webinar, Jon Noronha, Director of Product Management at Optimizely, shares some common ways he’s seen A/B testing go wrong, along with tips for avoiding these pitfalls.
What you’ll learn:
- The most common mistakes product teams make when running experiments
- How to scale experimentation across multiple teams and squads
- How the world’s top technology companies are able to experiment on all product decisions
Product Experimentation Pitfalls & How to Avoid Them
1. J O N N O R O N H A
D I R E C T O R , P R O D U C T M A N A G E M E N T
Building a Culture of Experimentation
Avoiding the Pitfalls
T W I T T E R : @ T H A T S J O N S E N S E E M A I L : J O N @ O P T I M I Z E L Y . C O M
34. Tips for choosing the right metrics
Put yourself in the user’s shoes
Ask: what if this went up and nothing else?
Constantly re-evaluate
Trust your gut!
56. “Every day, we run a thousand concurrent experiments
to quickly validate new ideas. These experiments run
across all our products, from mobile apps and tools
used by hoteliers to customer service phone lines and
internal systems. Experimentation has become so
ingrained in Booking.com culture that every change,
from entire redesigns and infrastructure changes to
bug fixes, is wrapped in an experiment.”
57.
58. Experimental Treatment
Server-Side Experimentation
Optimizely Full Stack
Client-Side Experimentation
Optimizely Web
Client
Snippet Experiment
Dashboard
Default Treatment
Server
Experimental Treatment
Experimental Treatment
Client
Server SDK Experiment
Dashboard
User User
61. Remove all
contact
buttons for
EMEA
+1
Pop-Up
Promoting
the live
stream
Add NYC +
Atlanta
User Groups
to Log-in
10/17
Navigation
Experiment
(oct release)
AB test
EXO
handbook LP
Test new
hero
nav on
Blog
Scroll PopupBlog Posts
? Personalize
Promo cards
based on
behavior
August
Product
Release
Butterbar
DONE?
Test making
"try it free"
button green
Add PhoneTrackingfor
Leads
Test old
homepage vs.new homepage
62.
63. Atlanta
User Groups
to Log-in
10/17
gation
eriment
(oct release)
st
dbook LP
Test new
hero
nav on
Blog
Scroll PopupBlog Posts
? Personalize
Promo cards
based on
behavior
Test making
"try it free"
button green
Add PhoneTrackingfor
Leads
64. Experimentation Hero
10’s experiments / year
Experimentation Program
100’s experiments / year
Culture of Experimentation
1000’s experiments / year
Scaling the Experimentation Program
72. J O N N O R O N H A
D I R E C T O R , P R O D U C T M A N A G E M E N T
Thank you!
Questions?
T W I T T E R : @ T H A T S J O N S E N S E E M A I L : J O N @ O P T I M I Z E L Y . C O M