Optimizely partnered with Bain & Company to take a deep dive on the topic of testing. Here are Optimizely's slides from that presentation which shares best practices on what testing programs look like and how they can impact your business.

1. Optimization
Best Practices
Jessie Becker
CMO, Optimizely

2. What is Optimizely?
Powering the ideal experience for
every customer at each touchpoint

3. Powerfully Simple Technology
•Code-free WYSIWYG visual
editor makes it easy to create
and launch new experiments
•Get started in minutes with
easy implementation
•Iterate faster and bypass
bottlenecks, such as code
freezes, resource restraints
and AppStore approvals.
Optimize websites
or mobile apps

4. Real-Time, Actionable Insights
•Real-time analytics provide valuable
insights into performance so you can
take action
•Launch statistically significant winning
variations with one click
•Create and share reports for greater
visibility, highlighting key metrics and
insights for your business

5. Unlocking the Power of Optimization
“As Google does, most of
our work is proven or
disproven by A/B
testing.
Only that which leads to
an improved experience
gets rolled out to
everyone.”
- Netflix Investor Relations
“It’s really important to
get it out in the market
and test it with people to
see what they think.”
- Mike Schroepfer, CTO of
Facebook
“If you double the
number of experiments
you do per year,
you’re going to double
your inventiveness.”
- Jeff Bezos, CEO of Amazon

6. Unlocking the Power of Optimization

7. Experimentation at Netflix

8. Experimentation at Netflix

9. Best Practice #1:
Use optimization
to transform traditional
processes.

10. Testing & Optimization are Related
Continuous
Optimization
One-Off
Testing

11. Optimization Encourages Constant
Iteration and Discovery
• Data is actionable,
improvement measurable,
and risks calculated
• Iteration through testing is
the first step
• Dynamic experiments enable
new discoveries

12. Iron Mountain
Enterprise information management services
$7B market capitalization
Target High-Performing Experiences

13. Target High-Performing Experiences
+123%
Engagement
Generic Control Healthcare Audience Variation
Industry-specific information delivered based on IP address

14. Transform Marketing Campaigns
Code.org
Nonprofit backed by Bill Gates and Mark Zuckerberg
91M participants to date

15. Transform Marketing Campaigns
Simple copy experiments garnered 12 million new participants
+40%
Participants
Headline and CTA Control Winning Headline and CTA Variation

16. Transform Marketing Campaigns
• Campaigns are dynamic
• Tremendous performance upside
• Closing the loop from execution to
measurement prompts new action
• Improves the ROI of many digital
marketing channels
• Social/PR
• Paid Advertising
• Email Marketing

17. Best Practice #2:
Swing for the fences;
then scale to
what’s feasible.

18. Experimentation is More than Iteration
• Generate experiment data to qualify strategic choices
• Quantify the impact of investments and future improvements

19. Incentive Experiment: Netflix Referrals

20. Incentive Experiment: Netflix Referrals
Hypothesis: Incentives to invite friends will
generate cost-effective new users.
Variable: What value of offer will elicit a
response?
Lessons Learned:
• Test at the limits to determine effect
• Scale back to a reasonable level once
viable marketing channel has been
established

21. Organic Referral
Control
Invite a friend to Netflix!
Give One, Get One
Variation
Invite a friend for $100 credit!
$100 Credit
Variation
Incentive Experiment: Netflix Referrals

22. New Feature: Payment Options
Painted Door Experiment
Answer the question:
Is a new payment option attractive to customers?

23. New Feature: Payment Options
Hypothesis: Offering a new payment
method to users will increase signups.
Variable: Does the existence of this
feature matter to visitors?
Lessons Learned:
• Use a ‘painted door’ to determine if
development resources are needed to
build the option
• Quantify estimated value in advance

24. Best Practice #3:
Leverage the collective
creativity of your team.

25. Leadership Leadership
Individual Contributors Individual Contributors
Optimization Fosters Innovation

26. Leadership Leadership
Individual Contributors Individual Contributors
• Flow of ideas is multi-directional and
meritocratic
• Evaluate and execute on more ideas
• Uncover talent for both creativity and
data intelligence
• Diversity of skills and experiences is
a key strength of an optimization
program
Optimization Fosters Innovation

27. E-Commerce Signup Experiment
Trunk Club
Subscription clothing retailer
Recently acquired by Nordstrom for $350M

28. E-Commerce Signup Experiment
Counterintuitive concept immediately coded to production
+150%
New customer
acquisition
Onboarding Experience Variation

29. Best Practice #2:
Swing for the fences; then scale to
what’s feasible.
Best Practice #1:
Use optimization to transform
traditional processes.
Best Practice #3:
Leverage the collective creativity of
your team.

30. Questions?