SlideShare a Scribd company logo

Optimizely Workshop: Take Action on Results with Statistics

•
•
Optimizely
Optimizely

Optimizely recently released the stats engine, which moves away from the traditional statistics model and into a new framework that is more aligned with modern business operations. In this workshop, we’ll walk you through the core trade-offs in A/B Testing, and how you can use them to decide when to stop running your test.

Data & AnalyticsTechnology
1 of 67
Download to read offline
Take Action on Results With Statisitcs An Optimizely Online Workshop Statistician: Leonid Pekelis
Optimizely’s Stats Engine is designed to work with you, not against you, to provide results which are reliable and accurate, without requiring statistical training. At the same time, by knowing some statistics of your own, you can tune Stats Engine to get the most performance for your unique needs.
1. Which two A/B Testing pitfalls inflate error rates when using classical statistics, and are avoided with Stats Engine? 2. What are the three tradeoffs in an A/B Test? And how are they related? 3. How can you use Optimizely’s results page to best tune the tradeoffs to achieve your experimentation goals? After this workshop, you should be able to answer…
How to choose the number of goals and variations for your experiment. We will also preview
First, some vocabulary (yay!)
• A) The original, or baseline version of content that you are testing through a variation. • B) Metric used to measure impact of control and variation • C) The control group’s expected conversion rate. • D) The relative percentage difference of your variation from baseline. • E) The number of visitors in your test. Which is the Improvement?
• A) Control and Variation The original, or baseline version of content that you are testing through a variation. • B) Goal Metric used to measure impact of control and variation • C) Baseline conversion rate The control group’s expected conversion rate. • D) Improvement The relative percentage difference of your variation from baseline. • E) Sample size The number of visitors in your test.
Stats Engine corrects the pitfalls of A/B Testing with classical statistics.
A procedure for classical statistics (a.k.a. “T-test”, a.k.a. “Traditional Frequentist”, a.k.a “Fixed Horizon Testing” ) Farmer Fred wants to compare the effect of two fertilizers on crop yield. 1. Chooses how many plots to use (sample size). 2. Waits for a crop cycle, collects data once at the end. 3. Asks “What are the chances I’d have gotten these results if there was no difference between the fertilizers?” (a.k.a. p-value) If p-value < 5%, his results are significant. 4. Goes on, maybe to test irrigation methods.
1915 Data is expensive. Data is slow. Practitioners are trained. 2015 Data is cheap. Data is real-time. Practitioners are everyone. Classical statistics were designed for an offline world.
The modern A/B Testing procedure is different 1. Start without good estimate of sample size. 2. Check results early and often. Estimate ROI as quickly as possible. 3. Ask “How likely did my testing procedure give a wrong answer?” 4. Many variations on multiple goals, not just 1. 5. Iterate. Iterate. Iterate.
Pitfall 1. Peeking
p-Value < 5%. Significant! p-Value > 5%. Inconclusive. p-Value > 5%. Inconclusive. Min Sample Size Peeking Time Experiment Starts p-Value > 5%. Inconclusive.
Why is this a problem? There is a ~5% chance of false positive each time you peek.
p-Value < 5%. Significant! p-Value > 5%. Inconclusive. p-Value > 5%. Inconclusive. Min Sample Size Peeking Time Experiment Starts p-Value > 5%. Inconclusive. 4 peeks —> ~18% chance of seeing a false positive
Pitfall 2. Mistaking “False Positive Rate” for “Chance of a wrong conclusion”
Say I run an experiment.
1 original page, 5 variations, 6 goals = 30 “A/B Tests”
After I reach my minimum sample size, I stop the experiment and see 2 of my variations beating control and 1 variation losing to control
Winner Winner Loser Classical statistics guarantee <= 5% false positives. What % of my 2 winners and 1 loser do I expect to be false positives?
Winner Winner Loser Inconclusive Inconclusive Inconclusive Inconclusive Inconclusive Inconclusive Inconclusive Inconclusive Inconclusive Inconclusive Inconclusive Inconclusive Inconclusive Inconclusive Inconclusive Inconclusive Inconclusive Inconclusive Inconclusive Inconclusive Inconclusive Inconclusive Inconclusive Inconclusive Inconclusive Inconclusive Inconclusive 2 winners, 1 loser, and 27 inconclusives
Winner Winner Loser Inconclusive Inconclusive Inconclusive Inconclusive Inconclusive Inconclusive Inconclusive Inconclusive Inconclusive Inconclusive Inconclusive Inconclusive Inconclusive Inconclusive Inconclusive Inconclusive Inconclusive Inconclusive Inconclusive Inconclusive Inconclusive Inconclusive Inconclusive Inconclusive Inconclusive Inconclusive Inconclusive 30 A/B Tests x 5% = 1.5 false positives!
Winner Winner Loser Classical statistics guarantee <= 5% false positives. What % of my winners & losers do I expect to be false positives? Answer: C) With 30 A/B Tests, we can expect a = 50% chance of a wrong conclusion! In general, we can’t say without knowing how many other goals & variations were tested. 1.5 3
1. Which two A/B Testing pitfalls inflate error rates when using classical statistics, and are avoided with Stats Engine? 2. What are the three tradeoffs in an A/B Test? And how are they related? 3. How can you use Optimizely’s results page to best tune the tradeoffs to achieve your experimentation goals? After this workshop you should be able to answer …
1. Which two A/B Testing pitfalls inflate error rates when using classical statistics, and are avoided with Stats Engine? A. Peeking and mistaking “False Positive Rate” for “Chance of a wrong conclusion.” After this webinar, you should be able to answer …
The tradeoffs of A/B Testing
Error rates Runtime Improvement & Baseline CR
Error rates Runtime Improvement & Baseline CR “Chance of a wrong conclusion”
Error rates Runtime Improvement & Baseline CR “Chance of a wrong conclusion calling a non- winner a winner, or a non- loser a loser.”
Error rates Runtime Improvement & Baseline CR
Where is the error rate on Optimizely’s results page? I. II. III. IV. Statistical Significance = “Chance of a right conclusion” = (a.k.a.) 100 x (1 - False Discovery Rate)
How can you control the error rate?
Error rates Runtime Improvement & Baseline CR
Where is runtime on Optimizely’s results page?
Error rates Runtime Were you expecting a funny picture? Improvement & Baseline CR
Where is effect size on Optimizely’s results page?
Improvement & Baseline CR These three quantities are all … Error rates Runtime Inversely Related
At any number of visitors, the higher error rate I allow, the smaller improvement you can detect. Error rates Runtime Inversely Related Improvement & Baseline CR
Error rates Runtime Inversely Related At any error rate threshold, stopping your test earlier means you can only detect larger improvements. Improvement & Baseline CR
For any improvement, the lower error rate you want, the longer you need to run your test. Error rates Runtime Inversely Related Improvement & Baseline CR
What does this look like in practice? Average Visitors needed to reach significance with Stats Engine Improvement (relative) 5% 10% 25% Significance Threshold (Error Rate) 95 (5%) 62 K 14 K 1,800 90 (10%) 59 K 12 K 1,700 80 (20%) 53 K 11 K 1,500 Baseline conversion rate = 10%
~ 1 K visitors per day Average Visitors needed to reach significance with Stats Engine Improvement (relative) 5% 10% 25% Significance Threshold (Error Rate) 95 (5%) 62 K 14 K 1,800 90 (10%) 59 K 12 K 1,700 80 (20%) 53 K 11 K 1,500 (1 day) Baseline conversion rate = 10%
~ 10K visitors per day Average Visitors needed to reach significance with Stats Engine Improvement (relative) 5% 10% 25% Significance Threshold (Error Rate) 95 (5%) 62 K 14 K 1,800 90 (10%) 59 K 12 K 1,700 80 (20%) 53 K 11 K (1 day) 1,500 Baseline conversion rate = 10%
~ 50K visitors per day Average Visitors needed to reach significance with Stats Engine Improvement (relative) 3% 5% 10% Significance Threshold (Error Rate) 95 (5%) 190 K 62 K 14 K 90 (10%) 180 K 59 K 12 K 80 (20%) 160 K 53 K (1 day) 11 K Baseline conversion rate = 10%
> 100K visitors per day Average Visitors needed to reach significance with Stats Engine Improvement (relative) 3% 5% 10% Significance Threshold (Error Rate) 95 (5%) 190 K 62 K 14 K 90 (10%) 180 K 59 K 12 K 80 (20%) 160 K (1 day) 53 K 11 K Baseline conversion rate = 10%
1. Which two A/B Testing pitfalls inflate error rates when using classical statistics, and are avoided with Stats Engine? 2. What are the three tradeoffs in an A/B Test? And how are they related? 3. How can you use Optimizely’s results page to best tune the tradeoffs to achieve your experimentation goals? After this workshop, you should be able to answer …
1. Which two A/B Testing pitfalls inflate error rates when using classical statistics, and are avoided with Stats Engine? 2. What are the three tradeoffs in an A/B Test? And how are they related? A. Error Rates, Runtime, and Effect Size. They are all inversely related. After this workshop, you should be able to answer …
Use tradeoffs to align your testing goals
5% Error rates Runtime Improvement & Baseline CR Inversely Related +5%, 10% 53 K ? In the beginning, we make an educated guess …
… but after 1 day … Data! How can we update the tradeoffs?
1. Adjust your timeline
Improvement turns out to be better … Instead of: 53K - 10K = 43K 5% 1,600 Error rates Runtime +13%, 10% Inversely Related Improvement & Baseline CR
… or worse. 5% 75 K Error rates Runtime +2%, 8% Inversely Related Improvement & Baseline CR
2. Accept higher / lower error rate
Improvement turns out to be better … 1% 43 K Error rates Runtime +13%, 10% Inversely Related Improvement & Baseline CR
… or worse. 30% 43 K Error rates Runtime +2%, 8% Inversely Related Improvement & Baseline CR
3. Admit it. It’s inconclusive.
… or a lot worse. > 99% > 100K Error rates Runtime +.2%, 8% Inversely Related Improvement & Baseline CR iterate, iterate, iterate!
Your experiments will not always have the same improvement over time. So, run A/B Tests for at least a business cycle appropriate for that test and your company. Seasonality & Time Variation
1. Which two A/B Testing pitfalls inflate error rates when using classical statistics, and are avoided with Stats Engine? 2. What are the three tradeoffs in an A/B Test? And how are they related? 3. How can you use Optimizely’s results page to best tune the tradeoffs to achieve your experimentation goals? After this workshop, you should be able to answer …
1. Which two A/B Testing pitfalls inflate error rates when using classical statistics, and are avoided with for Stats Engine? 2. What are the three tradeoffs in one A/B Test? 3. How can you use Optimizely’s results page to best tune the tradeoffs to achieve your experimentation goals? A. Adjust your timeline. Accept higher / lower error rate. Admit an inconclusive result. After this workshop, you should be able to answer …
1. Which two A/B Testing pitfalls inflate error rates when using classical statistics, and are avoided with Stats Engine? A. Peeking and mistaking “False Positive Rate” for “Chance of a Wrong Answer.” 2. What are the three tradeoffs in one A/B Test? B. Error Rates, Runtime, and Effect Size. They are all negatively related. 3. How can you use Optimizely’s results page to best tune the tradeoffs to achieve your experimentation goals? C. Accept higher / lower error rate. Adjust your timeline. Admit an inconclusive result. 1. Which two A/B Testing pitfalls inflate error rates when using classical statistics, and are avoided with Stats Engine? A. Peeking and mistaking “False Positive Rate” for “Chance of a Wrong Answer.” 2. What are the three tradeoffs in one A/B Test? B. Error Rates, Runtime, and Effect Size. They are all negatively related. 3. How can you use Optimizely’s results page to best tune the tradeoffs to achieve your experimentation goals? C. Accept higher / lower error rate. Adjust your timeline. Admit an inconclusive result. 1. Which two A/B Testing pitfalls inflate error rates when using classical statistics, and are avoided with Stats Engine? A. Peeking and mistaking “False Positive Rate” for “Chance of a Wrong Answer.” 2. What are the three tradeoffs in one A/B Test? B. Error Rates, Runtime, and Effect Size. They are all negatively related. 3. How can you use Optimizely’s results page to best tune the tradeoffs to achieve your experimentation goals? C. Accept higher / lower error rate. Adjust your timeline. Admit an inconclusive result. 1. Which two A/B Testing pitfalls inflate error rates when using classical statistics, and are avoided with Stats Engine? A. Peeking and mistaking “False Positive Rate” for “Chance of a Wrong Answer.” 2. What are the three tradeoffs in one A/B Test? B. Error Rates, Runtime, and Effect Size. They are all negatively related. 3. How can you use Optimizely’s results page to best tune the tradeoffs to achieve your experimentation goals? C. Accept higher / lower error rate. Adjust your timeline. Admit an inconclusive result. Review
Preview: How many goals and variations should I use?
Stats Engine is more conservative when there are more goals that are not affected by a variation. So, adding a lot of “random” goals will slow down your experiment.
Tips & Tricks for using Stats Engine with multiple goals and variations • Ask: Which goal is most important to me? -This should be the primary goal (not impacted by all other goals) • Run large, or large multivariate tests without fear of finding spurious results, but be prepared for the cost of exploration. • For maximum velocity, only test goals and variations that you believe will have highest impact.
1. Which two A/B Testing pitfalls inflate error rates when using classical statistics, and are avoided with Stats Engine? A. Peeking and mistaking “False Positive Rate” for “Chance of a Wrong Answer.” 2. What are the three tradeoffs in one A/B Test? B. Error Rates, Runtime, and Effect Size. They are all negatively related. 3. How can you use Optimizely’s results page to best tune the tradeoffs to achieve your experimentation goals? C. Accept higher / lower error rate. Adjust your timeline. Admit an inconclusive result. 1. Which two A/B Testing pitfalls inflate error rates when using classical statistics, and are avoided with Stats Engine? A. Peeking and mistaking “False Positive Rate” for “Chance of a Wrong Answer.” 2. What are the three tradeoffs in one A/B Test? B. Error Rates, Runtime, and Effect Size. They are all negatively related. 3. How can you use Optimizely’s results page to best tune the tradeoffs to achieve your experimentation goals? C. Accept higher / lower error rate. Adjust your timeline. Admit an inconclusive result. 1. Which two A/B Testing pitfalls inflate error rates when using classical statistics, and are avoided with Stats Engine? A. Peeking and mistaking “False Positive Rate” for “Chance of a Wrong Answer.” 2. What are the three tradeoffs in one A/B Test? B. Error Rates, Runtime, and Effect Size. They are all negatively related. 3. How can you use Optimizely’s results page to best tune the tradeoffs to achieve your experimentation goals? C. Accept higher / lower error rate. Adjust your timeline. Admit an inconclusive result. 1. Which two A/B Testing pitfalls inflate error rates when using classical statistics, and are avoided with Stats Engine? A. Peeking and mistaking “False Positive Rate” for “Chance of a Wrong Answer.” 2. What are the three tradeoffs in one A/B Test? B. Error Rates, Runtime, and Effect Size. They are all negatively related. 3. How can you use Optimizely’s results page to best tune the tradeoffs to achieve your experimentation goals? C. Accept higher / lower error rate. Adjust your timeline. Admit an inconclusive result. Review

Recommended

Workshop 6: Build Your Organization's Optimization Culture
Workshop 6: Build Your Organization's Optimization CultureWorkshop 6: Build Your Organization's Optimization Culture
Workshop 6: Build Your Organization's Optimization Culture
Optimizely
 
Optimizely Workshop 1: Prioritize your roadmap
Optimizely Workshop 1: Prioritize your roadmapOptimizely Workshop 1: Prioritize your roadmap
Optimizely Workshop 1: Prioritize your roadmap
Optimizely
 
Optimizely Workshop: Mobile Walkthrough
Optimizely Workshop: Mobile Walkthrough Optimizely Workshop: Mobile Walkthrough
Optimizely Workshop: Mobile Walkthrough
Optimizely
 
Website Redesigns: Why they Fail and How to Ensure Success
Website Redesigns: Why they Fail and How to Ensure SuccessWebsite Redesigns: Why they Fail and How to Ensure Success
Website Redesigns: Why they Fail and How to Ensure Success
Optimizely
 
Testing Your Testing Program
Testing Your Testing ProgramTesting Your Testing Program
Testing Your Testing Program
Jill Martay
 
Optimizely & Photobox - DON'T PANIC: The No-Confusion Experimentation Startup...
Optimizely & Photobox - DON'T PANIC: The No-Confusion Experimentation Startup...Optimizely & Photobox - DON'T PANIC: The No-Confusion Experimentation Startup...
Optimizely & Photobox - DON'T PANIC: The No-Confusion Experimentation Startup...
Optimizely
 
Losing is the New Winning
Losing is the New WinningLosing is the New Winning
Losing is the New Winning
Optimizely
 
Optimizely Experience Customer Story - Atlassian
Optimizely Experience Customer Story - AtlassianOptimizely Experience Customer Story - Atlassian
Optimizely Experience Customer Story - Atlassian
Optimizely
 

Recommended

Workshop 6: Build Your Organization's Optimization Culture
Workshop 6: Build Your Organization's Optimization CultureWorkshop 6: Build Your Organization's Optimization Culture
Workshop 6: Build Your Organization's Optimization Culture
Optimizely
 
Optimizely Workshop 1: Prioritize your roadmap
Optimizely Workshop 1: Prioritize your roadmapOptimizely Workshop 1: Prioritize your roadmap
Optimizely Workshop 1: Prioritize your roadmap
Optimizely
 
Optimizely Workshop: Mobile Walkthrough
Optimizely Workshop: Mobile Walkthrough Optimizely Workshop: Mobile Walkthrough
Optimizely Workshop: Mobile Walkthrough
Optimizely
 
Website Redesigns: Why they Fail and How to Ensure Success
Website Redesigns: Why they Fail and How to Ensure SuccessWebsite Redesigns: Why they Fail and How to Ensure Success
Website Redesigns: Why they Fail and How to Ensure Success
Optimizely
 
Testing Your Testing Program
Testing Your Testing ProgramTesting Your Testing Program
Testing Your Testing Program
Jill Martay
 
Optimizely & Photobox - DON'T PANIC: The No-Confusion Experimentation Startup...
Optimizely & Photobox - DON'T PANIC: The No-Confusion Experimentation Startup...Optimizely & Photobox - DON'T PANIC: The No-Confusion Experimentation Startup...
Optimizely & Photobox - DON'T PANIC: The No-Confusion Experimentation Startup...
Optimizely
 
Losing is the New Winning
Losing is the New WinningLosing is the New Winning
Losing is the New Winning
Optimizely
 
Optimizely Experience Customer Story - Atlassian
Optimizely Experience Customer Story - AtlassianOptimizely Experience Customer Story - Atlassian
Optimizely Experience Customer Story - Atlassian
Optimizely
 
An Experimentation Framework: How to Position for Triple Digit Growth
An Experimentation Framework: How to Position for Triple Digit GrowthAn Experimentation Framework: How to Position for Triple Digit Growth
An Experimentation Framework: How to Position for Triple Digit Growth
Optimizely
 
Optimizely's Optimization Benchmark Findings Webinar Slides
Optimizely's Optimization Benchmark Findings Webinar SlidesOptimizely's Optimization Benchmark Findings Webinar Slides
Optimizely's Optimization Benchmark Findings Webinar Slides
Optimizely
 
Optimizely, HEMA & Farfetch - Unlock the Potential of Digital Experimentation...
Optimizely, HEMA & Farfetch - Unlock the Potential of Digital Experimentation...Optimizely, HEMA & Farfetch - Unlock the Potential of Digital Experimentation...
Optimizely, HEMA & Farfetch - Unlock the Potential of Digital Experimentation...
Optimizely
 
Build a Winning Conversion Optimization Strategy
Build a Winning Conversion Optimization StrategyBuild a Winning Conversion Optimization Strategy
Build a Winning Conversion Optimization Strategy
Savage Marketing
 
Optimizing Your B2B Demand Generation Machine
Optimizing Your B2B Demand Generation MachineOptimizing Your B2B Demand Generation Machine
Optimizing Your B2B Demand Generation Machine
Optimizely
 
Experimentation as a growth strategy: A conversation with The Motley Fool
Experimentation as a growth strategy: A conversation with The Motley FoolExperimentation as a growth strategy: A conversation with The Motley Fool
Experimentation as a growth strategy: A conversation with The Motley Fool
Chris Goward
 
VWO Webinar: How To Plan Your Optimisation Roadmap
VWO Webinar: How To Plan Your Optimisation RoadmapVWO Webinar: How To Plan Your Optimisation Roadmap
VWO Webinar: How To Plan Your Optimisation Roadmap
VWO
 
Making Your Hypothesis Work Harder to Inform Future Product Strategy
Making Your Hypothesis Work Harder to Inform Future Product StrategyMaking Your Hypothesis Work Harder to Inform Future Product Strategy
Making Your Hypothesis Work Harder to Inform Future Product Strategy
Optimizely
 
4 Steps Toward Scientific A/B Testing
4 Steps Toward Scientific A/B Testing4 Steps Toward Scientific A/B Testing
4 Steps Toward Scientific A/B Testing
Janessa Lantz
 
Improve your content: The What, Why, Where and How about A/B Testing
Improve your content: The What, Why, Where and How about A/B TestingImprove your content: The What, Why, Where and How about A/B Testing
Improve your content: The What, Why, Where and How about A/B Testing
introtodigital
 
#ImpactSalesforceSaturday: Drum into understanding of prediction builder with...
#ImpactSalesforceSaturday: Drum into understanding of prediction builder with...#ImpactSalesforceSaturday: Drum into understanding of prediction builder with...
#ImpactSalesforceSaturday: Drum into understanding of prediction builder with...
New Delhi Salesforce Developer Group
 
A/B Mythbusters: Common Optimization Objections Debunked
A/B Mythbusters: Common Optimization Objections DebunkedA/B Mythbusters: Common Optimization Objections Debunked
A/B Mythbusters: Common Optimization Objections Debunked
Optimizely
 
SXSW 2016 - Everything you think about A/B testing is wrong
SXSW 2016 - Everything you think about A/B testing is wrongSXSW 2016 - Everything you think about A/B testing is wrong
SXSW 2016 - Everything you think about A/B testing is wrong
Dan Chuparkoff
 
Product Experimentation | Forming Strong Experiment Hypotheses
Product Experimentation | Forming Strong Experiment HypothesesProduct Experimentation | Forming Strong Experiment Hypotheses
Product Experimentation | Forming Strong Experiment Hypotheses
Optimizely
 
Definition of A/B testing and Case Studies by Optimizely
Definition of A/B testing and Case Studies by OptimizelyDefinition of A/B testing and Case Studies by Optimizely
Definition of A/B testing and Case Studies by Optimizely
RusseWeb
 
How To Build a Winning Experimentation Program & Team | Optimizely ANZ Webinar 8
How To Build a Winning Experimentation Program & Team | Optimizely ANZ Webinar 8How To Build a Winning Experimentation Program & Team | Optimizely ANZ Webinar 8
How To Build a Winning Experimentation Program & Team | Optimizely ANZ Webinar 8
Optimizely
 
Cro webinar what you're doing wrong in your cro program (sharable version)
Cro webinar what you're doing wrong in your cro program (sharable version)Cro webinar what you're doing wrong in your cro program (sharable version)
Cro webinar what you're doing wrong in your cro program (sharable version)
VWO
 
The Science of Getting Testing Right
The Science of Getting Testing RightThe Science of Getting Testing Right
The Science of Getting Testing Right
Optimizely
 
Intuit - How to Scale Your Experimentation Program
Intuit - How to Scale Your Experimentation ProgramIntuit - How to Scale Your Experimentation Program
Intuit - How to Scale Your Experimentation Program
Optimizely
 
Intro to A/B Testing by Ever's Senior Product Manager
Intro to A/B Testing by Ever's Senior Product ManagerIntro to A/B Testing by Ever's Senior Product Manager
Intro to A/B Testing by Ever's Senior Product Manager
Product School
 
Meaningful Data - Best Internet Conference 2015 (Lithuania)
Meaningful Data - Best Internet Conference 2015 (Lithuania)Meaningful Data - Best Internet Conference 2015 (Lithuania)
Meaningful Data - Best Internet Conference 2015 (Lithuania)
Simo Ahava
 
7 Steps for Applying Big Data Patterns to Decision Making
7 Steps for Applying Big Data Patterns to Decision Making7 Steps for Applying Big Data Patterns to Decision Making
7 Steps for Applying Big Data Patterns to Decision Making
Wiley
 

More Related Content

What's hot

An Experimentation Framework: How to Position for Triple Digit Growth
An Experimentation Framework: How to Position for Triple Digit GrowthAn Experimentation Framework: How to Position for Triple Digit Growth
An Experimentation Framework: How to Position for Triple Digit Growth
Optimizely
 
Optimizely's Optimization Benchmark Findings Webinar Slides
Optimizely's Optimization Benchmark Findings Webinar SlidesOptimizely's Optimization Benchmark Findings Webinar Slides
Optimizely's Optimization Benchmark Findings Webinar Slides
Optimizely
 
Optimizely, HEMA & Farfetch - Unlock the Potential of Digital Experimentation...
Optimizely, HEMA & Farfetch - Unlock the Potential of Digital Experimentation...Optimizely, HEMA & Farfetch - Unlock the Potential of Digital Experimentation...
Optimizely, HEMA & Farfetch - Unlock the Potential of Digital Experimentation...
Optimizely
 
Build a Winning Conversion Optimization Strategy
Build a Winning Conversion Optimization StrategyBuild a Winning Conversion Optimization Strategy
Build a Winning Conversion Optimization Strategy
Savage Marketing
 
Optimizing Your B2B Demand Generation Machine
Optimizing Your B2B Demand Generation MachineOptimizing Your B2B Demand Generation Machine
Optimizing Your B2B Demand Generation Machine
Optimizely
 
Experimentation as a growth strategy: A conversation with The Motley Fool
Experimentation as a growth strategy: A conversation with The Motley FoolExperimentation as a growth strategy: A conversation with The Motley Fool
Experimentation as a growth strategy: A conversation with The Motley Fool
Chris Goward
 
VWO Webinar: How To Plan Your Optimisation Roadmap
VWO Webinar: How To Plan Your Optimisation RoadmapVWO Webinar: How To Plan Your Optimisation Roadmap
VWO Webinar: How To Plan Your Optimisation Roadmap
VWO
 
Making Your Hypothesis Work Harder to Inform Future Product Strategy
Making Your Hypothesis Work Harder to Inform Future Product StrategyMaking Your Hypothesis Work Harder to Inform Future Product Strategy
Making Your Hypothesis Work Harder to Inform Future Product Strategy
Optimizely
 
4 Steps Toward Scientific A/B Testing
4 Steps Toward Scientific A/B Testing4 Steps Toward Scientific A/B Testing
4 Steps Toward Scientific A/B Testing
Janessa Lantz
 
Improve your content: The What, Why, Where and How about A/B Testing
Improve your content: The What, Why, Where and How about A/B TestingImprove your content: The What, Why, Where and How about A/B Testing
Improve your content: The What, Why, Where and How about A/B Testing
introtodigital
 
#ImpactSalesforceSaturday: Drum into understanding of prediction builder with...
#ImpactSalesforceSaturday: Drum into understanding of prediction builder with...#ImpactSalesforceSaturday: Drum into understanding of prediction builder with...
#ImpactSalesforceSaturday: Drum into understanding of prediction builder with...
New Delhi Salesforce Developer Group
 
A/B Mythbusters: Common Optimization Objections Debunked
A/B Mythbusters: Common Optimization Objections DebunkedA/B Mythbusters: Common Optimization Objections Debunked
A/B Mythbusters: Common Optimization Objections Debunked
Optimizely
 
SXSW 2016 - Everything you think about A/B testing is wrong
SXSW 2016 - Everything you think about A/B testing is wrongSXSW 2016 - Everything you think about A/B testing is wrong
SXSW 2016 - Everything you think about A/B testing is wrong
Dan Chuparkoff
 
Product Experimentation | Forming Strong Experiment Hypotheses
Product Experimentation | Forming Strong Experiment HypothesesProduct Experimentation | Forming Strong Experiment Hypotheses
Product Experimentation | Forming Strong Experiment Hypotheses
Optimizely
 
Definition of A/B testing and Case Studies by Optimizely
Definition of A/B testing and Case Studies by OptimizelyDefinition of A/B testing and Case Studies by Optimizely
Definition of A/B testing and Case Studies by Optimizely
RusseWeb
 
How To Build a Winning Experimentation Program & Team | Optimizely ANZ Webinar 8
How To Build a Winning Experimentation Program & Team | Optimizely ANZ Webinar 8How To Build a Winning Experimentation Program & Team | Optimizely ANZ Webinar 8
How To Build a Winning Experimentation Program & Team | Optimizely ANZ Webinar 8
Optimizely
 
Cro webinar what you're doing wrong in your cro program (sharable version)
Cro webinar what you're doing wrong in your cro program (sharable version)Cro webinar what you're doing wrong in your cro program (sharable version)
Cro webinar what you're doing wrong in your cro program (sharable version)
VWO
 
The Science of Getting Testing Right
The Science of Getting Testing RightThe Science of Getting Testing Right
The Science of Getting Testing Right
Optimizely
 
Intuit - How to Scale Your Experimentation Program
Intuit - How to Scale Your Experimentation ProgramIntuit - How to Scale Your Experimentation Program
Intuit - How to Scale Your Experimentation Program
Optimizely
 
Intro to A/B Testing by Ever's Senior Product Manager
Intro to A/B Testing by Ever's Senior Product ManagerIntro to A/B Testing by Ever's Senior Product Manager
Intro to A/B Testing by Ever's Senior Product Manager
Product School
 

What's hot (20)

An Experimentation Framework: How to Position for Triple Digit Growth
An Experimentation Framework: How to Position for Triple Digit GrowthAn Experimentation Framework: How to Position for Triple Digit Growth
An Experimentation Framework: How to Position for Triple Digit Growth
 
Optimizely's Optimization Benchmark Findings Webinar Slides
Optimizely's Optimization Benchmark Findings Webinar SlidesOptimizely's Optimization Benchmark Findings Webinar Slides
Optimizely's Optimization Benchmark Findings Webinar Slides
 
Optimizely, HEMA & Farfetch - Unlock the Potential of Digital Experimentation...
Optimizely, HEMA & Farfetch - Unlock the Potential of Digital Experimentation...Optimizely, HEMA & Farfetch - Unlock the Potential of Digital Experimentation...
Optimizely, HEMA & Farfetch - Unlock the Potential of Digital Experimentation...
 
Build a Winning Conversion Optimization Strategy
Build a Winning Conversion Optimization StrategyBuild a Winning Conversion Optimization Strategy
Build a Winning Conversion Optimization Strategy
 
Optimizing Your B2B Demand Generation Machine
Optimizing Your B2B Demand Generation MachineOptimizing Your B2B Demand Generation Machine
Optimizing Your B2B Demand Generation Machine
 
Experimentation as a growth strategy: A conversation with The Motley Fool
Experimentation as a growth strategy: A conversation with The Motley FoolExperimentation as a growth strategy: A conversation with The Motley Fool
Experimentation as a growth strategy: A conversation with The Motley Fool
 
VWO Webinar: How To Plan Your Optimisation Roadmap
VWO Webinar: How To Plan Your Optimisation RoadmapVWO Webinar: How To Plan Your Optimisation Roadmap
VWO Webinar: How To Plan Your Optimisation Roadmap
 
Making Your Hypothesis Work Harder to Inform Future Product Strategy
Making Your Hypothesis Work Harder to Inform Future Product StrategyMaking Your Hypothesis Work Harder to Inform Future Product Strategy
Making Your Hypothesis Work Harder to Inform Future Product Strategy
 
4 Steps Toward Scientific A/B Testing
4 Steps Toward Scientific A/B Testing4 Steps Toward Scientific A/B Testing
4 Steps Toward Scientific A/B Testing
 
Improve your content: The What, Why, Where and How about A/B Testing
Improve your content: The What, Why, Where and How about A/B TestingImprove your content: The What, Why, Where and How about A/B Testing
Improve your content: The What, Why, Where and How about A/B Testing
 
#ImpactSalesforceSaturday: Drum into understanding of prediction builder with...
#ImpactSalesforceSaturday: Drum into understanding of prediction builder with...#ImpactSalesforceSaturday: Drum into understanding of prediction builder with...
#ImpactSalesforceSaturday: Drum into understanding of prediction builder with...
 
A/B Mythbusters: Common Optimization Objections Debunked
A/B Mythbusters: Common Optimization Objections DebunkedA/B Mythbusters: Common Optimization Objections Debunked
A/B Mythbusters: Common Optimization Objections Debunked
 
SXSW 2016 - Everything you think about A/B testing is wrong
SXSW 2016 - Everything you think about A/B testing is wrongSXSW 2016 - Everything you think about A/B testing is wrong
SXSW 2016 - Everything you think about A/B testing is wrong
 
Product Experimentation | Forming Strong Experiment Hypotheses
Product Experimentation | Forming Strong Experiment HypothesesProduct Experimentation | Forming Strong Experiment Hypotheses
Product Experimentation | Forming Strong Experiment Hypotheses
 
Definition of A/B testing and Case Studies by Optimizely
Definition of A/B testing and Case Studies by OptimizelyDefinition of A/B testing and Case Studies by Optimizely
Definition of A/B testing and Case Studies by Optimizely
 
How To Build a Winning Experimentation Program & Team | Optimizely ANZ Webinar 8
How To Build a Winning Experimentation Program & Team | Optimizely ANZ Webinar 8How To Build a Winning Experimentation Program & Team | Optimizely ANZ Webinar 8
How To Build a Winning Experimentation Program & Team | Optimizely ANZ Webinar 8
 
Cro webinar what you're doing wrong in your cro program (sharable version)
Cro webinar what you're doing wrong in your cro program (sharable version)Cro webinar what you're doing wrong in your cro program (sharable version)
Cro webinar what you're doing wrong in your cro program (sharable version)
 
The Science of Getting Testing Right
The Science of Getting Testing RightThe Science of Getting Testing Right
The Science of Getting Testing Right
 
Intuit - How to Scale Your Experimentation Program
Intuit - How to Scale Your Experimentation ProgramIntuit - How to Scale Your Experimentation Program
Intuit - How to Scale Your Experimentation Program
 
Intro to A/B Testing by Ever's Senior Product Manager
Intro to A/B Testing by Ever's Senior Product ManagerIntro to A/B Testing by Ever's Senior Product Manager
Intro to A/B Testing by Ever's Senior Product Manager
 

Viewers also liked

Meaningful Data - Best Internet Conference 2015 (Lithuania)
Meaningful Data - Best Internet Conference 2015 (Lithuania)Meaningful Data - Best Internet Conference 2015 (Lithuania)
Meaningful Data - Best Internet Conference 2015 (Lithuania)
Simo Ahava
 
7 Steps for Applying Big Data Patterns to Decision Making
7 Steps for Applying Big Data Patterns to Decision Making7 Steps for Applying Big Data Patterns to Decision Making
7 Steps for Applying Big Data Patterns to Decision Making
Wiley
 
Machine Learning, Deep Learning and Data Analysis Introduction
Machine Learning, Deep Learning and Data Analysis IntroductionMachine Learning, Deep Learning and Data Analysis Introduction
Machine Learning, Deep Learning and Data Analysis Introduction
Te-Yen Liu
 
The Human Side of Data By Colin Strong
The Human Side of Data By Colin StrongThe Human Side of Data By Colin Strong
The Human Side of Data By Colin Strong
MarTech Conference
 
Big Data Analytics for Non Programmers
Big Data Analytics for Non ProgrammersBig Data Analytics for Non Programmers
Big Data Analytics for Non Programmers
Edureka!
 
Grounded theory methodology of qualitative data analysis
Grounded theory methodology of qualitative data analysisGrounded theory methodology of qualitative data analysis
Grounded theory methodology of qualitative data analysis
Dr. Shiv S Tripathi
 
Palantir, Quid, RecordedFuture: Augmented Intelligence Frontier
Palantir, Quid, RecordedFuture: Augmented Intelligence FrontierPalantir, Quid, RecordedFuture: Augmented Intelligence Frontier
Palantir, Quid, RecordedFuture: Augmented Intelligence Frontier
Daniel Kornev
 
PROG_UntoldStory ISV eBook_0706c FINAL
PROG_UntoldStory ISV eBook_0706c FINALPROG_UntoldStory ISV eBook_0706c FINAL
PROG_UntoldStory ISV eBook_0706c FINALSolarWinds MSP
 
Cyber Tech Israel 2016: Get Your Head in the Cloud
Cyber Tech Israel 2016: Get Your Head in the CloudCyber Tech Israel 2016: Get Your Head in the Cloud
Cyber Tech Israel 2016: Get Your Head in the Cloud
Symantec
 
IOT & Machine Learning
IOT & Machine LearningIOT & Machine Learning
IOT & Machine Learning
Avanade Nederland
 
Lambda architecture for real time big data
Lambda architecture for real time big dataLambda architecture for real time big data
Lambda architecture for real time big data
Trieu Nguyen
 
Big Data Revolution: Are You Ready for the Data Overload?
Big Data Revolution: Are You Ready for the Data Overload?Big Data Revolution: Are You Ready for the Data Overload?
Big Data Revolution: Are You Ready for the Data Overload?
Aleah Radovich
 
Riot Games Scalable Data Warehouse Lecture at UCSB / UCLA
Riot Games Scalable Data Warehouse Lecture at UCSB / UCLARiot Games Scalable Data Warehouse Lecture at UCSB / UCLA
Riot Games Scalable Data Warehouse Lecture at UCSB / UCLA
sean_seannery
 
How to Conquer Artificial Intelligence
How to Conquer Artificial IntelligenceHow to Conquer Artificial Intelligence
How to Conquer Artificial Intelligence
The Added Value Group
 
Effective presentation skills
Effective presentation skillsEffective presentation skills
Effective presentation skills
Subagini Manivannan
 
How to give a good 10min presentation
How to give a good 10min presentation How to give a good 10min presentation
How to give a good 10min presentation
Jodie Martin
 
5 Factors Impacting Your Big Data Project's Performance
5 Factors Impacting Your Big Data Project's Performance 5 Factors Impacting Your Big Data Project's Performance
5 Factors Impacting Your Big Data Project's Performance
Qubole
 
Introduction to Machine Learning
Introduction to Machine LearningIntroduction to Machine Learning
Introduction to Machine Learning
Rahul Jain
 
The 2016 CES Report: The Trend Behind the Trend
The 2016 CES Report: The Trend Behind the TrendThe 2016 CES Report: The Trend Behind the Trend
The 2016 CES Report: The Trend Behind the Trend
360i
 
Introduction to Machine Learning
Introduction to Machine LearningIntroduction to Machine Learning
Introduction to Machine LearningLior Rokach
 

Viewers also liked (20)

Meaningful Data - Best Internet Conference 2015 (Lithuania)
Meaningful Data - Best Internet Conference 2015 (Lithuania)Meaningful Data - Best Internet Conference 2015 (Lithuania)
Meaningful Data - Best Internet Conference 2015 (Lithuania)
 
7 Steps for Applying Big Data Patterns to Decision Making
7 Steps for Applying Big Data Patterns to Decision Making7 Steps for Applying Big Data Patterns to Decision Making
7 Steps for Applying Big Data Patterns to Decision Making
 
Machine Learning, Deep Learning and Data Analysis Introduction
Machine Learning, Deep Learning and Data Analysis IntroductionMachine Learning, Deep Learning and Data Analysis Introduction
Machine Learning, Deep Learning and Data Analysis Introduction
 
The Human Side of Data By Colin Strong
The Human Side of Data By Colin StrongThe Human Side of Data By Colin Strong
The Human Side of Data By Colin Strong
 
Big Data Analytics for Non Programmers
Big Data Analytics for Non ProgrammersBig Data Analytics for Non Programmers
Big Data Analytics for Non Programmers
 
Grounded theory methodology of qualitative data analysis
Grounded theory methodology of qualitative data analysisGrounded theory methodology of qualitative data analysis
Grounded theory methodology of qualitative data analysis
 
Palantir, Quid, RecordedFuture: Augmented Intelligence Frontier
Palantir, Quid, RecordedFuture: Augmented Intelligence FrontierPalantir, Quid, RecordedFuture: Augmented Intelligence Frontier
Palantir, Quid, RecordedFuture: Augmented Intelligence Frontier
 
PROG_UntoldStory ISV eBook_0706c FINAL
PROG_UntoldStory ISV eBook_0706c FINALPROG_UntoldStory ISV eBook_0706c FINAL
PROG_UntoldStory ISV eBook_0706c FINAL
 
Cyber Tech Israel 2016: Get Your Head in the Cloud
Cyber Tech Israel 2016: Get Your Head in the CloudCyber Tech Israel 2016: Get Your Head in the Cloud
Cyber Tech Israel 2016: Get Your Head in the Cloud
 
IOT & Machine Learning
IOT & Machine LearningIOT & Machine Learning
IOT & Machine Learning
 
Lambda architecture for real time big data
Lambda architecture for real time big dataLambda architecture for real time big data
Lambda architecture for real time big data
 
Big Data Revolution: Are You Ready for the Data Overload?
Big Data Revolution: Are You Ready for the Data Overload?Big Data Revolution: Are You Ready for the Data Overload?
Big Data Revolution: Are You Ready for the Data Overload?
 
Riot Games Scalable Data Warehouse Lecture at UCSB / UCLA
Riot Games Scalable Data Warehouse Lecture at UCSB / UCLARiot Games Scalable Data Warehouse Lecture at UCSB / UCLA
Riot Games Scalable Data Warehouse Lecture at UCSB / UCLA
 
How to Conquer Artificial Intelligence
How to Conquer Artificial IntelligenceHow to Conquer Artificial Intelligence
How to Conquer Artificial Intelligence
 
Effective presentation skills
Effective presentation skillsEffective presentation skills
Effective presentation skills
 
How to give a good 10min presentation
How to give a good 10min presentation How to give a good 10min presentation
How to give a good 10min presentation
 
5 Factors Impacting Your Big Data Project's Performance
5 Factors Impacting Your Big Data Project's Performance 5 Factors Impacting Your Big Data Project's Performance
5 Factors Impacting Your Big Data Project's Performance
 
Introduction to Machine Learning
Introduction to Machine LearningIntroduction to Machine Learning
Introduction to Machine Learning
 
The 2016 CES Report: The Trend Behind the Trend
The 2016 CES Report: The Trend Behind the TrendThe 2016 CES Report: The Trend Behind the Trend
The 2016 CES Report: The Trend Behind the Trend
 
Introduction to Machine Learning
Introduction to Machine LearningIntroduction to Machine Learning
Introduction to Machine Learning
 

Similar to Optimizely Workshop: Take Action on Results with Statistics

Tale of Two Tests
Tale of Two TestsTale of Two Tests
Tale of Two Tests
Optimizely
 
The Finishing Line
The Finishing LineThe Finishing Line
The Finishing Line
Oban International
 
Opticon 2017 Experimenting with Stats Engine
Opticon 2017 Experimenting with Stats EngineOpticon 2017 Experimenting with Stats Engine
Opticon 2017 Experimenting with Stats Engine
Optimizely
 
What is AB Testing? A Beginner's Guide
What is AB Testing? A Beginner's GuideWhat is AB Testing? A Beginner's Guide
What is AB Testing? A Beginner's Guide
PPCexpo
 
신뢰할 수 있는 A/B 테스트를 위해 알아야 할 것들
신뢰할 수 있는 A/B 테스트를 위해 알아야 할 것들신뢰할 수 있는 A/B 테스트를 위해 알아야 할 것들
신뢰할 수 있는 A/B 테스트를 위해 알아야 할 것들
Minho Lee
 
Basics of AB testing in online products
Basics of AB testing in online productsBasics of AB testing in online products
Basics of AB testing in online products
Ashish Dua
 
The ways to fuck up ab testing (from data products meetup)
The ways to fuck up ab testing (from data products meetup)The ways to fuck up ab testing (from data products meetup)
The ways to fuck up ab testing (from data products meetup)
Data Products Meetup
 
Can I Test More Than One Variable at a Time? Statisticians answer some of th...
Can I Test More Than One Variable at a Time? Statisticians answer some of th...Can I Test More Than One Variable at a Time? Statisticians answer some of th...
Can I Test More Than One Variable at a Time? Statisticians answer some of th...
MarketingExperiments
 
Automated testing handbook
Automated testing handbookAutomated testing handbook
Automated testing handbookAndrei Hortúa
 
SAMPLE SIZE – The indispensable A/B test calculation that you’re not making
SAMPLE SIZE – The indispensable A/B test calculation that you’re not makingSAMPLE SIZE – The indispensable A/B test calculation that you’re not making
SAMPLE SIZE – The indispensable A/B test calculation that you’re not making
Zack Notes
 
Ab testing 101
Ab testing 101Ab testing 101
Ab testing 101
Ashish Dua
 
Calculating a Sample Size
Calculating a Sample SizeCalculating a Sample Size
Calculating a Sample Size
Matt Hansen
 
SOFTWARE TESTING TRAFUNDAMENTALS OF SOFTWARE TESTING.pptx
SOFTWARE TESTING TRAFUNDAMENTALS OF SOFTWARE TESTING.pptxSOFTWARE TESTING TRAFUNDAMENTALS OF SOFTWARE TESTING.pptx
SOFTWARE TESTING TRAFUNDAMENTALS OF SOFTWARE TESTING.pptx
Financial Services Innovators
 
Data-Driven UI/UX Design with A/B Testing
Data-Driven UI/UX Design with A/B TestingData-Driven UI/UX Design with A/B Testing
Data-Driven UI/UX Design with A/B Testing
Jack Nguyen (Hung Tien)
 
A/B Testing myths and quagmires
A/B Testing myths and quagmiresA/B Testing myths and quagmires
A/B Testing myths and quagmires
Married2Growth
 
Critical Checks for Pharmaceuticals and Healthcare: Validating Your Data Inte...
Critical Checks for Pharmaceuticals and Healthcare: Validating Your Data Inte...Critical Checks for Pharmaceuticals and Healthcare: Validating Your Data Inte...
Critical Checks for Pharmaceuticals and Healthcare: Validating Your Data Inte...
Minitab, LLC
 
Download Invesp’s The Essentials of Multivariate & AB Testing
Download Invesp’s The Essentials of Multivariate & AB TestingDownload Invesp’s The Essentials of Multivariate & AB Testing
Download Invesp’s The Essentials of Multivariate & AB Testing
Duy, Vo Hoang
 
Why learn Six Sigma, 4,28,15
Why learn Six Sigma, 4,28,15Why learn Six Sigma, 4,28,15
Why learn Six Sigma, 4,28,15James F. McCarthy
 
6 Guidelines for A/B Testing
6 Guidelines for A/B Testing6 Guidelines for A/B Testing
6 Guidelines for A/B Testing
Emily Robinson
 
Data-Driven Product Management by Shutterfly Director of Product
Data-Driven Product Management by Shutterfly Director of ProductData-Driven Product Management by Shutterfly Director of Product
Data-Driven Product Management by Shutterfly Director of Product
Product School
 

Similar to Optimizely Workshop: Take Action on Results with Statistics (20)

Tale of Two Tests
Tale of Two TestsTale of Two Tests
Tale of Two Tests
 
The Finishing Line
The Finishing LineThe Finishing Line
The Finishing Line
 
Opticon 2017 Experimenting with Stats Engine
Opticon 2017 Experimenting with Stats EngineOpticon 2017 Experimenting with Stats Engine
Opticon 2017 Experimenting with Stats Engine
 
What is AB Testing? A Beginner's Guide
What is AB Testing? A Beginner's GuideWhat is AB Testing? A Beginner's Guide
What is AB Testing? A Beginner's Guide
 
신뢰할 수 있는 A/B 테스트를 위해 알아야 할 것들
신뢰할 수 있는 A/B 테스트를 위해 알아야 할 것들신뢰할 수 있는 A/B 테스트를 위해 알아야 할 것들
신뢰할 수 있는 A/B 테스트를 위해 알아야 할 것들
 
Basics of AB testing in online products
Basics of AB testing in online productsBasics of AB testing in online products
Basics of AB testing in online products
 
The ways to fuck up ab testing (from data products meetup)
The ways to fuck up ab testing (from data products meetup)The ways to fuck up ab testing (from data products meetup)
The ways to fuck up ab testing (from data products meetup)
 
Can I Test More Than One Variable at a Time? Statisticians answer some of th...
Can I Test More Than One Variable at a Time? Statisticians answer some of th...Can I Test More Than One Variable at a Time? Statisticians answer some of th...
Can I Test More Than One Variable at a Time? Statisticians answer some of th...
 
Automated testing handbook
Automated testing handbookAutomated testing handbook
Automated testing handbook
 
SAMPLE SIZE – The indispensable A/B test calculation that you’re not making
SAMPLE SIZE – The indispensable A/B test calculation that you’re not makingSAMPLE SIZE – The indispensable A/B test calculation that you’re not making
SAMPLE SIZE – The indispensable A/B test calculation that you’re not making
 
Ab testing 101
Ab testing 101Ab testing 101
Ab testing 101
 
Calculating a Sample Size
Calculating a Sample SizeCalculating a Sample Size
Calculating a Sample Size
 
SOFTWARE TESTING TRAFUNDAMENTALS OF SOFTWARE TESTING.pptx
SOFTWARE TESTING TRAFUNDAMENTALS OF SOFTWARE TESTING.pptxSOFTWARE TESTING TRAFUNDAMENTALS OF SOFTWARE TESTING.pptx
SOFTWARE TESTING TRAFUNDAMENTALS OF SOFTWARE TESTING.pptx
 
Data-Driven UI/UX Design with A/B Testing
Data-Driven UI/UX Design with A/B TestingData-Driven UI/UX Design with A/B Testing
Data-Driven UI/UX Design with A/B Testing
 
A/B Testing myths and quagmires
A/B Testing myths and quagmiresA/B Testing myths and quagmires
A/B Testing myths and quagmires
 
Critical Checks for Pharmaceuticals and Healthcare: Validating Your Data Inte...
Critical Checks for Pharmaceuticals and Healthcare: Validating Your Data Inte...Critical Checks for Pharmaceuticals and Healthcare: Validating Your Data Inte...
Critical Checks for Pharmaceuticals and Healthcare: Validating Your Data Inte...
 
Download Invesp’s The Essentials of Multivariate & AB Testing
Download Invesp’s The Essentials of Multivariate & AB TestingDownload Invesp’s The Essentials of Multivariate & AB Testing
Download Invesp’s The Essentials of Multivariate & AB Testing
 
Why learn Six Sigma, 4,28,15
Why learn Six Sigma, 4,28,15Why learn Six Sigma, 4,28,15
Why learn Six Sigma, 4,28,15
 
6 Guidelines for A/B Testing
6 Guidelines for A/B Testing6 Guidelines for A/B Testing
6 Guidelines for A/B Testing
 
Data-Driven Product Management by Shutterfly Director of Product
Data-Driven Product Management by Shutterfly Director of ProductData-Driven Product Management by Shutterfly Director of Product
Data-Driven Product Management by Shutterfly Director of Product
 

More from Optimizely

Clover Rings Up Digital Growth to Drive Experimentation
Clover Rings Up Digital Growth to Drive ExperimentationClover Rings Up Digital Growth to Drive Experimentation
Clover Rings Up Digital Growth to Drive Experimentation
Optimizely
 
Make Every Touchpoint Count: How to Drive Revenue in an Increasingly Online W...
Make Every Touchpoint Count: How to Drive Revenue in an Increasingly Online W...Make Every Touchpoint Count: How to Drive Revenue in an Increasingly Online W...
Make Every Touchpoint Count: How to Drive Revenue in an Increasingly Online W...
Optimizely
 
Atlassian's Mystique CLI, Minimizing the Experiment Development Cycle
Atlassian's Mystique CLI, Minimizing the Experiment Development CycleAtlassian's Mystique CLI, Minimizing the Experiment Development Cycle
Atlassian's Mystique CLI, Minimizing the Experiment Development Cycle
Optimizely
 
Autotrader Case Study: Migrating from Home-Grown Testing to Best-in-Class Too...
Autotrader Case Study: Migrating from Home-Grown Testing to Best-in-Class Too...Autotrader Case Study: Migrating from Home-Grown Testing to Best-in-Class Too...
Autotrader Case Study: Migrating from Home-Grown Testing to Best-in-Class Too...
Optimizely
 
Zillow + Optimizely: Building the Bridge to $20 Billion Revenue
Zillow + Optimizely: Building the Bridge to $20 Billion RevenueZillow + Optimizely: Building the Bridge to $20 Billion Revenue
Zillow + Optimizely: Building the Bridge to $20 Billion Revenue
Optimizely
 
The Future of Optimizely for Technical Teams
The Future of Optimizely for Technical TeamsThe Future of Optimizely for Technical Teams
The Future of Optimizely for Technical Teams
Optimizely
 
Empowering Agents to Provide Service from Anywhere: Contact Centers in the Ti...
Empowering Agents to Provide Service from Anywhere: Contact Centers in the Ti...Empowering Agents to Provide Service from Anywhere: Contact Centers in the Ti...
Empowering Agents to Provide Service from Anywhere: Contact Centers in the Ti...
Optimizely
 
Experimentation Everywhere: Create Exceptional Online Shopping Experiences an...
Experimentation Everywhere: Create Exceptional Online Shopping Experiences an...Experimentation Everywhere: Create Exceptional Online Shopping Experiences an...
Experimentation Everywhere: Create Exceptional Online Shopping Experiences an...
Optimizely
 
Building an Experiment Pipeline for GitHub’s New Free Team Offering
Building an Experiment Pipeline for GitHub’s New Free Team OfferingBuilding an Experiment Pipeline for GitHub’s New Free Team Offering
Building an Experiment Pipeline for GitHub’s New Free Team Offering
Optimizely
 
AMC Networks Experiments Faster on the Server Side
AMC Networks Experiments Faster on the Server SideAMC Networks Experiments Faster on the Server Side
AMC Networks Experiments Faster on the Server Side
Optimizely
 
Evolving Experimentation from CRO to Product Development
Evolving Experimentation from CRO to Product DevelopmentEvolving Experimentation from CRO to Product Development
Evolving Experimentation from CRO to Product Development
Optimizely
 
Overcoming the Challenges of Experimentation on a Service Oriented Architecture
Overcoming the Challenges of Experimentation on a Service Oriented ArchitectureOvercoming the Challenges of Experimentation on a Service Oriented Architecture
Overcoming the Challenges of Experimentation on a Service Oriented Architecture
Optimizely
 
How The Zebra Utilized Feature Experiments To Increase Carrier Card Engagemen...
How The Zebra Utilized Feature Experiments To Increase Carrier Card Engagemen...How The Zebra Utilized Feature Experiments To Increase Carrier Card Engagemen...
How The Zebra Utilized Feature Experiments To Increase Carrier Card Engagemen...
Optimizely
 
Kick Your Assumptions: How Scholl's Test-Everything Culture Drives Revenue
Kick Your Assumptions: How Scholl's Test-Everything Culture Drives RevenueKick Your Assumptions: How Scholl's Test-Everything Culture Drives Revenue
Kick Your Assumptions: How Scholl's Test-Everything Culture Drives Revenue
Optimizely
 
Experimentation through Clients' Eyes
Experimentation through Clients' EyesExperimentation through Clients' Eyes
Experimentation through Clients' Eyes
Optimizely
 
Shipping to Learn and Accelerate Growth with GitHub
Shipping to Learn and Accelerate Growth with GitHubShipping to Learn and Accelerate Growth with GitHub
Shipping to Learn and Accelerate Growth with GitHub
Optimizely
 
Test Everything: TrustRadius Delivers Customer Value with Experimentation
Test Everything: TrustRadius Delivers Customer Value with ExperimentationTest Everything: TrustRadius Delivers Customer Value with Experimentation
Test Everything: TrustRadius Delivers Customer Value with Experimentation
Optimizely
 
Optimizely Agent: Scaling Resilient Feature Delivery
Optimizely Agent: Scaling Resilient Feature DeliveryOptimizely Agent: Scaling Resilient Feature Delivery
Optimizely Agent: Scaling Resilient Feature Delivery
Optimizely
 
The Future of Software Development
The Future of Software DevelopmentThe Future of Software Development
The Future of Software Development
Optimizely
 
Practical Use Case: How Dosh Uses Feature Experiments To Accelerate Mobile De...
Practical Use Case: How Dosh Uses Feature Experiments To Accelerate Mobile De...Practical Use Case: How Dosh Uses Feature Experiments To Accelerate Mobile De...
Practical Use Case: How Dosh Uses Feature Experiments To Accelerate Mobile De...
Optimizely
 

More from Optimizely (20)

Clover Rings Up Digital Growth to Drive Experimentation
Clover Rings Up Digital Growth to Drive ExperimentationClover Rings Up Digital Growth to Drive Experimentation
Clover Rings Up Digital Growth to Drive Experimentation
 
Make Every Touchpoint Count: How to Drive Revenue in an Increasingly Online W...
Make Every Touchpoint Count: How to Drive Revenue in an Increasingly Online W...Make Every Touchpoint Count: How to Drive Revenue in an Increasingly Online W...
Make Every Touchpoint Count: How to Drive Revenue in an Increasingly Online W...
 
Atlassian's Mystique CLI, Minimizing the Experiment Development Cycle
Atlassian's Mystique CLI, Minimizing the Experiment Development CycleAtlassian's Mystique CLI, Minimizing the Experiment Development Cycle
Atlassian's Mystique CLI, Minimizing the Experiment Development Cycle
 
Autotrader Case Study: Migrating from Home-Grown Testing to Best-in-Class Too...
Autotrader Case Study: Migrating from Home-Grown Testing to Best-in-Class Too...Autotrader Case Study: Migrating from Home-Grown Testing to Best-in-Class Too...
Autotrader Case Study: Migrating from Home-Grown Testing to Best-in-Class Too...
 
Zillow + Optimizely: Building the Bridge to $20 Billion Revenue
Zillow + Optimizely: Building the Bridge to $20 Billion RevenueZillow + Optimizely: Building the Bridge to $20 Billion Revenue
Zillow + Optimizely: Building the Bridge to $20 Billion Revenue
 
The Future of Optimizely for Technical Teams
The Future of Optimizely for Technical TeamsThe Future of Optimizely for Technical Teams
The Future of Optimizely for Technical Teams
 
Empowering Agents to Provide Service from Anywhere: Contact Centers in the Ti...
Empowering Agents to Provide Service from Anywhere: Contact Centers in the Ti...Empowering Agents to Provide Service from Anywhere: Contact Centers in the Ti...
Empowering Agents to Provide Service from Anywhere: Contact Centers in the Ti...
 
Experimentation Everywhere: Create Exceptional Online Shopping Experiences an...
Experimentation Everywhere: Create Exceptional Online Shopping Experiences an...Experimentation Everywhere: Create Exceptional Online Shopping Experiences an...
Experimentation Everywhere: Create Exceptional Online Shopping Experiences an...
 
Building an Experiment Pipeline for GitHub’s New Free Team Offering
Building an Experiment Pipeline for GitHub’s New Free Team OfferingBuilding an Experiment Pipeline for GitHub’s New Free Team Offering
Building an Experiment Pipeline for GitHub’s New Free Team Offering
 
AMC Networks Experiments Faster on the Server Side
AMC Networks Experiments Faster on the Server SideAMC Networks Experiments Faster on the Server Side
AMC Networks Experiments Faster on the Server Side
 
Evolving Experimentation from CRO to Product Development
Evolving Experimentation from CRO to Product DevelopmentEvolving Experimentation from CRO to Product Development
Evolving Experimentation from CRO to Product Development
 
Overcoming the Challenges of Experimentation on a Service Oriented Architecture
Overcoming the Challenges of Experimentation on a Service Oriented ArchitectureOvercoming the Challenges of Experimentation on a Service Oriented Architecture
Overcoming the Challenges of Experimentation on a Service Oriented Architecture
 
How The Zebra Utilized Feature Experiments To Increase Carrier Card Engagemen...
How The Zebra Utilized Feature Experiments To Increase Carrier Card Engagemen...How The Zebra Utilized Feature Experiments To Increase Carrier Card Engagemen...
How The Zebra Utilized Feature Experiments To Increase Carrier Card Engagemen...
 
Kick Your Assumptions: How Scholl's Test-Everything Culture Drives Revenue
Kick Your Assumptions: How Scholl's Test-Everything Culture Drives RevenueKick Your Assumptions: How Scholl's Test-Everything Culture Drives Revenue
Kick Your Assumptions: How Scholl's Test-Everything Culture Drives Revenue
 
Experimentation through Clients' Eyes
Experimentation through Clients' EyesExperimentation through Clients' Eyes
Experimentation through Clients' Eyes
 
Shipping to Learn and Accelerate Growth with GitHub
Shipping to Learn and Accelerate Growth with GitHubShipping to Learn and Accelerate Growth with GitHub
Shipping to Learn and Accelerate Growth with GitHub
 
Test Everything: TrustRadius Delivers Customer Value with Experimentation
Test Everything: TrustRadius Delivers Customer Value with ExperimentationTest Everything: TrustRadius Delivers Customer Value with Experimentation
Test Everything: TrustRadius Delivers Customer Value with Experimentation
 
Optimizely Agent: Scaling Resilient Feature Delivery
Optimizely Agent: Scaling Resilient Feature DeliveryOptimizely Agent: Scaling Resilient Feature Delivery
Optimizely Agent: Scaling Resilient Feature Delivery
 
The Future of Software Development
The Future of Software DevelopmentThe Future of Software Development
The Future of Software Development
 
Practical Use Case: How Dosh Uses Feature Experiments To Accelerate Mobile De...
Practical Use Case: How Dosh Uses Feature Experiments To Accelerate Mobile De...Practical Use Case: How Dosh Uses Feature Experiments To Accelerate Mobile De...
Practical Use Case: How Dosh Uses Feature Experiments To Accelerate Mobile De...
 

Recently uploaded

一比一原版(YU毕业证)约克大学毕业证成绩单
一比一原版(YU毕业证)约克大学毕业证成绩单一比一原版(YU毕业证)约克大学毕业证成绩单
一比一原版(YU毕业证)约克大学毕业证成绩单
enxupq
 
Best best suvichar in gujarati english meaning of this sentence as Silk road ...
Best best suvichar in gujarati english meaning of this sentence as Silk road ...Best best suvichar in gujarati english meaning of this sentence as Silk road ...
Best best suvichar in gujarati english meaning of this sentence as Silk road ...
AbhimanyuSinha9
 
一比一原版(ArtEZ毕业证)ArtEZ艺术学院毕业证成绩单
一比一原版(ArtEZ毕业证)ArtEZ艺术学院毕业证成绩单一比一原版(ArtEZ毕业证)ArtEZ艺术学院毕业证成绩单
一比一原版(ArtEZ毕业证)ArtEZ艺术学院毕业证成绩单
vcaxypu
 
FP Growth Algorithm and its Applications
FP Growth Algorithm and its ApplicationsFP Growth Algorithm and its Applications
FP Growth Algorithm and its Applications
MaleehaSheikh2
 
Business update Q1 2024 Lar España Real Estate SOCIMI
Business update Q1 2024 Lar España Real Estate SOCIMIBusiness update Q1 2024 Lar España Real Estate SOCIMI
Business update Q1 2024 Lar España Real Estate SOCIMI
AlejandraGmez176757
 
The affect of service quality and online reviews on customer loyalty in the E...
The affect of service quality and online reviews on customer loyalty in the E...The affect of service quality and online reviews on customer loyalty in the E...
The affect of service quality and online reviews on customer loyalty in the E...
jerlynmaetalle
 
Innovative Methods in Media and Communication Research by Sebastian Kubitschk...
Innovative Methods in Media and Communication Research by Sebastian Kubitschk...Innovative Methods in Media and Communication Research by Sebastian Kubitschk...
Innovative Methods in Media and Communication Research by Sebastian Kubitschk...
correoyaya
 
一比一原版(NYU毕业证)纽约大学毕业证成绩单
一比一原版(NYU毕业证)纽约大学毕业证成绩单一比一原版(NYU毕业证)纽约大学毕业证成绩单
一比一原版(NYU毕业证)纽约大学毕业证成绩单
ewymefz
 
Q1’2024 Update: MYCI’s Leap Year Rebound
Q1’2024 Update: MYCI’s Leap Year ReboundQ1’2024 Update: MYCI’s Leap Year Rebound
Q1’2024 Update: MYCI’s Leap Year Rebound
Oppotus
 
Algorithmic optimizations for Dynamic Levelwise PageRank (from STICD) : SHORT...
Algorithmic optimizations for Dynamic Levelwise PageRank (from STICD) : SHORT...Algorithmic optimizations for Dynamic Levelwise PageRank (from STICD) : SHORT...
Algorithmic optimizations for Dynamic Levelwise PageRank (from STICD) : SHORT...
Subhajit Sahu
 
一比一原版(CBU毕业证)卡普顿大学毕业证成绩单
一比一原版(CBU毕业证)卡普顿大学毕业证成绩单一比一原版(CBU毕业证)卡普顿大学毕业证成绩单
一比一原版(CBU毕业证)卡普顿大学毕业证成绩单
nscud
 
一比一原版(RUG毕业证)格罗宁根大学毕业证成绩单
一比一原版(RUG毕业证)格罗宁根大学毕业证成绩单一比一原版(RUG毕业证)格罗宁根大学毕业证成绩单
一比一原版(RUG毕业证)格罗宁根大学毕业证成绩单
vcaxypu
 
一比一原版(IIT毕业证)伊利诺伊理工大学毕业证成绩单
一比一原版(IIT毕业证)伊利诺伊理工大学毕业证成绩单一比一原版(IIT毕业证)伊利诺伊理工大学毕业证成绩单
一比一原版(IIT毕业证)伊利诺伊理工大学毕业证成绩单
ewymefz
 
Jpolillo Amazon PPC - Bid Optimization Sample
Jpolillo Amazon PPC - Bid Optimization SampleJpolillo Amazon PPC - Bid Optimization Sample
Jpolillo Amazon PPC - Bid Optimization Sample
James Polillo
 
Chatty Kathy - UNC Bootcamp Final Project Presentation - Final Version - 5.23...
Chatty Kathy - UNC Bootcamp Final Project Presentation - Final Version - 5.23...Chatty Kathy - UNC Bootcamp Final Project Presentation - Final Version - 5.23...
Chatty Kathy - UNC Bootcamp Final Project Presentation - Final Version - 5.23...
John Andrews
 
Criminal IP - Threat Hunting Webinar.pdf
Criminal IP - Threat Hunting Webinar.pdfCriminal IP - Threat Hunting Webinar.pdf
Criminal IP - Threat Hunting Webinar.pdf
Criminal IP
 
一比一原版(CBU毕业证)不列颠海角大学毕业证成绩单
一比一原版(CBU毕业证)不列颠海角大学毕业证成绩单一比一原版(CBU毕业证)不列颠海角大学毕业证成绩单
一比一原版(CBU毕业证)不列颠海角大学毕业证成绩单
nscud
 
一比一原版(TWU毕业证)西三一大学毕业证成绩单
一比一原版(TWU毕业证)西三一大学毕业证成绩单一比一原版(TWU毕业证)西三一大学毕业证成绩单
一比一原版(TWU毕业证)西三一大学毕业证成绩单
ocavb
 
社内勉強会資料_LLM Agents　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　.
社内勉強会資料_LLM Agents　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　.社内勉強会資料_LLM Agents　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　.
社内勉強会資料_LLM Agents　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　.
NABLAS株式会社
 
一比一原版(BU毕业证)波士顿大学毕业证成绩单
一比一原版(BU毕业证)波士顿大学毕业证成绩单一比一原版(BU毕业证)波士顿大学毕业证成绩单
一比一原版(BU毕业证)波士顿大学毕业证成绩单
ewymefz
 

Recently uploaded (20)

一比一原版(YU毕业证)约克大学毕业证成绩单
一比一原版(YU毕业证)约克大学毕业证成绩单一比一原版(YU毕业证)约克大学毕业证成绩单
一比一原版(YU毕业证)约克大学毕业证成绩单
 
Best best suvichar in gujarati english meaning of this sentence as Silk road ...
Best best suvichar in gujarati english meaning of this sentence as Silk road ...Best best suvichar in gujarati english meaning of this sentence as Silk road ...
Best best suvichar in gujarati english meaning of this sentence as Silk road ...
 
一比一原版(ArtEZ毕业证)ArtEZ艺术学院毕业证成绩单
一比一原版(ArtEZ毕业证)ArtEZ艺术学院毕业证成绩单一比一原版(ArtEZ毕业证)ArtEZ艺术学院毕业证成绩单
一比一原版(ArtEZ毕业证)ArtEZ艺术学院毕业证成绩单
 
FP Growth Algorithm and its Applications
FP Growth Algorithm and its ApplicationsFP Growth Algorithm and its Applications
FP Growth Algorithm and its Applications
 
Business update Q1 2024 Lar España Real Estate SOCIMI
Business update Q1 2024 Lar España Real Estate SOCIMIBusiness update Q1 2024 Lar España Real Estate SOCIMI
Business update Q1 2024 Lar España Real Estate SOCIMI
 
The affect of service quality and online reviews on customer loyalty in the E...
The affect of service quality and online reviews on customer loyalty in the E...The affect of service quality and online reviews on customer loyalty in the E...
The affect of service quality and online reviews on customer loyalty in the E...
 
Innovative Methods in Media and Communication Research by Sebastian Kubitschk...
Innovative Methods in Media and Communication Research by Sebastian Kubitschk...Innovative Methods in Media and Communication Research by Sebastian Kubitschk...
Innovative Methods in Media and Communication Research by Sebastian Kubitschk...
 
一比一原版(NYU毕业证)纽约大学毕业证成绩单
一比一原版(NYU毕业证)纽约大学毕业证成绩单一比一原版(NYU毕业证)纽约大学毕业证成绩单
一比一原版(NYU毕业证)纽约大学毕业证成绩单
 
Q1’2024 Update: MYCI’s Leap Year Rebound
Q1’2024 Update: MYCI’s Leap Year ReboundQ1’2024 Update: MYCI’s Leap Year Rebound
Q1’2024 Update: MYCI’s Leap Year Rebound
 
Algorithmic optimizations for Dynamic Levelwise PageRank (from STICD) : SHORT...
Algorithmic optimizations for Dynamic Levelwise PageRank (from STICD) : SHORT...Algorithmic optimizations for Dynamic Levelwise PageRank (from STICD) : SHORT...
Algorithmic optimizations for Dynamic Levelwise PageRank (from STICD) : SHORT...
 
一比一原版(CBU毕业证)卡普顿大学毕业证成绩单
一比一原版(CBU毕业证)卡普顿大学毕业证成绩单一比一原版(CBU毕业证)卡普顿大学毕业证成绩单
一比一原版(CBU毕业证)卡普顿大学毕业证成绩单
 
一比一原版(RUG毕业证)格罗宁根大学毕业证成绩单
一比一原版(RUG毕业证)格罗宁根大学毕业证成绩单一比一原版(RUG毕业证)格罗宁根大学毕业证成绩单
一比一原版(RUG毕业证)格罗宁根大学毕业证成绩单
 
一比一原版(IIT毕业证)伊利诺伊理工大学毕业证成绩单
一比一原版(IIT毕业证)伊利诺伊理工大学毕业证成绩单一比一原版(IIT毕业证)伊利诺伊理工大学毕业证成绩单
一比一原版(IIT毕业证)伊利诺伊理工大学毕业证成绩单
 
Jpolillo Amazon PPC - Bid Optimization Sample
Jpolillo Amazon PPC - Bid Optimization SampleJpolillo Amazon PPC - Bid Optimization Sample
Jpolillo Amazon PPC - Bid Optimization Sample
 
Chatty Kathy - UNC Bootcamp Final Project Presentation - Final Version - 5.23...
Chatty Kathy - UNC Bootcamp Final Project Presentation - Final Version - 5.23...Chatty Kathy - UNC Bootcamp Final Project Presentation - Final Version - 5.23...
Chatty Kathy - UNC Bootcamp Final Project Presentation - Final Version - 5.23...
 
Criminal IP - Threat Hunting Webinar.pdf
Criminal IP - Threat Hunting Webinar.pdfCriminal IP - Threat Hunting Webinar.pdf
Criminal IP - Threat Hunting Webinar.pdf
 
一比一原版(CBU毕业证)不列颠海角大学毕业证成绩单
一比一原版(CBU毕业证)不列颠海角大学毕业证成绩单一比一原版(CBU毕业证)不列颠海角大学毕业证成绩单
一比一原版(CBU毕业证)不列颠海角大学毕业证成绩单
 
一比一原版(TWU毕业证)西三一大学毕业证成绩单
一比一原版(TWU毕业证)西三一大学毕业证成绩单一比一原版(TWU毕业证)西三一大学毕业证成绩单
一比一原版(TWU毕业证)西三一大学毕业证成绩单
 
社内勉強会資料_LLM Agents　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　.
社内勉強会資料_LLM Agents　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　.社内勉強会資料_LLM Agents　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　.
社内勉強会資料_LLM Agents　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　.
 
一比一原版(BU毕业证)波士顿大学毕业证成绩单
一比一原版(BU毕业证)波士顿大学毕业证成绩单一比一原版(BU毕业证)波士顿大学毕业证成绩单
一比一原版(BU毕业证)波士顿大学毕业证成绩单
 

Optimizely Workshop: Take Action on Results with Statistics

  • 1. Take Action on Results With Statisitcs An Optimizely Online Workshop Statistician: Leonid Pekelis
  • 2. Optimizely’s Stats Engine is designed to work with you, not against you, to provide results which are reliable and accurate, without requiring statistical training. At the same time, by knowing some statistics of your own, you can tune Stats Engine to get the most performance for your unique needs.
  • 3. 1. Which two A/B Testing pitfalls inflate error rates when using classical statistics, and are avoided with Stats Engine? 2. What are the three tradeoffs in an A/B Test? And how are they related? 3. How can you use Optimizely’s results page to best tune the tradeoffs to achieve your experimentation goals? After this workshop, you should be able to answer…
  • 4. How to choose the number of goals and variations for your experiment. We will also preview
  • 6. • A) The original, or baseline version of content that you are testing through a variation. • B) Metric used to measure impact of control and variation • C) The control group’s expected conversion rate. • D) The relative percentage difference of your variation from baseline. • E) The number of visitors in your test. Which is the Improvement?
  • 7. • A) Control and Variation The original, or baseline version of content that you are testing through a variation. • B) Goal Metric used to measure impact of control and variation • C) Baseline conversion rate The control group’s expected conversion rate. • D) Improvement The relative percentage difference of your variation from baseline. • E) Sample size The number of visitors in your test.
  • 8. Stats Engine corrects the pitfalls of A/B Testing with classical statistics.
  • 9. A procedure for classical statistics (a.k.a. “T-test”, a.k.a. “Traditional Frequentist”, a.k.a “Fixed Horizon Testing” ) Farmer Fred wants to compare the effect of two fertilizers on crop yield. 1. Chooses how many plots to use (sample size). 2. Waits for a crop cycle, collects data once at the end. 3. Asks “What are the chances I’d have gotten these results if there was no difference between the fertilizers?” (a.k.a. p-value) If p-value < 5%, his results are significant. 4. Goes on, maybe to test irrigation methods.
  • 10. 1915 Data is expensive. Data is slow. Practitioners are trained. 2015 Data is cheap. Data is real-time. Practitioners are everyone. Classical statistics were designed for an offline world.
  • 11. The modern A/B Testing procedure is different 1. Start without good estimate of sample size. 2. Check results early and often. Estimate ROI as quickly as possible. 3. Ask “How likely did my testing procedure give a wrong answer?” 4. Many variations on multiple goals, not just 1. 5. Iterate. Iterate. Iterate.
  • 13. p-Value < 5%. Significant! p-Value > 5%. Inconclusive. p-Value > 5%. Inconclusive. Min Sample Size Peeking Time Experiment Starts p-Value > 5%. Inconclusive.
  • 14. Why is this a problem? There is a ~5% chance of false positive each time you peek.
  • 15. p-Value < 5%. Significant! p-Value > 5%. Inconclusive. p-Value > 5%. Inconclusive. Min Sample Size Peeking Time Experiment Starts p-Value > 5%. Inconclusive. 4 peeks —> ~18% chance of seeing a false positive
  • 16. Pitfall 2. Mistaking “False Positive Rate” for “Chance of a wrong conclusion”
  • 17. Say I run an experiment.
  • 18. 1 original page, 5 variations, 6 goals = 30 “A/B Tests”
  • 19. After I reach my minimum sample size, I stop the experiment and see 2 of my variations beating control and 1 variation losing to control
  • 20. Winner Winner Loser Classical statistics guarantee <= 5% false positives. What % of my 2 winners and 1 loser do I expect to be false positives?
  • 23. Winner Winner Loser Classical statistics guarantee <= 5% false positives. What % of my winners & losers do I expect to be false positives? Answer: C) With 30 A/B Tests, we can expect a = 50% chance of a wrong conclusion! In general, we can’t say without knowing how many other goals & variations were tested. 1.5 3
  • 24. 1. Which two A/B Testing pitfalls inflate error rates when using classical statistics, and are avoided with Stats Engine? 2. What are the three tradeoffs in an A/B Test? And how are they related? 3. How can you use Optimizely’s results page to best tune the tradeoffs to achieve your experimentation goals? After this workshop you should be able to answer …
  • 25. 1. Which two A/B Testing pitfalls inflate error rates when using classical statistics, and are avoided with Stats Engine? A. Peeking and mistaking “False Positive Rate” for “Chance of a wrong conclusion.” After this webinar, you should be able to answer …
  • 26. The tradeoffs of A/B Testing
  • 28. Error rates Runtime Improvement & Baseline CR “Chance of a wrong conclusion”
  • 29. Error rates Runtime Improvement & Baseline CR “Chance of a wrong conclusion calling a non- winner a winner, or a non- loser a loser.”
  • 31. Where is the error rate on Optimizely’s results page? I. II. III. IV. Statistical Significance = “Chance of a right conclusion” = (a.k.a.) 100 x (1 - False Discovery Rate)
  • 32. How can you control the error rate?
  • 34. Where is runtime on Optimizely’s results page?
  • 35. Error rates Runtime Were you expecting a funny picture? Improvement & Baseline CR
  • 36. Where is effect size on Optimizely’s results page?
  • 37. Improvement & Baseline CR These three quantities are all … Error rates Runtime Inversely Related
  • 38. At any number of visitors, the higher error rate I allow, the smaller improvement you can detect. Error rates Runtime Inversely Related Improvement & Baseline CR
  • 39. Error rates Runtime Inversely Related At any error rate threshold, stopping your test earlier means you can only detect larger improvements. Improvement & Baseline CR
  • 40. For any improvement, the lower error rate you want, the longer you need to run your test. Error rates Runtime Inversely Related Improvement & Baseline CR
  • 41. What does this look like in practice? Average Visitors needed to reach significance with Stats Engine Improvement (relative) 5% 10% 25% Significance Threshold (Error Rate) 95 (5%) 62 K 14 K 1,800 90 (10%) 59 K 12 K 1,700 80 (20%) 53 K 11 K 1,500 Baseline conversion rate = 10%
  • 42. ~ 1 K visitors per day Average Visitors needed to reach significance with Stats Engine Improvement (relative) 5% 10% 25% Significance Threshold (Error Rate) 95 (5%) 62 K 14 K 1,800 90 (10%) 59 K 12 K 1,700 80 (20%) 53 K 11 K 1,500 (1 day) Baseline conversion rate = 10%
  • 43. ~ 10K visitors per day Average Visitors needed to reach significance with Stats Engine Improvement (relative) 5% 10% 25% Significance Threshold (Error Rate) 95 (5%) 62 K 14 K 1,800 90 (10%) 59 K 12 K 1,700 80 (20%) 53 K 11 K (1 day) 1,500 Baseline conversion rate = 10%
  • 44. ~ 50K visitors per day Average Visitors needed to reach significance with Stats Engine Improvement (relative) 3% 5% 10% Significance Threshold (Error Rate) 95 (5%) 190 K 62 K 14 K 90 (10%) 180 K 59 K 12 K 80 (20%) 160 K 53 K (1 day) 11 K Baseline conversion rate = 10%
  • 45. > 100K visitors per day Average Visitors needed to reach significance with Stats Engine Improvement (relative) 3% 5% 10% Significance Threshold (Error Rate) 95 (5%) 190 K 62 K 14 K 90 (10%) 180 K 59 K 12 K 80 (20%) 160 K (1 day) 53 K 11 K Baseline conversion rate = 10%
  • 46. 1. Which two A/B Testing pitfalls inflate error rates when using classical statistics, and are avoided with Stats Engine? 2. What are the three tradeoffs in an A/B Test? And how are they related? 3. How can you use Optimizely’s results page to best tune the tradeoffs to achieve your experimentation goals? After this workshop, you should be able to answer …
  • 47. 1. Which two A/B Testing pitfalls inflate error rates when using classical statistics, and are avoided with Stats Engine? 2. What are the three tradeoffs in an A/B Test? And how are they related? A. Error Rates, Runtime, and Effect Size. They are all inversely related. After this workshop, you should be able to answer …
  • 48. Use tradeoffs to align your testing goals
  • 49. 5% Error rates Runtime Improvement & Baseline CR Inversely Related +5%, 10% 53 K ? In the beginning, we make an educated guess …
  • 50. … but after 1 day … Data! How can we update the tradeoffs?
  • 51. 1. Adjust your timeline
  • 52. Improvement turns out to be better … Instead of: 53K - 10K = 43K 5% 1,600 Error rates Runtime +13%, 10% Inversely Related Improvement & Baseline CR
  • 53. … or worse. 5% 75 K Error rates Runtime +2%, 8% Inversely Related Improvement & Baseline CR
  • 54. 2. Accept higher / lower error rate
  • 55. Improvement turns out to be better … 1% 43 K Error rates Runtime +13%, 10% Inversely Related Improvement & Baseline CR
  • 56. … or worse. 30% 43 K Error rates Runtime +2%, 8% Inversely Related Improvement & Baseline CR
  • 57. 3. Admit it. It’s inconclusive.
  • 58. … or a lot worse. > 99% > 100K Error rates Runtime +.2%, 8% Inversely Related Improvement & Baseline CR iterate, iterate, iterate!
  • 59. Your experiments will not always have the same improvement over time. So, run A/B Tests for at least a business cycle appropriate for that test and your company. Seasonality & Time Variation
  • 60. 1. Which two A/B Testing pitfalls inflate error rates when using classical statistics, and are avoided with Stats Engine? 2. What are the three tradeoffs in an A/B Test? And how are they related? 3. How can you use Optimizely’s results page to best tune the tradeoffs to achieve your experimentation goals? After this workshop, you should be able to answer …
  • 61. 1. Which two A/B Testing pitfalls inflate error rates when using classical statistics, and are avoided with for Stats Engine? 2. What are the three tradeoffs in one A/B Test? 3. How can you use Optimizely’s results page to best tune the tradeoffs to achieve your experimentation goals? A. Adjust your timeline. Accept higher / lower error rate. Admit an inconclusive result. After this workshop, you should be able to answer …
  • 62. 1. Which two A/B Testing pitfalls inflate error rates when using classical statistics, and are avoided with Stats Engine? A. Peeking and mistaking “False Positive Rate” for “Chance of a Wrong Answer.” 2. What are the three tradeoffs in one A/B Test? B. Error Rates, Runtime, and Effect Size. They are all negatively related. 3. How can you use Optimizely’s results page to best tune the tradeoffs to achieve your experimentation goals? C. Accept higher / lower error rate. Adjust your timeline. Admit an inconclusive result. 1. Which two A/B Testing pitfalls inflate error rates when using classical statistics, and are avoided with Stats Engine? A. Peeking and mistaking “False Positive Rate” for “Chance of a Wrong Answer.” 2. What are the three tradeoffs in one A/B Test? B. Error Rates, Runtime, and Effect Size. They are all negatively related. 3. How can you use Optimizely’s results page to best tune the tradeoffs to achieve your experimentation goals? C. Accept higher / lower error rate. Adjust your timeline. Admit an inconclusive result. 1. Which two A/B Testing pitfalls inflate error rates when using classical statistics, and are avoided with Stats Engine? A. Peeking and mistaking “False Positive Rate” for “Chance of a Wrong Answer.” 2. What are the three tradeoffs in one A/B Test? B. Error Rates, Runtime, and Effect Size. They are all negatively related. 3. How can you use Optimizely’s results page to best tune the tradeoffs to achieve your experimentation goals? C. Accept higher / lower error rate. Adjust your timeline. Admit an inconclusive result. 1. Which two A/B Testing pitfalls inflate error rates when using classical statistics, and are avoided with Stats Engine? A. Peeking and mistaking “False Positive Rate” for “Chance of a Wrong Answer.” 2. What are the three tradeoffs in one A/B Test? B. Error Rates, Runtime, and Effect Size. They are all negatively related. 3. How can you use Optimizely’s results page to best tune the tradeoffs to achieve your experimentation goals? C. Accept higher / lower error rate. Adjust your timeline. Admit an inconclusive result. Review
  • 63. Preview: How many goals and variations should I use?
  • 64. Stats Engine is more conservative when there are more goals that are not affected by a variation. So, adding a lot of “random” goals will slow down your experiment.
  • 65. Tips & Tricks for using Stats Engine with multiple goals and variations • Ask: Which goal is most important to me? -This should be the primary goal (not impacted by all other goals) • Run large, or large multivariate tests without fear of finding spurious results, but be prepared for the cost of exploration. • For maximum velocity, only test goals and variations that you believe will have highest impact.
  • 66.
  • 67. 1. Which two A/B Testing pitfalls inflate error rates when using classical statistics, and are avoided with Stats Engine? A. Peeking and mistaking “False Positive Rate” for “Chance of a Wrong Answer.” 2. What are the three tradeoffs in one A/B Test? B. Error Rates, Runtime, and Effect Size. They are all negatively related. 3. How can you use Optimizely’s results page to best tune the tradeoffs to achieve your experimentation goals? C. Accept higher / lower error rate. Adjust your timeline. Admit an inconclusive result. 1. Which two A/B Testing pitfalls inflate error rates when using classical statistics, and are avoided with Stats Engine? A. Peeking and mistaking “False Positive Rate” for “Chance of a Wrong Answer.” 2. What are the three tradeoffs in one A/B Test? B. Error Rates, Runtime, and Effect Size. They are all negatively related. 3. How can you use Optimizely’s results page to best tune the tradeoffs to achieve your experimentation goals? C. Accept higher / lower error rate. Adjust your timeline. Admit an inconclusive result. 1. Which two A/B Testing pitfalls inflate error rates when using classical statistics, and are avoided with Stats Engine? A. Peeking and mistaking “False Positive Rate” for “Chance of a Wrong Answer.” 2. What are the three tradeoffs in one A/B Test? B. Error Rates, Runtime, and Effect Size. They are all negatively related. 3. How can you use Optimizely’s results page to best tune the tradeoffs to achieve your experimentation goals? C. Accept higher / lower error rate. Adjust your timeline. Admit an inconclusive result. 1. Which two A/B Testing pitfalls inflate error rates when using classical statistics, and are avoided with Stats Engine? A. Peeking and mistaking “False Positive Rate” for “Chance of a Wrong Answer.” 2. What are the three tradeoffs in one A/B Test? B. Error Rates, Runtime, and Effect Size. They are all negatively related. 3. How can you use Optimizely’s results page to best tune the tradeoffs to achieve your experimentation goals? C. Accept higher / lower error rate. Adjust your timeline. Admit an inconclusive result. Review