Nikolay Novozhilov gave a presentation on common problems with A/B testing and statistics. He discussed how looking at data before testing can invalidate results and showed through a Monte Carlo simulation that variants can appear to "win" just by chance. Multivariate testing and multiple comparisons were also cited as issues. Novozhilov recommended starting with a clear hypothesis, replicating tests, and considering sample size, significance, effect size, and power to obtain more reliable results from A/B tests.
The problem
Not so much Winner! Not so much Not so much Not so much Not so much Not so much Not so much Not so much Not so much
The real problem
One month Winner! Two months Three months Four months Five months Six months Seven months Eight months Nine months
1 winner per year
QUOTE FROM ELON MUSK BIOGRAPHY “ He would place this urgency that he expected the revenue in ten years to be ten million dollars a day and that every day we were slower to achieve our goals was a day of missing out on that money.
We’re missing out on this
But…why?
TOO MANY IDEAS!
170 ideas in 3 minutes
Try new CTA Try new CTA location Test multiple CTAs Test different button colors Add an “About” section Try a different headline Add an image to the homepage Shorten my form Auto play the video Try modals for forms Test a free trial Try annual billing Try a high anchor price
Er.
We need a process
Find out what’s stopping folks from converting…
…fix it.
Thanks
I’ll be here all night
Step 1: Take these ideas…
Try new CTA Try new CTA location Test multiple CTAs Test different button colors Add an “About” section Try a different headline Add an image to the homepage Shorten my form Auto play the video Try modals for forms Test a free trial Try annual billing Try a high anchor price
…and throw them away.
Step 2: Listen to your customers and prospects
Try new CTA Try new CTA location Test multiple CTAs Test different button colors Add an “About” section Try a different headline Add an image to the homepage Shorten my form Auto play the video Try modals for forms Test a free trial Try annual billing Try a high anchor price These ideas are good. They just might not be for you.
Gut Your gut is good. It just knows too much.
Curse of knowledge
USER TESTING
Send 3 users through your funnel
Send 3 users to your competition
Send 3 users to a HUGE company in your space
PEEK
QUALAROO
CHAT TRANSCRIPTS
SALES PEOPLE AND SUPPORT
Step 3: Use a framework to make decisions
Test summary Test objective Hypothesis Opportunity size Time to test Likely scenarios Next steps What worked? What didn’t work? HTTP://BIT.LY/ABTEST-FRAMEWORK
The problem
Not so much Winner! Not so much Not so much Not so much Not so much Not so much Not so much Not so much Not so much
The problem opportunity
Not so much Winner! Not so much Not so much Not so much Not so much Not so much Not so much Not so much Not so much
Not so much Winner! Not so much Not so much Not so much Winner! Not so much Not so much Not so much Not so much
Not so much Not so much Not so much Winner! Not so much Not so much Winner! Not so much Not so much Winner!
To build a successful A/B testing strategy, you'll need more than just ideas of what to test, you'll need a plan that builds data into a repeatable strategy for producing winning experiments.
[Elite Camp 2016] Phil Nottingham - CRO with Video: Tips, Tricks and TacticsCXL
Everyone knows video can be an incredibly powerful conversion tool, but few know the secrets of running lean and mean CRO campaign with video. Armed with data from studying the performance of Wistia's 10 million videos, Phil will explain how to get the most of the videos across your channels and will explain how video analytics data can be used to automate and improve your creative processes.
The Truth Is Out There - User Research Based AB-TestingAGConsult
Stop guessing! Stop copying tests you've seen! Don't blindly follow best practices! Your website, your users, your products are unique. Use Google Analytics, surveys, mouse tracking, eyetracking etc. to find the gaps in your website and texts. This will enable you to create better ab-tests.
The problem
Not so much Winner! Not so much Not so much Not so much Not so much Not so much Not so much Not so much Not so much
The real problem
One month Winner! Two months Three months Four months Five months Six months Seven months Eight months Nine months
1 winner per year
QUOTE FROM ELON MUSK BIOGRAPHY “ He would place this urgency that he expected the revenue in ten years to be ten million dollars a day and that every day we were slower to achieve our goals was a day of missing out on that money.
We’re missing out on this
But…why?
TOO MANY IDEAS!
170 ideas in 3 minutes
Try new CTA Try new CTA location Test multiple CTAs Test different button colors Add an “About” section Try a different headline Add an image to the homepage Shorten my form Auto play the video Try modals for forms Test a free trial Try annual billing Try a high anchor price
Er.
We need a process
Find out what’s stopping folks from converting…
…fix it.
Thanks
I’ll be here all night
Step 1: Take these ideas…
Try new CTA Try new CTA location Test multiple CTAs Test different button colors Add an “About” section Try a different headline Add an image to the homepage Shorten my form Auto play the video Try modals for forms Test a free trial Try annual billing Try a high anchor price
…and throw them away.
Step 2: Listen to your customers and prospects
Try new CTA Try new CTA location Test multiple CTAs Test different button colors Add an “About” section Try a different headline Add an image to the homepage Shorten my form Auto play the video Try modals for forms Test a free trial Try annual billing Try a high anchor price These ideas are good. They just might not be for you.
Gut Your gut is good. It just knows too much.
Curse of knowledge
USER TESTING
Send 3 users through your funnel
Send 3 users to your competition
Send 3 users to a HUGE company in your space
PEEK
QUALAROO
CHAT TRANSCRIPTS
SALES PEOPLE AND SUPPORT
Step 3: Use a framework to make decisions
Test summary Test objective Hypothesis Opportunity size Time to test Likely scenarios Next steps What worked? What didn’t work? HTTP://BIT.LY/ABTEST-FRAMEWORK
The problem
Not so much Winner! Not so much Not so much Not so much Not so much Not so much Not so much Not so much Not so much
The problem opportunity
Not so much Winner! Not so much Not so much Not so much Not so much Not so much Not so much Not so much Not so much
Not so much Winner! Not so much Not so much Not so much Winner! Not so much Not so much Not so much Not so much
Not so much Not so much Not so much Winner! Not so much Not so much Winner! Not so much Not so much Winner!
To build a successful A/B testing strategy, you'll need more than just ideas of what to test, you'll need a plan that builds data into a repeatable strategy for producing winning experiments.
[Elite Camp 2016] Phil Nottingham - CRO with Video: Tips, Tricks and TacticsCXL
Everyone knows video can be an incredibly powerful conversion tool, but few know the secrets of running lean and mean CRO campaign with video. Armed with data from studying the performance of Wistia's 10 million videos, Phil will explain how to get the most of the videos across your channels and will explain how video analytics data can be used to automate and improve your creative processes.
The Truth Is Out There - User Research Based AB-TestingAGConsult
Stop guessing! Stop copying tests you've seen! Don't blindly follow best practices! Your website, your users, your products are unique. Use Google Analytics, surveys, mouse tracking, eyetracking etc. to find the gaps in your website and texts. This will enable you to create better ab-tests.
If you made A/B test, found significant result, implemented to production and see no real improvement - you just suffered from "imaginary uplift". In this presentation I explain why this is happening (and so often!) and what do to about it.
From my talk on Web Analytics Wednesdays Singapore (analyticsmaturity.com)
Post with source code for all experiments coming on www.novozhilov.co
Keynote Ton Wesseling at Superweek 2020: How an analyst can add value!Ton Wesseling
How an analyst can add value to your A/B-testing program.
What should be done with the A/B-test program?
A: Increase budgets (More a/b-tests (quantity))
B: Increase knowledge (Better a/b-tests (quality))
C: Decrease budgets (Less a/b-tests (quantity))
As an analyst:
You can calculate the answer
You have a big influence on the outcome
A/B testing, optimization and results analysis by Mariia Bocheva, ATD'18Mariia Bocheva
While working with data we usually face several problems: we don't have enough data, we have too much data, we don't know what to do with this data.
In this session, I'll show how to make sure you can rely on your data and share my favorite ideas on how you can use Google Analytics and other for A/B testing, optimization and analysis.
You’ll gain a better understanding on what to look at to answer your UX questions, how to run a test properly and evaluate the its results.
What's going to happen to SEM when there's no such thing as mobile anymore? Chronicling the rise of voice search, how PWA's impact app campaigns and what we'll have to do in the future.
The goal of analysis should provide leadership with insight into risk and uncertainty and guidance on actions that can be taken. However, common analysis methods of using point estimates to generate forward-looking business plans disregard uncertainty and ignore risk.
In this presentation, you will learn how to incorporate uncertainty directly into a decision support application. The results is a range estimate with likelihoods of exceeding thresholds based on assumption values, providing leadership with the insight into uncertainty and actions that can be taken to reduce risk.
SearchLove London 2018 - John Campbell - Voice Search – Calculating and Seizi...Distilled
Voice has and continues to be the hot topic of 2018, fuelled by the statistic that “50% of searches" will be conducted via voice by 2020. Voice is still an emerging marking as such there is very little data or tools which allow individuals to understand the opportunity in the market by vertical or key phrase.
John will be talking about the voice market landscape today, how you can calculate the opportunity at a key phrase level and practical steps to how to cease the opportunity ready for the supposedly “50% of searches” in 2020.
MeasureWorks - Emerce eFinancials - Content is King, but Experience is your k...MeasureWorks
Slides from presentation at Emerce efinancials event. During this session Jeroen Tjepkema, CEO & Founder of MeasureWorks, togehter with Matthew Niederberger, Digital Insights & Analytics Manager at Philips Lighting talk about why user experience and performance are essential in persuading online consumers to buy something on your website. Based on both real data and case studies Matthew and Jeroen will share practical information about the relationship between content and experience and how to implement these at your website.
How to do A/B in scale - Nature Intelligence styleKobi Salinas
Our slides from the previous meet up we had, talking about how we do A/B testing, what is a correct hypostasis and how did we scale up our testing using the bayesian method
Keynote Ton Wesseling at the Web Analytics Wednesday Copenhagen #wawcph at Se...Online Dialogue
Ton was asked to talk about things that get him excited as a web analyst looking at conversion rate optimization. He picked 5 things:
- The real fun part of web analytics is analyzing how user behavior is changing (analyzing experiments), not creating campaign reports...
- Win: inject your website feedback form responses into your analytics and be able to segment behavior based on goals.
- Run your experiments with an automated and free GTM / GA / EXCEL results set-up!
- To make sure business gets it: apply Bayesian statistics on experiment results, don't report on P values, confidence levels etc.
- Bandit algoritms: use www.smartnotifications.com for automated persuasive messaging on your website.
Brighton CRO Meetup #1 - Oh Boy These AB tests Sure Look Like Bullshit to MeCraig Sullivan
An updated deck of a short talk (30m) given at the first Brighton CRO meetup. Contains useful AB testing tools as well as full speaker notes for most of the slides.
If you made A/B test, found significant result, implemented to production and see no real improvement - you just suffered from "imaginary uplift". In this presentation I explain why this is happening (and so often!) and what do to about it.
From my talk on Web Analytics Wednesdays Singapore (analyticsmaturity.com)
Post with source code for all experiments coming on www.novozhilov.co
Keynote Ton Wesseling at Superweek 2020: How an analyst can add value!Ton Wesseling
How an analyst can add value to your A/B-testing program.
What should be done with the A/B-test program?
A: Increase budgets (More a/b-tests (quantity))
B: Increase knowledge (Better a/b-tests (quality))
C: Decrease budgets (Less a/b-tests (quantity))
As an analyst:
You can calculate the answer
You have a big influence on the outcome
A/B testing, optimization and results analysis by Mariia Bocheva, ATD'18Mariia Bocheva
While working with data we usually face several problems: we don't have enough data, we have too much data, we don't know what to do with this data.
In this session, I'll show how to make sure you can rely on your data and share my favorite ideas on how you can use Google Analytics and other for A/B testing, optimization and analysis.
You’ll gain a better understanding on what to look at to answer your UX questions, how to run a test properly and evaluate the its results.
What's going to happen to SEM when there's no such thing as mobile anymore? Chronicling the rise of voice search, how PWA's impact app campaigns and what we'll have to do in the future.
The goal of analysis should provide leadership with insight into risk and uncertainty and guidance on actions that can be taken. However, common analysis methods of using point estimates to generate forward-looking business plans disregard uncertainty and ignore risk.
In this presentation, you will learn how to incorporate uncertainty directly into a decision support application. The results is a range estimate with likelihoods of exceeding thresholds based on assumption values, providing leadership with the insight into uncertainty and actions that can be taken to reduce risk.
SearchLove London 2018 - John Campbell - Voice Search – Calculating and Seizi...Distilled
Voice has and continues to be the hot topic of 2018, fuelled by the statistic that “50% of searches" will be conducted via voice by 2020. Voice is still an emerging marking as such there is very little data or tools which allow individuals to understand the opportunity in the market by vertical or key phrase.
John will be talking about the voice market landscape today, how you can calculate the opportunity at a key phrase level and practical steps to how to cease the opportunity ready for the supposedly “50% of searches” in 2020.
MeasureWorks - Emerce eFinancials - Content is King, but Experience is your k...MeasureWorks
Slides from presentation at Emerce efinancials event. During this session Jeroen Tjepkema, CEO & Founder of MeasureWorks, togehter with Matthew Niederberger, Digital Insights & Analytics Manager at Philips Lighting talk about why user experience and performance are essential in persuading online consumers to buy something on your website. Based on both real data and case studies Matthew and Jeroen will share practical information about the relationship between content and experience and how to implement these at your website.
How to do A/B in scale - Nature Intelligence styleKobi Salinas
Our slides from the previous meet up we had, talking about how we do A/B testing, what is a correct hypostasis and how did we scale up our testing using the bayesian method
Keynote Ton Wesseling at the Web Analytics Wednesday Copenhagen #wawcph at Se...Online Dialogue
Ton was asked to talk about things that get him excited as a web analyst looking at conversion rate optimization. He picked 5 things:
- The real fun part of web analytics is analyzing how user behavior is changing (analyzing experiments), not creating campaign reports...
- Win: inject your website feedback form responses into your analytics and be able to segment behavior based on goals.
- Run your experiments with an automated and free GTM / GA / EXCEL results set-up!
- To make sure business gets it: apply Bayesian statistics on experiment results, don't report on P values, confidence levels etc.
- Bandit algoritms: use www.smartnotifications.com for automated persuasive messaging on your website.
Brighton CRO Meetup #1 - Oh Boy These AB tests Sure Look Like Bullshit to MeCraig Sullivan
An updated deck of a short talk (30m) given at the first Brighton CRO meetup. Contains useful AB testing tools as well as full speaker notes for most of the slides.
Techniques to optimize the pagerank algorithm usually fall in two categories. One is to try reducing the work per iteration, and the other is to try reducing the number of iterations. These goals are often at odds with one another. Skipping computation on vertices which have already converged has the potential to save iteration time. Skipping in-identical vertices, with the same in-links, helps reduce duplicate computations and thus could help reduce iteration time. Road networks often have chains which can be short-circuited before pagerank computation to improve performance. Final ranks of chain nodes can be easily calculated. This could reduce both the iteration time, and the number of iterations. If a graph has no dangling nodes, pagerank of each strongly connected component can be computed in topological order. This could help reduce the iteration time, no. of iterations, and also enable multi-iteration concurrency in pagerank computation. The combination of all of the above methods is the STICD algorithm. [sticd] For dynamic graphs, unchanged components whose ranks are unaffected can be skipped altogether.
Adjusting primitives for graph : SHORT REPORT / NOTESSubhajit Sahu
Graph algorithms, like PageRank Compressed Sparse Row (CSR) is an adjacency-list based graph representation that is
Multiply with different modes (map)
1. Performance of sequential execution based vs OpenMP based vector multiply.
2. Comparing various launch configs for CUDA based vector multiply.
Sum with different storage types (reduce)
1. Performance of vector element sum using float vs bfloat16 as the storage type.
Sum with different modes (reduce)
1. Performance of sequential execution based vs OpenMP based vector element sum.
2. Performance of memcpy vs in-place based CUDA based vector element sum.
3. Comparing various launch configs for CUDA based vector element sum (memcpy).
4. Comparing various launch configs for CUDA based vector element sum (in-place).
Sum with in-place strategies of CUDA mode (reduce)
1. Comparing various launch configs for CUDA based vector element sum (in-place).
Explore our comprehensive data analysis project presentation on predicting product ad campaign performance. Learn how data-driven insights can optimize your marketing strategies and enhance campaign effectiveness. Perfect for professionals and students looking to understand the power of data analysis in advertising. for more details visit: https://bostoninstituteofanalytics.org/data-science-and-artificial-intelligence/
Show drafts
volume_up
Empowering the Data Analytics Ecosystem: A Laser Focus on Value
The data analytics ecosystem thrives when every component functions at its peak, unlocking the true potential of data. Here's a laser focus on key areas for an empowered ecosystem:
1. Democratize Access, Not Data:
Granular Access Controls: Provide users with self-service tools tailored to their specific needs, preventing data overload and misuse.
Data Catalogs: Implement robust data catalogs for easy discovery and understanding of available data sources.
2. Foster Collaboration with Clear Roles:
Data Mesh Architecture: Break down data silos by creating a distributed data ownership model with clear ownership and responsibilities.
Collaborative Workspaces: Utilize interactive platforms where data scientists, analysts, and domain experts can work seamlessly together.
3. Leverage Advanced Analytics Strategically:
AI-powered Automation: Automate repetitive tasks like data cleaning and feature engineering, freeing up data talent for higher-level analysis.
Right-Tool Selection: Strategically choose the most effective advanced analytics techniques (e.g., AI, ML) based on specific business problems.
4. Prioritize Data Quality with Automation:
Automated Data Validation: Implement automated data quality checks to identify and rectify errors at the source, minimizing downstream issues.
Data Lineage Tracking: Track the flow of data throughout the ecosystem, ensuring transparency and facilitating root cause analysis for errors.
5. Cultivate a Data-Driven Mindset:
Metrics-Driven Performance Management: Align KPIs and performance metrics with data-driven insights to ensure actionable decision making.
Data Storytelling Workshops: Equip stakeholders with the skills to translate complex data findings into compelling narratives that drive action.
Benefits of a Precise Ecosystem:
Sharpened Focus: Precise access and clear roles ensure everyone works with the most relevant data, maximizing efficiency.
Actionable Insights: Strategic analytics and automated quality checks lead to more reliable and actionable data insights.
Continuous Improvement: Data-driven performance management fosters a culture of learning and continuous improvement.
Sustainable Growth: Empowered by data, organizations can make informed decisions to drive sustainable growth and innovation.
By focusing on these precise actions, organizations can create an empowered data analytics ecosystem that delivers real value by driving data-driven decisions and maximizing the return on their data investment.
2. A/B TESTING AND THE MOST COMMON
PROBLEMS, A LOOK AT STATISTICS AND
THE WAYS TO GET IT WRONG
Nikolay Novozhilov - Product director of data
platforms @ Wego.com.
Nikolay is building big data capabilities at Wego.com,
the Asia Pacific and the Middle East's leading travel
metasearch engine. He has 7+ years of experience in
data analytics working for IT startups and previously
for consulting. Nikolay received an MBA from INSEAD
in Singapore and before that lived and worked in
Moscow.
3. A/B testing and problems with statistics
Web Analytics Wednesday, Singapore
Nikolay Novozhilov, Wego.com
www.novozhilov.co
7. Lies, damned lies, and statistics
All different! All based on assumptions!!!
Tool Test used
Optimizely Two-tailed sequential likelihood ratio test
with false discovery rate controls
Google Analytics Bayes estimate with uniform beta prior
VWO Intersection of confidence intervals for
binominal distribution
Leanplum Confidence intervals at p=5%, unknown
statistic
Usereffect Chi-square statistics
Commerce Sciences Welch's t-test
8. What is p-value and why it is 5%?
All tests are
based on
assumptions!
Assumption #1:
You don’t look at
the data upfront
9. What happens if you look?
I played Monte Carlo in
Excel
And here is the result:
• 5% p-value
• 1000 “users” in each
sample
• CR of 2%
• A wins over A 29% of
the times!
10. What do you do about it?
Don’t look! (just kidding)
Google “O'Brien & Fleming interim
analysis” (no, still kidding )
Keep calm, more stuff coming!
