A/B Testing Ultimate Guideline. How to design and analyze digital testing.

1A/B Testing Ultimate Guide / GPeC Summit Bucharest / Ricardo Tayar - @rtayar / ricardo@flat101.com
A/B Testing Ultimate Guideline
How to think, design, release and analyze A/B testing
and learn about them
30/05/2018
GPeC Summit - Bucharest
Ricardo Tayar - @rtayar

Hello Bucharest!

¿What is A/B testing?

In web analytics, A/B testing (bucket tests or split-run testing) is a controlled
experiment with two variants, A and B.[1] [2] It is a form of statistical hypothesis
testing or "two-sample hypothesis testing" as used in the ﬁeld of statistics. A/B testing
is a way to compare two versions of a single variable typically by testing a subject's
response to variable A against variable B, and determining which of the two
variables is more effective.
Wikipedia

¿Why A/B testing
(and personalization)
is so needful in a
digital business?

A/B testing is the main tool we have to prove that something we have built a
hypothesis around (action, strategy, change) will improve our results in our digital
business, starting from data.
A/B testing allow us to validate with quantitative data how useful and needful a
changue could be in our context in a speciﬁc moment.
It helps us to avoid guessing or self-reference design. Both things are great problems
in digital design processes today.
It helps us in our constant learning about how can we built a stronger and more
efficient digital product.

Data put design in
business context

Goals

Goals
Revenue
Leads
Increasing %
Engagement

Data

Goals Data
Facturación
GA / Digital
Analytics
Leads Qualitative data
Incrementos % Logs
Engagement CRM / LTV
User Research

Hypothesis

Goals Data Hypothesis
Facturación
GA / Digital
Analytics
Brainstorming
Leads
Datos
cualitativos
5 why rule
Incrementos % Logs
Covariation
hypothesis
Correlation /
Cause-effect
User Research FUDs

A hypothesis should be a statement, a proposition that you make to describe what
will happen in a system given speciﬁc circumstances when we modify some
variables. It often follow this form: “If we do X, user will do Y with will impact metric
A”.
Hypotheses should be descriptive but short and they have to generate a
consequence in the system.
Example: “If we improve our information architecture user will ﬁnd easyly our
products, so our bounce rat will decrease and tire will be a bigger amount of users
starting checkout process”

Correlations:

FUDs:
Fears
Uncertainties
Doubts
UX research, interviews, online surveys… will give you real feedback about the fears,
uncertainties and doubts.

Designing solutions

Goals Data Hypothesis Solutions
Facturación
GA / Digital
Analytics
Brainstorming UX
Leads
Datos
cualitativos
5 why rule Traffic
Incrementos % Logs
Covariation
hypothesis
Technology
Correlation /
Cause-effect
Business model
User Research FUDs Triggers

Test brieﬁng:

Testing

Goals Data Hypothesis Solutions Test
Facturación
GA / Digital
Analytics
Brainstorming UX A/B test
Leads
Datos
cualitativos
5 why rule Traffic MVT
Incrementos % Logs
Covariation
hypothesis
Technology Split
Correlation /
Cause-effect
Business model JS Custom
User Research FUDs Triggers Personalization

Testing - scientist method:
Independent variable: The variable you´re going to study and modify, and the one
that affect results in the test. You should control that variable. Classic example:
nutrition. Digital Example: information architecture.
Dependent variable/s: DIt depends on the independent variable and its value is
affected by it. They respond to the change made to the independent variable. Classic
example: nutrition. Digital example: bounce rate.
Controlled variables: Controlled variables are quantities that should remain
constant to make the test reliable, and they has to be observed them carefully
because any change in these can modify test results. Classic example: training.
Digital example: Release of new Adwords campaing.

Test brochure:

How many users do I need for a solid testing? - https://www.optimizely.com/sample-
size-calculator/:

size-calculator/:

Conversion rate metric we´re measuring:
conversion rate, download, bounce rate, etc
Miminum relative change we want to detect
in that metric. In this case we want to detect
a range between 0,75%-2,25%
How reliable your test is. Tthe probability
that the results don´t come from chance /
luck (p-value)
size-calculator/:

Traffic volume - statistical signiﬁcance:
Can I do A/B testing if I don´t have enough amount of traffic to gain statistical
signiﬁcance?, Should I do it?, is A/B testing only for big projects?

Testing type:
A/B Test - Split testing: The metrics tested of two versions of a page — version A
and version B — are compared to one another. Site visitors are bucketed into one
version or the other. There´s only one variable changing at the same time.
MVT Test: Compares a higher number of variables at the time times. Several items
(variables) are tested in the same main layout.
Personalization: Based on cookies and navigation paths, we offer a customized
digital experience to the users that ﬁx into a set of rules previously deﬁned.

A/B Test - Split testing:

A/B Test - Split testing:
A: Original Image
B: Alternative Image
C: Alternative Image

MVT Test:

MVT Test:
A: Original Image
B: Alternative Image
C: Alternative Image
A: Original CTA
B: Alternative CTA
C: Alternative CTA

Split Test:

Segments and distribution:
Before launching any kind of A/B testing we have to set the amount of traffic we
want to send to the test (statistical signiﬁcance - reliable) and we have to deﬁne the
right segments tu run the test.

Main segments we have to consider:
Desktop / Smartphone: Una Same solution can be solved so differently depending
on the interface type, so we can have very different results from the same test.
Countries: Same experiment can give us different data depending on the country we
´re runing it.
Business units: Although we´re working in the same e-commerce, each one has
several business units or activities. Not all actions will give us same feedback
depending on the business unit.
Traffic channels: Different results depending n the origin of the traffic.

Main segments we have to consider:
Date: Different results depending on running the test in Christmas, summer, etc.
Political / Society variables
Login or not
Weather

Tools to run an A/B testing
• Optimize (Google)
• Optimizely
• Visual Website Optimizer
• A/B Tasty
• Qubit
• Target
• Wise Conversion
• Maximizer
• Convert

Analyze

Goals Data Hypothesis Solutions Test Analyze
Facturación
GA / Digital
Analytics
Brainstorming UX A/B test Context
Leads
Datos
cualitativos
Statistical
signiﬁcance
Incrementos % Logs
Covariation
hypothesis
Technology Split Cognitive bias
Correlation /
Cause-effect
User Research Triggers Personalization

Lead generation performance - Insurance:

E-commerce add to basket:

E-commerce related products:

E-commerce price management:

Insurance process:

Improvement

Goals Data Hypothesis Solutions Test Analyze Improvement
Facturación
GA / Digital
Analytics
Brainstorming UX A/B test Context Learning
Leads
Datos
cualitativos
Statistical
signiﬁcance
Change
Incrementos % Logs
Covariation
hypothesis
Technology Split Cognitive bias Fail
Correlation /
Cause-effect
User Research Triggers Personalization

Stop guessing. Start
thinking. Put design
in business context.
Go testing.

Thank you Bucharest!
Ricardo Tayar
ricardo@ﬂat101.com

A/B Testing Ultimate Guideline. How to design and analyze digital testing.

More Related Content

Similar to A/B Testing Ultimate Guideline. How to design and analyze digital testing.

More from Ricardo Tayar López

A/B Testing Ultimate Guideline. How to design and analyze digital testing.