2. Introduction
• A/B testing (also known as bucket testing, split-run testing, or split
testing) is a user experience research methodology.
• A/B tests consist of a randomized experiment that usually involves
two variants (A and B), although the concept can be also extended to
multiple variants of the same variable.
• It includes application of statistical hypothesis testing or "two-sample
hypothesis testing" as used in the field of statistics.
• A/B testing is a way to compare multiple versions of a single variable,
for example by testing a subject's response to variant A against
variant B, and determining which of the variants is more effective.
3. What is A/B Testing
• A/B testing (sometimes called split testing) is comparing two versions of a
web page to see which one preforms better.
• You compare two web pages by showing the two variants (let’s call them A
and B) to similar visitors at the same time. The one that gives a better
conversion rate, wins1.
• Running an A/B test that directly compares a variation against a current
experience lets you ask focus question about changes to you website or
application, and then collect data about the impact of that change.
• Testing takes the guesswork out website optimization and enables data-
informed decisions that shift business conversations from ”we think” to
“we know.” By measuring the impact that changes have on your metrics,
you can ensure that every change produces positive results
4. How A/B Test Work
• In an A/B test, you take a webpage or application screen and modify it
to create a second version of the same page. This change can be as
simple as a single headline or button, or be a complete redesign of
the page. Then, half of your traffic is shown the origin version of the
page (know as the control) and half are shown the modified version
of the page (the variation)
All websites on the web have a goal –a reason for them to exist
-eCommerce websites want visitors buying products
-Saas web apps want visitors readers to click to on ads or sign up for
paid subscription
5. How A/B Test Work
-News and media websites have a goal –a reason for them to exist
• Every business website wants visitors converting just visitors to
something else. The rate at which a website is able to do this is its “
conversion rate”. Measuring the performance of a variation (A or B)
means measuring the rate at which it converts visitors to goal
achievers
7. Why should You Perform A/B Test
• A/B testing allows you to make more out of your existing traffic. While
the cost of acquiring paid traffic can be huge, the cost of increasing
your conversions is minimal. To compare, a small business plan of
visual website optimizer start at $49. That is the cost of 5 to 10 google
Adwords clicks. The return on investment of A/B testing can be
massive as even small changes on a landing page or website can result
in significant increases in leads generated, sales and revenues
8. What Can You Test ?
• Headlines
• Sub headlines
• Paragraph Test
• Testimonials
• Call to Action test
• Call to Action Button
• Link
• Images
• Content near the fold
• Social mentions
• Awards and badges
Almost anything on your website that affects visitor behavior can be A/B tested
9. A/B Testing Process
• 1. Study your Website Data: Use a website analytics tool such as
Google Analytics, and find the problem areas in your conversion
funnel. For example, you can identify the pages with the highest
bounce rate. Let’s say, your homepage has an unusually high bounce
rate.
• 2. Observe User Behavior: Utilize visitor behavior analysis tools such
as Heatmaps, visitor recordings, Form analysis and On-page surveys
and find what is stopping the visitors form converting. For example, “
The CTA button is not prominent on the home page.”
The correct way to run an A/B testing experiment is to follow a
scientific process. It includes the following step
10. A/B Testing Process (cont…)
• 3. Construct a Hypothesis: Per the insight from visitor behavior analysis tools,
build a hypothesis aimed at increasing conversions. For example. “Increase the
size of the CTA button will make it more prominent and will increase
conversions”.
• 4. Test your Hypothesis: Create a variation per your hypothesis, and A/B test it
against the original page.
For Example. “A/B Test your original home page against a version that has a larger
CTA button.”
Calculate the test duration with respect to the number of your monthly visitors,
current conversion rate and the expected change in the conversion rate
.5. Analyze Test Data and Draw Conclusions: Analyzer the A/B test results, and see
which variation delivered the highest conversions. If there is a clear winner among
the variations, go ahead with its implementation. If the test remains inconclusive,
go back to step number three rework your hypothesis.
11. Concluding Thoughts
A/B Testing is a powerful tool to optimize your product
When you test, understand what you gain
When you don’t test, be aware of how you’re making decisions
Keep in mind your ultimate KPI goal
Be considerate!
Statistical Vs Practical significance
Downstream Metrics
Local Maxima
