Definition of A/B testing and Case Studies by Optimizely
16/05/2014 by RusseWeb
Learn A/B testing
Deﬁnition of A/B testing and Case Studies by Optimizely
✤ Conversion Rate - the
percentage of visitors who take a
✤ The average conversion rate on
the web - the rate at which
visitors convert into customers -
is at only 2%. !
✤ That means that the 98% of
potential customers that reste
never will convert on YOUR web
What is A/B testing ?
✤ A/B testing is the simple idea of showing several
different versions of a web page to live traﬁc.!
✤ You can then mesure the effect the each version has on
✤ By doing so you can drastically improve the
effectiveness of your marketing and user experience
The Main Goal is to Improve your Conversion Rate
What to test first ?
Optimization guide in 5 steps
The hardest part of A/B testing is
determine what to test first
✤ Step One : Deﬁne success or the reasons of your site’s
✤ Step Two : Identify bottlenecks or where do you lose your
✤ Step Three : Construct a hypothesis!
✤ Step Four : Prioritize!
✤ Step Five : Test !
Define success of your site
✤ What is your website for ? !
✤ Identify what constitutes - and does not constitues - a
« conversion » for your web site. !
✤ Depending on your business model deﬁning a success
conversion metrics can be a tricky task.
✤ Bottlenecks are the places where your users are
Construct a Hypothesis
✤ After you found the bottlenecks of your site, try to
understand why your potential costumers leave at this step ? !
✤ Make a hypothesis, try to think as your clients!
✤ To understand your customers vision you can use
interviews, feedback forms, groupes.!
✤ If you are running tests without a hypothesis beforehand,
you might gather information that’s helpful anecdotally
while missing the deeper lesson.
✤ You should keep a sense of your testing priorities in
✤ Your projected ROI (Return On Investment) from each
test will itself be derived from a combination of your
core success metrics that you determine in Step One,
the Bottlenecks (Step two) and your hypothesis about
user’s behavior (Step Three).
✤ Randomly selected users!
✤ Different site versions!
✤ Before ﬁnish the test check if it has reached its
✤ Remember : often a completed test brings not only
answers but more questions.
✤ The Local Maximum - the nearest target or your
✤ The Global Maximum - the main goal of your
Think BIG !
Break from the Status Quo :ABC Family
✤ Experiment of Disney on their ABC Family homepage
using Optimizely platform :
The main page displayed a large promotion for a television show you might be interested in. After some
researches Disney team discovered that a lot of people were searching for the exact title of shows and speciﬁc
The team decided to reevaluate their entire approach : alternative view, less visual and more hierarchical. Users can
search from top to bottom for a speciﬁc show.!
By being open to this big, fundamental change
Disney was able to effect an engagement increase of
more than 600% !
✤ Use the insights from small tests to guide and inform
your thinking about bigger changes.!
✤ Consider entirely new alternative approach to your
principal business goals. !
✤ if you’re working on a major site redesign or overhaul,
don’t wait until the new design is live to A/B test it.
A/B test the design itself. Test each feature step by
✤ Site that uses templates to design pages instead of
designing each page. !
✤ Testing one element across all pages of one type :
product pages, content pages, funnel pages.!
✤ Build a change one time and you can use it for several
Travel company VacationRoost has several travel-
lodging sites that propose searching for rental
properties, destinations, or deals.
✤ Visitors enter on sites via targeted landing page based
on researches they made earlier!
✤ Tested changes on the site : Security seals and logos,
call-to-action buttons and copy, content length and
format, number of calls-to-action displayed, search
✤ Two primary goals : CTR to search results page and
VacationRoost and Results
after A/B testing
✤ Security seals and logos boost user engagement!
✤ Large and orange call-to-action button provides the biggest
✤ Longer pages have bigger conversion (more content and
✤ More Calls-to-action increase the number of users who entered
✤ Adding mini-ﬁlters or search options increase engagement
Testing methods ofVacationRoost
✤ Test and iterate: ﬁrst large-scale design changes then
more granular, subtle revisions like CTA copy or
security seal placement. !
✤ For the fast results, test to highly trafﬁcked pages.!
✤ Test across multiple pages at once.
✤ Optimizely is a world-class
website optimization platform
that enables users with and
without technical expertise to
make dynamic changes to their
websites, test the variations to
live trafﬁc, gather immediate
results and start achieving goals. !
✤ Optimizely helps more than 6,000
clients drive more leads, sales
and conversions through simple
website testing and optimization
Optimizely By the Numbers
(As of Oct. 2013)
2.2 Billion visitors tested across all customer websites
300,000+ experiments run since launch
✤ Book « A/B testing. The most powerful way to turn
clicks into customers » by Dan Siroker and Pete Koomen
Blog sur l’internet russe et l’actualité du web russe. Les particularités du référencement sur le moteur de
recherche russe Yandex