What is A/B testing, how to put in into place and the essential guidelines for getting started

1. consultingSTARTUP MINDED
THE GUIDE
TO A/B TESTING

2. 2
CONTENTS
A - WHAT IS A/B TESTING ?..................................................................................................3
ThePrincipleofA/BTesting..............................................................................................................................3
Improving the conversion rate: an important leverage effect................................................................6
B - HOW TO PUT A/B TESTING INTO PLACE: THE MILESTONES.................................7
Identify the opportunites for optimisation and construct hypotheses...............................................7
Prepare the tests: conception of variants and planning.........................................................................9
Interpret the results........................................................................................................................................11
C - GUIDELINES FOR GETTING STARTED: ......................................................................12
Adapttheorganization...................................................................................................................................12
Choosetherighttools.....................................................................................................................................13
consultingSTARTUP MINDED

3. 3
A/B Testing is a technique that enables testing of two versions of the same device in which a
single variable is different. A/B Testing therefore measures the impact of that modified variable
on the achievement of a given objective (rate of conversion, rate of clicks, lead generation...).
In the american presidential race of 2008, Barack Obama’s team implemented an innovative
approach for boosting fundraising by relying on A/B Testing. The obtained result: 60 million
dollars of additional collection, which is around 8% of the total 760 million dollars spent
during the campaign! During the long months, Obama’s team tested multiple variants of
communication supports in order to retain only the most successful. In the example below, to
optimize the registration page, several visuals and texts for the button were tested: the visual
on the right, with Obama and his family, generated 13% additional conversion.
ILLUSTRATION 1 : VARIANTS USED FOR FUNDRAISING FOR OBAMA’S 2007 CAMPAIGN
(SOURCE : OPTIMIZELY)
THE PRINCIPLE OF A/B TESTING
A - WHAT IS A/B TESTING?
A B

4. 4
Beyond the field of political communication, a growing number of sites will now implement
A/B Testing techniques: e-commerce sites, service sites, content sites... Dell conducted a
number of tests on their contact page which illustrate what is possible in this area:
A/B Testing allows the user to be very efficient in the conception phases. In fact, the
dedicated meeting for the conception of application screens, or other marketing materials, are
often interspersed with long exchanges over ergonomic choices, the content of catch phrases,
the position and color of buttons... Pieces of advice collide, returning to this or that «leading»
practice of a key player, without being able to prove that it is really transposable / relevant in
the given context.
A/B Testing can often provide an objective response to this problem. The approach consists
in comparing several versions of a support to determine which performs best according
to a given criterion: the rate at which it is «added to cart», the rate of inscription, or any
other measurable reference indicator. The example on the next page shows a schematic
representation of two variants in a conversion funnel: the replacement of a text by a visual in
the second version leads to an improvement in the rate of conversion of 4 points.
Result: the version B enables a gain for 2 KPIs: -27% rebound rate, 36% form submission from version A
ILLUSTRATION 2 : DELL CASE, CONTACT PAGE OPTIMIZATION FOR DELL ENTERPRISE
(WWW.WHICHTESTWON.COM)
A B

5. 5
Concretely, the completed tests can affect the optimisation of a large number of elements.
The most common are given in the table below:
ILLUSTRATION 3 : TWO MODELS ARE TESTED AND EVALUATED BASED ON THE OBTAINED
CONVERSION RATE (SOURCE : KELEY CONSULTING)
STANDARD MEDIAS
Screen > Web page
> Mobiel App screen/
touchpad
Direct
Marketing
> E-mail
> SMS
> Mobile App push
notification
Advertising
Support
> Banner
> Sponsored link, including
Adwords
STANDARD ITEMS
Hook > Title
> Subtitle
> Tagline
Text > Description
> Benefits
> Testimonials, social proof
> Awards
Image > Icons
> Photos
> Videos
Chart > Color schemes
> Size of the items
Ergonomy > Zoning
> Scroll vs sequence
> Menu
> Links
TABLE 1 : THE USUAL DIGITAL ELEMENTS TO A/B TEST
(SOURCE : KELEY CONSULTING)
Model A (control) Model B (variant)

6. 6
The current guide concentrates on A/B Testing in the context of digital materials, but the me-
thod also applies to numerous areas of «offline» marketing, notably:
> Direct marketing: tests for mail, coupons...
> Merchandising: changes in stalls of large distribution, organization of rays...
> Packaging: during the testing phase, presentation of different variants
> TV Advertisement: comparison of multiple commercial concepts
> Product development: tests of variants in food, perfume...
A/B Testing enters into the large family of methodologies and tools of CRO (Conversion
Rate Optimization). Enabling the continual optimisation of the performance of digital devices
and relying on the scientifc method, A/B Testing offers both significant ways to save time in
arbitrating complex decisions, and a «depersonalization» of the debate.
Fairly often, the economic effect of an increase in the conversion rate is underestimated,
the efforts primarily focusing on the increase in traffic. However, the improvement of the
transformation rate of a device often produces a significant leverage effect on the margin rate
of an operation. In the example on the following page, corresponding to an online site selling
physical products, we can see that an increase of 20% in daily sales steps up the net margin by
80% (leverage x4), hence a boosted ROI and a project that funds itself much quickly.
IMPROVING THE CONVERSION RATE: AN IMPORTANT
LEVERAGE EFFECT

7. 7
B - HOW TO PUT A/B TESTING INTO PLACE:
THE MILESTONES
The first step consists of clarifying the objectives of the devices to optimize, as indicators of
improvement, notably: rate of inscription, rate of addition to cart, cart
abandonment rate, average basket...
It is then necessary to identify the best optimization opportunities: those that will have the
strongest impact on performance. We are traditionally interested with the analysis of weak
points, via the Web Analytics tool, for example: the pages with the strongest rebound rate, the
steps generating the most cart abandonment.
IDENTIFY THE OPPORTUNITIES FOR OPTIMAZATION AND
CONSTRUCT HYPOTHESES
COSTS
Paid traffic share 60% 60% 60% 60% 60%
Paid visits / day 1200 1200 1200 1200 1200
Cost / paid visit 0,6 € 0,6 € 0,6 € 0,6 € 0,6 €
Paid visits cost 720 € 720 € 720 € 720 € 720 €
Acquisition cost 18,0 € 17,1 € 16,4 € 15,0 € 13,8 €
Production cost share 50% 50% 50% 50% 50%
Production cost / basket 40 € 40 € 40 € 40 € 40 €
Production cost 1 600 € 1 680 € 1 760 € 1 920 € 2 080 €
Total costs 2 320 € 2 400 € 2 480 € 2 640 € 2 800 €
NET MARGIN
Net margin / day 240 € 288 € 336 € 432 € 528 €
Net margin / year 87 660 € 105 192 € 122 724 € 157 788 € 192 852 €
Net margin variation - 20 % 40 % 80 % 120 %
Annual earnings via A/B
Testing
0 € 17 532 € 35 064 € 70 128 € 105 192 €
An increase of 20% of turnover excl. taxes ( 0.40 points of conversion rate) enables the increase in
the net margin of 80% in the featured model.
TABLE 2 : EFFET OF A CONVERSION RATE INCREASE ON NET MARGIN
(SOURCE : KELEY CONSULTING.)
Initial situation Conversion rate
increased by 0.10
point
Conversion rate
increased by 0.20
point
Conversion rate
increased by 0.40
point
Conversion rate
increased by 0.60
point
Conversion rate 2% 2,10% 2,20% 2,40% 2,60%
Turnover variation
excl. taxes
- 5% 10% 20% 30%
INCOME
Visits / day 2000 2000 2000 2000 2000
Sells / day 40 42 44 48 52
Average basket 80 € 80 € 80 € 80 € 80 €
Turnover excl. taxes / day 2 560 € 2 688 € 2 816 € 3 072 € 3 328 €

8. 8
Sources of optimization can likewise be identified by comparison with benchmarks from the
competition. It therefore focuses on improving aspects that clearly underperform, compared
to competitive sites or references.
Once the priority optimizations and the objectives are identified, it is necessary to identify
hypotheses for the brakes to raise and the levers to activate to increase the performance of the
items concerned. To identify these hypotheses, one can carry out the following activities:
> Internally: workshops with the marketing teams and the selling power
> With the customers: online questionnaires, chats, interviews by telephone or in person
The I.A.R. model published by Conversionista is useful for forming hypotheses:
ILLUSTRATION 4 : DISPLAY OF THE PAGES WITH THE STRONGEST REBOUND
RATE ON GOOGLE ANALYTICS
Insight As we observe that: (Result)
By : (observation method)
Action We wish to: (implement this change)
for: (this segment / this user profile)
Result Which should result in: (this change in behaviour)
And the effect will be measured in the following manner : (indicators / impacted metrics)

9. 9
Furthermore, there are numerous online resources, notably the site whichestwon.com, which
provide A/B Testing data bases, put together by a variety of organizations.
ILLUSTRATION 5 : WWW.WHICHTESTWON.COM, THE WEBSITE THAT PROVIDES MANY
EXAMPLES OF A/B TESTING
Once the testable hypotheses are identified, the variants of pages can be designed. In order
to accelerate the implementation of the tests, it is recommended to use the tools of the online
edition, provided by the solutions of A/B Testing, bypassing the purely graphic models.
As a matter of fact, the tools of A/B Testing generally allow the creation of variants to test,
without the need to program: modifications can be presented in a visual way.
We could, for example, displace the elements and change their size, modify the content (texts,
visuals, videos...), add or delete elements.
PREPARE THE TESTS:
VARIATIONS DESIGN AND PLANNING

10. 10
ILLUSTRATION 6 : OPTIMIZELY INTERFACE THAT ALLOWS TO CREATE
VARIATIONS FROM A WEBPAGE
Once the variants are created in the tool, it is important to make a recipe for controlling
their operation in all the provided terminals, notably cellphones and tablets. In fact, certain
modifications sometimes cause dysfunctions that must be corrected manually by a technical team,
directly in the A/B Testing tool.
Finally, before running the tests, it is advised to inform the teams in charge of customer
relations, if the impacts are significant, so that they can properly support users as needed.
A/B Testing does not degrade the natural referencing of a site as long as it is put in place with
respect to a few simple rules. Google also encourages the implementation of A/B tests via the
distribution of tools such as Google Analytics.
Search engine teams communicate by rules of implementation:
> Avoid the “cloaking”: Presenting a different version of a Web page to indexing
robots of search engines and to visitors. This practice is discoverable by search engines
which can degrade the ranking of a suspected site or even dereference it from SERPs.
Therefore, one should not use A / B testing to target the crawlers, such as Google.
> Avoid the “duplicate content”: Pages that exist under multiple variants can be
signaled to engines using the tag «rel=canonical». This confirms to the engine that the
variants are connected to the principle page.
> Do not slow down the loading of the site: The implementation solution does
not have to slow down the loading of the pages, an important factor taken into account
by the engines to establish the ranking of pages in the search results.

11. 11
Interpretation of the results is a delicate stage because it requires the mastery of basic statistical
concepts, including on the representativeness of the results. Indeed, a performance variation can
only be considered statistically valid if it has been observed on a representative sample of the
population studied. Fortunately, online tools facilitate interpretation by performing the calculations
automatically, including the following tool: Sample Size Calculator (Evan’s Awesome A / B Tools).
Sample Size Calculator (Evan’s Awesome A/B Tools).
THE GUIDE TO A/B TESTING
ILLUSTRATION 7 : ONLINE TOOL FOR EVALUATING IF A TEST IS
STATISTICALLY SIGNIFICANT
Moreover, it happens regularly that the A/B tests are not relevant, especially for the
following reasons:
> Not enough users tested the proposed variant, so the test is not statistically significant
> There were several changes in the same test and they are offset
> The modification did not have a connection with the measurement used to evaluate the
result of the test
When these situations present themselves, it is necessary to document them, as to capitalise
on their lessons and use them for the next tests.

12. 12
C - GUIDELINES FOR GETTING STARTED
The organizations that effectively use A/B Testing often carry out hundreds of tests per month. The
website Etsy, for example, generates 25 A/B tests per day and Shop Direct, more than 100 per day.
To reach this pace, the organization must be adapted to avoid the pitfalls associated with classic cycle
projects : validation by several hierarchical layers, design review meetings...
To save time, it is thus recommended to appoint officials who will work with independent tests
within the framework imposed by a A/B Testing charter, of which the terms are negociated with
the stakeholders of the company usually called to intervene in the lifecycles of the projects.
Typical categories of an A/B Testing charter :
ADAPT THE ORGANISATION
Rules Description Examples
Authorized pages List the pages that should not be A / B
tested
No test on the following pages: the values
of the company, CSR policy ...
Texts List restrictions on certain texts Terms & Conditions (GTC), Terms of Service
(TOS)
Visuals List restrictions on visuals Only use visuals which are in the directory
selected by the internal communication
Periods List the restrictions on testing periods No test during sales weeks
I It is also useful to create a shared file to document the passed tests and conserve a memory of
accumulated knowledge in the framework of the tests : the hypotheses tested, the obtained results...

13. 13
Numerous tools appeared to complete the A/B tests, they can be classified into three categories :
1. Dedicated SaaS solutions for A / B testing: Optimizely, who knew strong
international success, or French A/B Tasty, with important parts of the market on the
national territory;
2. SA/B Testing solutions included in a marketing suite: Certain digital marketing suites
present a dedicated tool for A/B testing. This is the case with the Adobe Marketing Cloud
suite with Adobe Target, and with the Google Analytics 360 suite with Google Analytics;
3. Internal solution: The tools are sometimes implemented by the agencies or the
internal teams of big businesses.
It is recommended to take precautions regarding the internal solutions for several reasons:
> To limit the dependence on a provider, if the solution is proposed by an agency,
> Because the A/B Testing solutions are complex softwares, which require substantial
investments, difficult to mobilize by teams who haven’t achieved a certain critical success.
In April 2016, the ranking established by TrustRadius put the solutions of A/B Testing of
Optimizely and AB Tasty in the forefront.
To learn more about A/B Testing and to meet our teams, contact us :
Keley Consulting, 28 rue du Dr Finlay, 75015 Paris
Phone : 01.80.48.26.20. - Email : contact@keley-consulting.com
CHOOSE THE RIGHT TOOLS
ILLUSTRATION 8 : MAPPING TOOL PROVIDED BY TRUSTRADIUS
(HTTPS://WWW.TRUSTRADIUS.COM/AB-TESTING).