1.
A/B testing workshop
“In God we trust, all others must bring data”
JSConf Colombia Workshop 2015

2.
@shiota
github.com/eshiota
slideshare.net/eshiota
eshiota.com

3.
A/B

4.
A/B tests measure how a new idea
(version B/variant/test) performs
agains an existing implementation
(version A/base/control).

5.
Buy now Buy nowversus

6.
coin flip
Buy now
Buy now
50%
50%

7.
When the user sees or is affected by
the idea, they are tracked and
become part of the test.

8.
Buy now
Buy now
track(my_experiment)

9.
Data about the website is generated
as users browse through pages and
do their tasks.

10.
product added to cart
number of products added
purchase finished
average price per purchase
number of products seen
user has logged in
used guest checkout
customer service calls
…

11.
When there’s enough information to
make a decision, you can either stop
the test (keeping version A) or choose
version B, directing all traffic to it.

12.
Buy now Buy now
Duration: 14 days
Visitors: 45.140 (22.570 per variant)
339 (1.5%) 407 (1.8%)
20% up
144.500 COP 147.390 COP
2% up
Number of purchases:
Average price:

13.
coin flip
Buy now
Buy now
50%
50%

14.
B
Buy now
100%

15.
"But my design is obviously more
beautiful and intuitive than what
we have now! Why should I run
an A/B test?”
— the majority of designers

16.
Quiz time!(prizes included)

17.
A: Raise your left hand B: Raise your right hand
Neutral: Don’t raise your hands
Which performed better?

18.
Reduced bounce rate in 1.7%

19.
A: Raise your left hand B: Raise your right hand
Neutral: Don’t raise your hands
Which performed better?

20.
Which performed better?
Increased CTR in 203%

21.
A: Raise your left hand B: Raise your right hand
Neutral: Don’t raise your hands
Which performed better?

22.
Which performed better?
43.4% more purchases

23.
A: Raise your left hand B: Raise your right hand
Neutral: Don’t raise your hands
Which performed better?

24.
Both were statistically equivalent
Which performed better?

25.
Intuition
vs
Historical Analysis
vs.
Experimentation

26.
We have a 2/3 chance of being
wrong when trusting our intuition.

27.
People behave differently each
season/month/day of the week.

28.
Different cultures lead to
different patterns of usage.

29.
Data analysis alone provides
correlation but not causation.

30.
Running your A/B test
(in 5 simple steps)

31.
Step 1: Hypothesis

32.
Analyse all possible inputs to come up
with an hypothesis to work on.

33.
• Usability research
• Benchmarking
• Surveys
• Data mining
• Previous experiments

34.
Hypothesis:
“If users from South America
countries relate more to the
website, they will book more.”

35.
Step 2: Idea

36.
Idea:
“If we add the country’s flag next
to the website’s logo, users will
relate more to the brand.”

37.
Step 3: Setup

38.
• Who will participate?
• What is the primary metric?
• Any secondary impacts?
• How will it be implemented?

39.
• Users from Argentina, Bolivia, Brazil, Chile,
Colombia, Ecuador, Guyana, Paraguay, Peru,
Suriname, Uruguay and Venezuela, on all
platforms
• Conversion (net bookings) uplift is expected
• We expect more returning customers

40.
<h1 class="main-header__logo logo">
<% if user.is_from_south_america && track_experiment(:header_flag_for_south_america) == "b" %>
<span class="main-header__logo__country-flag">
<%= user.country %>
</span>
<% end %>
<%= image_tag "logo.png" %>
</h1>

41.
Step 4: Monitoring

42.
Keep checking the metrics to see if
anything’s terribly wrong.

43.
Avoid checking too often, let your test
get enough users and enough runtime.

44.
Step 5: Data, decisions,
and next steps

45.
When you reach the expected runtime,
number of visitors or effect, look at the
data and take a decision.

46.
product added to cart
number of products added
purchase finished
average price per purchase
number of products seen
user has logged in
used guest checkout
customer service calls
…

47.
Optimizely dashboard

48.
• How were the primary and secondary
metrics impacted?
• What were the results isolated by each
country?
• What were the results isolated by each
language?
• Did any particular platform (desktop,
mobile devices, tablets) perform better?
• Was the impact on returning customers
any higher than first time visitors?

49.
Based on the gathered data,
plan for next steps.

50.
• Should we add a copy to the flag?
• Should we add a tooltip to the flag?
• Should we increase/decrease the flag size?
• Should we restrict it just for desktop users?
• Should we try this for a single country, or
other countries?

51.
What can you test?

52.
(almost) Everything.

53.
You can test a small
design change.

54.
versus

55.
You can test large
design changes.

56.
versus

57.
You can test different copies.

58.
versus
Submit
Book now

59.
You can test technical
improvements and measure page
load time, repaints/reflows, and
conversion impact.

60.
versus
jQuery 1.11.3
jQuery 2.1.3

61.
You can even test back-end
optimisations and measure
page load time, rendering time,
CPU and memory usage etc.

62.
if track_experiment(:my_optimized_query)
@users = my_optimized_query
else
@users = do_the_normal_thing
end

63.
Live coding
(I hope that works.)

64.
Find the code at:
https://github.com/eshiota/ab_workshop
Additional links:
https://www.optimizely.com/
https://github.com/splitrb/split/
http://whichtestwon.com
http://unbounce.com/
http://blog.booking.com/hamburger-menu.html
http://blog.booking.com/concept-dne-execution.html
Gracias!