Advantages of an in-house testing software solution. Top 3 a/b testing mistakes and how to avoid them.

1. Data Products
Meetup @Outfittery
Fariz Bashirov
25/09/2018

2. The ways to fuck up
A/B testing

3. Why do we need
a custom testing
tool?
Learn about the advantages
you can get by using a
similar setup.
3

4. When can the experiment
be completed?
4
What will happen to the users in
each of the customer segments?
What is the impact on all the
relevant business KPIs?
More granularly defined test groups
Data steering with custom routes and
enrichments
Deeper evaluation of the experiment results using
custom KPIs
Calculating the necessary number of
samples based on the defined KPIs and
significance

5. Applications
Smart Gateway
Deciders
5
Smart Gateway Architecture
Services

6. Let’s see how it looks like
6
DEMO

7. Deciders
Deciders are the “atomic” machine learning services
providing the decision value/score
7
HTTP service
Constant value
Async
Template Output

8. Experiments
Experiment define the test groups, sampling and
treating periods.
8

9. Decision Points
Where all the configs get mapped:
● Cache configuration
● Enrichment
● Routing to deciders
● Sampling by experiments
● Treating by experiments
9

10. Decision Point, Experiment, Decider
10
Application (e.g. Web Funnel) Decision Point Default Decider
Default Decider Decider X Decider Y
Experiment ABC
Route 1
Route 2
Group 1 Group 2 Group 3
Enrichment
Service
Shadow Decider Z

11. 11
Regardless of the tooling
you use - it is possible to
avoid those mistakes.
Top 3 fuck ups
we had to learn
from.

12. You can estimate or even
calculate the end date for a
given a/b test.
Do you really want to set it
as a fixed date/time?
12
Test ended with
no results.

13. What happens if we find an
issue and switch the test
group B weight to 0?
...and switch it back a week
later?
13
“Unexpected”
user behaviour.

14. “Have fun” evaluating test results.
WHEN TEST GROUPS CHANGE DURING THE A/B TEST
100 / 0
50 / 50
50 / 50
14
100 / 0
70 / 30
Separate experiment revisions

15. Understand what will happen to the customers in each
of the test groups
Dive deeper into business KPIs to see the full
business impact
15
Summary
Calculate the amount samples with the significance
you need

16. Happy Testing!
Fariz Bashirov
Senior Product Manager - Data Solutions
fariz.bashirov@outfittery.de
25/09/2018