1.
A/B Testing with Geodata
in Booking.com
Elena Nikolaeva | November 2017
@ice_lenorice.lenor@gmail.com | smartpuffin.com |

2.
A/B testing
What is it?

3.
Which button color is
the best?
OR
?

4.
Let the users
decide!

5.
Experiment flow.
50%
50%
10% conversion
20% conversion Win!
Lose.

6.
10% conversion
20% conversion
What does “the best” mean?
50%
50% Win!
Lose.

7.
Metrics:
● Conversion
● Click-through rate
● Log-ins
● Bounce rate
... anything your business
wants the customers to do!

8.
Additional metrics:
● Clicked on an element
● Clicked on a different
element
● Spent time on the page
● Scrolled down
Helps to learn about user
behavior.

9.
Building a hypothesis

10.
“If I implement X,
the users will do Y,
and it’s good because Z”

11.
“If I... add a cat to my presentation,
the listeners will... say AAAaaw,
and it’s good because... they will
remember my talk”

12.
Success:
Congratulate yourself!
Continue on topic.
Where else can you
apply the same
approach?

13.
Failure:
That’s unfortunate.
But don’t give up just
yet!
Analyze what went
wrong.

14.
Repeat.

15.
RepeatAnalysisExperimentMetrics
Steps to a successful experiment:
Hypothesis

16.
Learning
is the most important thing

17.
A/B testing:
not only button colors

18.
What is
geodata?

19.
80%
of users think
location is very important
when choosing an accommodation.

20.
How do we run
A/B tests on geodata?

21.
Step 1. Build a hypothesis

22.
User research

23.
Street testing

24.
Surveys

25.
Data analysis

26.
“I want to stay in the historical center
of the city,
but I don’t know where it is.”

27.
2. Implement the experiment.

28.
Brainstorm

29.
Our Geo Team

30.
Data collection process.
MVP approach: collect a small piece of data
to experiment with.
Prioritize collection!

31.
We collected the data
for 30 cities
where our customers left reviews
mentioning ”Old Town”
the most
Data collection process.

32.
Lisbon Old town

33.
Old Town

34.
6. Repeat.

35.
B.old Town

36.
Now grow the data!
We collected 100 more
old towns!
Success?

37.
Our challenges

38.
Defining
customer problems
is hard.

39.
Many places.

40.
Many places.

41.
Very location-specific.

42.
Thank you!
Elena Nikolaeva | November 2017
@ice_lenorice.lenor@gmail.com | smartpuffin.com |