Experimentation as a growth strategy at Booking.com

1. Experimentation @Booking.com
Jorden Lentze
24th of January, 2019 Many thanks to Willem Isbrucker

2. 1. Booking.com, nice to meet you
2. The business impact of experimentation
3. Experimentation @Booking.com

3. Jorden Lentze
Product manager, Booking.com
● 1.5 year at Booking
● Previous at Google and ABN AMRO
● Conversion Rate Optimization
● 400+ experiments
● Rotterdam
https://www.linkedin.com/in/customerdrivenoptimization/

4. Timeline.
1996
Founded in Amsterdam

5. Winterberg

6. Winterberg

7. 2012
4,000 employees
2016
13,000+ employees
2000
8 employees
OVER
100
NATIONALITIES
2018
17,000+ employees

8. 28 million
total reported listings
43
Languages supported
200+
Offices
OVER
100
NATIONALITIES
161,000,000+
Reviews from real guests
1,550,000+
Room nights booked
every 24 hours

9. Agenda.
Source: Kleiner Perkins 2018 Internet trends: https://www.statista.com/statistics/277483/market-value-of-the-largest-internet-companies-worldwide/

10. 1. Booking.com, nice to meet you
2. The business impact of experimentation
3. Experimentation @Booking.com

11. Why?

12. 9/10 experiments fail
for users.
90% of product decisions have an
inconclusive or negative effect on
a product’s primary metric
Source: VWO 2016.
(8.75/10 tests fails according to AppSumo)
90%
10%
4

13. Evercore Equity Research
https://goo.gl/rFP
“Booking’s utilization of A/B testing has
contributed to Booking enjoying
conversion rates 2-3x higher than the
industry average.”
3

14. Product Death Cycle.
2

17. Experimentation at Booking.com

18. Base.
Which one performed better?
Variant.

19. Which one performed better?

20. Base.
Which one performed better?
Variant.

21. Base.
Which one performed better?
Variant.

22. Hypothesize &
Experiment.
• Hypothesize
• Calculate power
• Run for prescribed time
• Stop, rerun or Full On
• Iterate
Base.
Variant.
Search
Search

23. Experiment on everything.
4

24. Experiment atomically.
3

25. Interactions.
Experiment on
Everything
Test Atomically
A lot of
experiments!
+ =

26. 1,000+
concurrent tests.
5

27. Based on [prior] we believe that
changing [condition]
for users [sample] will make
them [outcome].
We will know this when we see
[effect(s)] happen to [metric(s)].
This will be good for customers,
partners and our business
because [motivation].
Idea ≠ Hypothesis.
Hypothesis Template.
3

28. Idea ≠ Hypothesis.
• All secondary metrics
selected a priori along with
their direction of change to
avoid Texas Sharpshooter
Fallacy

29. ● All experiments should have
run time calculated
beforehand using the power
calculator of your choice.
Or use ours :-)
https://bookingcom.github.io/
powercalculator/
● Experiments should then be
run for the next closest full
week cycle.
Properly Powered.
Full Week Cycles.
Experiments should:

30. No HIPPOs.
6

31. Nobody knows.
6

32. Teams made for
experimentation.
5

33. Oleksandr
Kaverin
- iOS developer-
George
R.R. Martin
Springthorpe
- UX Copywriter -
Serhii Tostii
Korin
- Back End Developer-
Simona
Damian
- Jr. Product Mk Manager -
Mat-log
Swainson
- Team Lead -
&
- Client Side Developer -
Jorden
Rotterdam
Lentze
- Product Owner -
Alexi
Munoz
- UX Designer-
Annuhh
Zuiker
- Data Scientist -
Les Paul
Bodron
- UX Designer -
Jay
Rambhia
- Android Developer -

34. Allow 100% access to
data.
3

35. Data is fuel for
hypotheses.
3

36. Guidelines, not rules.
3

37. Embrace failure.
4

38. Fail fast.
Learn
3

39. Don’t assume
correlation.

40. Don’t assume
reproducibility.
3

41. Experiment everywhere.

42. Product
Development.
Data.
Analyze.
Hypothesize.
Experiment.
Iterate.
The Data-Informed Decision Cycle.

43. Feeding the
experiment engine.

52. CRO is part of every
team.

53. Winterberg - At Booking customers drive the product

54. Most democratic way
to design a website.

55. Go where the
experiment takes you.

56. Thank you.
https://blog.booking.com/

57. Thanks to
Willem Isbrucker
Senior Product Owner, Booking.com
● Search products at
Booking.com
● 2 years at Booking
● Many more in Product
● 1000+ experiments
www.linkedin.com/in/willemisbrucker