The document discusses the importance of data science and experimentation at booking.com in product development, emphasizing their role in addressing customer needs. It outlines the data-driven decision cycle, the high failure rate of experiments, and the company's experimentation culture that encourages innovation and learning from failures. Key contributors, Anne Zuiker and Jorden Lentze, highlight the significant impact of A/B testing on conversion rates, achieving results 2-3 times higher than the industry average.

1. Data science and experimentation at
Booking.com: a data-driven approach to
solving customer needs
Anne Zuiker & Jorden Lentze
20th of September, 2018

2. Today.
Data science and experimentation combined are essential in product development at
Booking.com.
They enable us to avoid the product death cycle and address customer needs at scale.

3. Anne Zuiker
Data Scientist & Team Lead, Booking.com
● 3.5 years at Booking
● 9 years in Amsterdam
● I love cats!
https://www.linkedin.com/in/anne-zuiker-131bba65/

4. Jorden Lentze
Product Owner, Booking.com
● 1 year at Booking
● Many more in Product and
Conversion Rate Optimization
● 300+ experiments
● I love cats
● Rotterdam
https://www.linkedin.com/in/customerdrivenoptimization/

5. 1. Booking.com, nice to meet you
2. Data science @ Booking.com
3. Experimentation @ Booking.com
→ Solving customer needs

6. Winterberg

7. Winterberg

8. Timeline.
1996
Founded in Amsterdam

9. 2012
4,000 employees
2016
13,000+ employees
2000
8 employees
OVER
100
NATIONALITIES
2018
17,000+ employees

10. 28 million
total reported listings
43
Languages supported
200+
Offices
OVER
100
NATIONALITIES
161,000,000+
Reviews from real guests
1,550,000+
Room nights booked
every 24 hours

11. Agenda.
Source: Kleiner Perkins 2018 Internet trends: https://www.statista.com/statistics/277483/market-value-of-the-largest-internet-companies-worldwide/

12. New
Product
Development.
Data.
Analyze.
Hypothesize.
Experiment.
The Data-Driven Decision Cycle.

13. 1. Booking.com, nice to meet you
2. Data science @ Booking.com
3. Experimentation @ Booking.com
4. Solving customer needs

14. Data Science at Booking.com

17. relocations!

21. Data Scientists...
● Diagnose problems
● Identify new hypotheses to test
● Create metrics
● Enable experimentation

22. 1. Booking.com, nice to meet you
2. Data science @ Booking.com
3. Experimentation @ Booking.com
4. Solving customer needs

23. Why?

24. 9/10 experiments fail.
90% of product decisions have an
inconclusive or negative effect on
a product’s primary metric
Source: VWO 2016.
(8.75/10 tests fails according to AppSumo)
90%
10%

25. Evercore Equity Research
https://goo.gl/rFP
“Booking’s utilization of A/B testing has
contributed to Booking enjoying
conversion rates 2-3x higher than the
industry average.”

26. Product Death Cycle.

29. Experimentation at Booking.com

30. Base.
Which one performed better?
Variant.

31. Which one performed better?

32. Base.
Which one performed better?
Variant.

33. Hypothesize &
Experiment.
• Hypothesize
• Calculate power
• Run for prescribed time
• Stop, rerun or go live
• Iterate
Base.
Variant.
Search
Search

34. Experiment on everything.

35. Experiment atomically.

36. Base. Variant.
Test atomically.

37. Interactions.
Experiment on
Everything
Test Atomically
A lot of
experiments!
+ =

38. 1,000+
concurrent tests.

39. Based on [prior] we believe that
changing [condition]
for users [sample] will make
them [outcome].
We will know this when we see
[effect(s)] happen to [metric(s)].
This will be good for customers,
partners and our business
because [motivation].
Idea ≠ Hypothesis.
Hypothesis Template.

40. Idea ≠ Hypothesis.
• All secondary metrics
selected a priori along with
their direction of change to
avoid Texas Sharpshooter
Fallacy

41. ● All experiments should have
run time calculated
beforehand using the power
calculator of your choice.
Or use ours :-)
https://bookingcom.github.io/
powercalculator/
● Experiments should then be
run for the next closest full
week cycle.
Properly Powered.
Full Week Cycles.
Experiments should:

42. Experimentation culture

43. No HIPPOs.

44. Teams made for
experimentation.

45. Allow 100% access to
data.

46. Guidelines, not rules.

47. Embrace failure.

48. Fail fast.

49. Don’t assume
reproducibility.

50. Winterberg

51. Go where the experiment takes you

52. Thank you.
https://blog.booking.com/