Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

A/B testing of geodata in booking.com

870 views

Published on

I presented this talk on the ItNonStop conference on November 18th, 2017 in Wrocław (https://it-nonstop.net/city/wroclaw).
I explained what A/B testing is and how do we in Booking.com apply it to our location-related information, such as location of museums, landmarks, beach, shopping malls, or historical areas.

Published in: Technology
  • Be the first to comment

A/B testing of geodata in booking.com

  1. 1. A/B Testing with Geodata in Booking.com Elena Nikolaeva | November 2017 @ice_lenorice.lenor@gmail.com | smartpuffin.com |
  2. 2. A/B testing What is it?
  3. 3. Which button color is the best? OR ?
  4. 4. Let the users decide!
  5. 5. Experiment flow. 50% 50% 10% conversion 20% conversion Win! Lose.
  6. 6. 10% conversion 20% conversion What does “the best” mean? 50% 50% Win! Lose.
  7. 7. Metrics: ● Conversion ● Click-through rate ● Log-ins ● Bounce rate ... anything your business wants the customers to do!
  8. 8. Additional metrics: ● Clicked on an element ● Clicked on a different element ● Spent time on the page ● Scrolled down Helps to learn about user behavior.
  9. 9. Building a hypothesis
  10. 10. “If I implement X, the users will do Y, and it’s good because Z”
  11. 11. “If I... add a cat to my presentation, the listeners will... say AAAaaw, and it’s good because... they will remember my talk”
  12. 12. Success: Congratulate yourself! Continue on topic. Where else can you apply the same approach?
  13. 13. Failure: That’s unfortunate. But don’t give up just yet! Analyze what went wrong.
  14. 14. Repeat.
  15. 15. RepeatAnalysisExperimentMetrics Steps to a successful experiment: Hypothesis
  16. 16. Learning is the most important thing
  17. 17. A/B testing: not only button colors
  18. 18. What is geodata?
  19. 19. 80% of users think location is very important when choosing an accommodation.
  20. 20. How do we run A/B tests on geodata?
  21. 21. Step 1. Build a hypothesis
  22. 22. User research
  23. 23. Street testing
  24. 24. Surveys
  25. 25. Data analysis
  26. 26. “I want to stay in the historical center of the city, but I don’t know where it is.”
  27. 27. 2. Implement the experiment.
  28. 28. Brainstorm
  29. 29. Our Geo Team
  30. 30. Data collection process. MVP approach: collect a small piece of data to experiment with. Prioritize collection!
  31. 31. We collected the data for 30 cities where our customers left reviews mentioning ”Old Town” the most Data collection process.
  32. 32. Lisbon Old town
  33. 33. Old Town
  34. 34. 6. Repeat.
  35. 35. B.old Town
  36. 36. Now grow the data! We collected 100 more old towns! Success?
  37. 37. Our challenges
  38. 38. Defining customer problems is hard.
  39. 39. Many places.
  40. 40. Many places.
  41. 41. Very location-specific.
  42. 42. Thank you! Elena Nikolaeva | November 2017 @ice_lenorice.lenor@gmail.com | smartpuffin.com |

×