Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

A/B Testing

415 views

Published on

The presentation for students about a/b testing process, its advantages, best practices and common (and not that common) pitfalls.

Published in: Data & Analytics
  • D0WNL0AD FULL ▶ ▶ ▶ ▶ http://1lite.top/Q8WHe ◀ ◀ ◀ ◀
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

A/B Testing

  1. 1. A/B Testing Ivan Mylyanyk @ Booking.com
  2. 2. Prologue
  3. 3. Mylyanyk Ivan Lviv Physics and Mathematics lyceum Ivan Franko National University of L’viv Freelance Outsource Startup Booking.com
  4. 4. Booking.com planet’s #1 accommodation website HQ @ Amsterdam 10K+ employees 1M+ open properties 1M+ room nights / day
  5. 5. What is A/B Testing?
  6. 6. What you can test? ● color for your button ● title for your announcement ● new data science model ● algorithms ● fonts ● new functionality ● different photo resolutions ● email campaigns ● many more...
  7. 7. What can be your metric? ● #pageviews ● #signups ● profit ● something very specific: ○ number of uploaded profile pics
  8. 8. Do I need A/B Testing? ● customers “decide” what stays ● opinions and assumptions aren’t involved* ● easy to measure the business impact ● you have a shelter when something is failing* ● great way to learn your customers ○ not all experiments are charged for succeed ● general product deployment is easy easier *almost
  9. 9. Trade offs ● the “space” is limited ○ test one idea at time ■ not really ● the codebase can really get overcomplicated ● things change over time ● ethical part: experiments on people?!
  10. 10. Experiment workflow
  11. 11. Most important steps of the experiment 1. Problem 2. Hypothesis 3. Prediction 4. Experiment 5. Analysis 6. Decision
  12. 12. Problem
  13. 13. Hypothesis
  14. 14. Prediction
  15. 15. Experiment
  16. 16. Analysis
  17. 17. Decision
  18. 18. Why experiments fail?
  19. 19. Technical issues ● bugs ● increase in the page load time ● check different browsers ● mobile?!
  20. 20. Keep context in mind ● something became less notable ● where did you get space for your feature? ● [design] does it fit the overall experience?
  21. 21. Audience Do we show changes to the right audience? … but really, do we show it to the right audience?
  22. 22. Low traffic Did you collect enough of data? https://www.qualtrics.com/blog/determining-sample-size/
  23. 23. Exercise time!
  24. 24. New block on the page
  25. 25. What can go wrong?
  26. 26. Dart Vader paradox (c)
  27. 27. Experiments with translations
  28. 28. Summary
  29. 29. Should I use A/B testing? opinions and assumptions aren’t involved* easy to measure the business impact great way to learn your customers and their preferences not all experiments are charged for succeed general product deployment is easy easier takes some time things change over time ethical questions a lot of pitfalls complicated code on scale
  30. 30. Why is nobody clicking that button? it’s the wrong color it’s not big enough it says “buy now” not “add to cart” it should be round it should be flat Your product is ...less than perfect Nobody know what your product does Your product is overpriced Your website doesn’t look professional Your website is slow You charge too much Your product is out of season Source: http://www.slideshare.net/hayluke/ab-testing-uxcamp
  31. 31. Fancy stuff
  32. 32. Experiment game: http://lukasvermeer.github.io/confidence/ Booking.com AI game: https://workingatbooking.com/blog/hackman/ Careers: https://workingatbooking.com/
  33. 33. Thank you for your attention! Q&A time! Ivan Mylyanyk Blog @ Medium Social networks

×