Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

LITE 2017 – Online Experimentation [Thom Shurrock]

42 views

Published on

Published in: Software
  • Be the first to comment

  • Be the first to like this

LITE 2017 – Online Experimentation [Thom Shurrock]

  1. 1. Online experimentation
  2. 2. ● Introductions ● What is online experimentation ● Methodology ● Metrics ● Showcase of wins & losses ● Q&A ● My email - tshurrock@opentable.com Agenda Online experimentation
  3. 3. 3 Introductions
  4. 4. ● First launch in 1998 ● San Francisco HQ ● We have both restaurant facing and customer facing products ● Acquired by The Priceline Group in 2014 ● 40,000 restaurants worldwide ● 21 Million diners seated a month ● 850,000 reviews / month from our customers About Opentable INTRODUCTIONS
  5. 5. ● Senior Product Manager at Opentable ● Focused on international expansion ● Development background ● Agency side and client side experience About me INTRODUCTIONS
  6. 6. CONFIDENTIAL We want employees who continually challenge themselves to look at things differently, seek a better way, and know they have the freedom to do so. Overview
  7. 7. What is online experimentation?
  8. 8. CONFIDENTIAL An experiment is a controlled study used to test a hypothesis Experimentation helps us understand the impact of our ideas Overview A hypothesis is a proposed explanation made on the basis of limited evidence as a starting point for further investigation. In controlled study, subjects are assigned randomly to different treatment groups, if we see meaningful differences in the outcomes, then we can conclude that the treatment caused the difference
  9. 9. CONFIDENTIAL Overview
  10. 10. Experimentation methodology
  11. 11. 1. Science! 2. Hypothesise! 3. Minimum run time 4. Statistical significance 5. Validate your experiment 6. Learnings DO.. .
  12. 12. 1. Stop the test mid 7 day cycle 2. Pause the test early because of a negative result 3. Judge your test too early 4. Leave it running because it’s ‘almost there’ 5. Massage the results (aka the Texas sharpshooter fallacy) DO NOT!
  13. 13. Experimentation is not just UI changes ● Meta analysis completed by Qubit Digital on over 6,700 experiments ● Cosmetic changes have far smaller impact on revenue than treatments grounded in behavioural psychology. Experimentation methodology Treatment Avg Uplift Uplift probability Scarcity 2.9% 84% Social proof 2.3% 82% Urgency 1.5% 70%
  14. 14. Experimentation metrics
  15. 15. 140-200 A/B tests launched per month
  16. 16. KEY METRICS ● #hashtags are used to categorise experiments ● Allows for the easy reporting on experiment type ● #socialproof / #global / #UX are the biggest winners for us ● #scarcity is one of the lowest Use of hashtags
  17. 17. KEY METRICS Monthly tests launched
  18. 18. Showcasing of A/B testing wins
  19. 19. #hashtags #conversion #socialproof #urgency #yield #frequency #crosssell
  20. 20. #conversion
  21. 21. 21 ● Machine learning search results ● Positive result ● Ran for 14 days ● 99% confidence ● Using restaurant attributes to create quality score and rank accordingly ● Search count, average rating, rating count, reservation count, price, distance from city centre #CONVERSION
  22. 22. 22 ● Free text search ● Positive result ● 98% significant ● 7 days ● Great insight into what customers are searching for #CONVERSION
  23. 23. #socialproof
  24. 24. ● Social Proof in rows for dine again on the metro homepage ● Positive improvement ● Ran for 14 days ● 92% confidence #SOCIALPROOF
  25. 25. 25 ● Showing top dish tags and top restaurant tags above the fold on a restaurant profile page ● Positive improvement ● Information collected while customers leave reviews of the restaurant #SOCIALPROOF
  26. 26. 26 ● Showing instagram photos on restaurant profile pages ● Positive improvement ● Internal tool for getting images ● Moderation feature ● Dynamically moderate photos soon! #SOCIALPROOF
  27. 27. #urgency
  28. 28. #URGENCY ● Changing countdown timer text to ‘Hurry! We can only hold this table for you for XX:XX’ ● Positive improvement ● 95% confidence level ● Other than this, limited wins with urgency
  29. 29. #frequency
  30. 30. 30 ● Adding how many points customer have received when they made a booking to the past reservations tab ● Positive improvement ● Ran for 3 weeks (due to low traffic numbers) ● Increased repeat booking rate #FREQUENCY
  31. 31. #crosssell
  32. 32. 32 ● Restaurant profile page when there is no availability for the timeslot searched for (including 2.5 hours either side) ● Positive improvement ● Ran for 2 months (to get big enough sample pool) ● 97% significant #CROSS-SELL
  33. 33. 33 ● A lot of our customers land on restaurant profile pages from Google. ● Adding ‘Find Nearby Restaurants’ button on the bottom of the profile page ● Positive improvement #CROSS-SELL
  34. 34. Losses (#fail)
  35. 35. 35 #FAIL ● More timeslots do not mean more bookings ● Paradox of choice! ● Negative impact ● 99% significant
  36. 36. 36 #FAIL ● Showing images higher up the page does not help the customer make a decision quicker ● Negative impact ● 99% significant
  37. 37. Further reading
  38. 38. ● Influence: The Psychology of Persuasion - Robert Cialdini ● Pre-Suasion: A Revolutionary Way to Influence and Persuade - Robert Cialdini ● What works in e-commerce - a meta- analysis of 6700 online experiments - http://www.qubit.com Things to read Further reference reading list
  39. 39. ● You now (hopefully) have a good overview of how Opentable executes online experimentation ● My email - tshurrock@opentable.com ● Twitter - @shurrock Recap for the session A/B TESTING APPROACH AND LEARNINGS
  40. 40. Questions?

×