Ton Wesseling / @tonw Keynote - My Daily Job as a growth hacker / conversion rate optmizers

950 views

Published on

Ton Wesseling / @tonw his Keynote at Elite Camp Estonia #elitecamp June 14th 2014 - A/B-testing hands on - optimizing a bookingsengine.

Published in: Technology, Business
0 Comments
3 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
950
On SlideShare
0
From Embeds
0
Number of Embeds
77
Actions
Shares
0
Downloads
29
Comments
0
Likes
3
Embeds 0
No embeds

No notes for slide

Ton Wesseling / @tonw Keynote - My Daily Job as a growth hacker / conversion rate optmizers

  1. 1. ONLINE D IALOGU E G u i d i n g y o u t o d i g i t a l m a t u r i t y To n W e s s e l i n g – G r o w t h H a c k e r # e l i t e c a m p E s t o n i a 2 0 1 4 / 0 6 / 1 4 My D aily Job
  2. 2. Growth magic Data Driven Persuasion Psychology +
  3. 3. Email: ton@onlinedialogue.com onlinedialogue.com Twitter: @TonW #elitecamp 3 things ü Learn to code ü Rethink your optimization strategy ü Make sure everyone knows the drill
  4. 4. Email: ton@onlinedialogue.com onlinedialogue.com Twitter: @TonW #elitecamp Digital Elite Camp
  5. 5. Email: ton@onlinedialogue.com onlinedialogue.com Twitter: @TonW #elitecamp Friends
  6. 6. Email: ton@onlinedialogue.com onlinedialogue.com Twitter: @TonW #elitecamp It’s a social thing
  7. 7. Email: ton@onlinedialogue.com onlinedialogue.com Twitter: @TonW #elitecamp It’s a social thing 5PA1N
  8. 8. Email: ton@onlinedialogue.com onlinedialogue.com Twitter: @TonW #elitecamp Amazing
  9. 9. Email: ton@onlinedialogue.com onlinedialogue.com Twitter: @TonW #elitecamp Home
  10. 10. Email: ton@onlinedialogue.com onlinedialogue.com Twitter: @TonW #elitecamp About me
  11. 11. Email: ton@onlinedialogue.com onlinedialogue.com Twitter: @TonW #elitecamp As a kid
  12. 12. Email: ton@onlinedialogue.com onlinedialogue.com Twitter: @TonW #elitecamp Your lucky number 3
  13. 13. Email: ton@onlinedialogue.com onlinedialogue.com Twitter: @TonW #elitecamp But also 4
  14. 14. Email: ton@onlinedialogue.com onlinedialogue.com Twitter: @TonW #elitecamp So why not 34
  15. 15. Email: ton@onlinedialogue.com onlinedialogue.com Twitter: @TonW #elitecamp 34 years ago
  16. 16. Email: ton@onlinedialogue.com onlinedialogue.com Twitter: @TonW #elitecamp Sailing
  17. 17. Email: ton@onlinedialogue.com onlinedialogue.com Twitter: @TonW #elitecamp Sailing a very close operating team or just you
  18. 18. Email: ton@onlinedialogue.com onlinedialogue.com Twitter: @TonW #elitecamp Winning some big decisions and tweaking the details
  19. 19. Email: ton@onlinedialogue.com onlinedialogue.com Twitter: @TonW #elitecamp That’s our business #CRO #elitecamp @TonW
  20. 20. Email: ton@onlinedialogue.com onlinedialogue.com Twitter: @TonW #elitecamp Thanks to this man
  21. 21. Email: ton@onlinedialogue.com onlinedialogue.com Twitter: @TonW #elitecamp Quality circle
  22. 22. Email: ton@onlinedialogue.com onlinedialogue.com Twitter: @TonW #elitecamp Deming said
  23. 23. Email: ton@onlinedialogue.com onlinedialogue.com Twitter: @TonW #elitecamp Focus on quality “When you focus just on costs they will rise”
  24. 24. Email: ton@onlinedialogue.com onlinedialogue.com Twitter: @TonW #elitecamp Testing before rebuilding
  25. 25. Email: ton@onlinedialogue.com onlinedialogue.com Twitter: @TonW #elitecamp Back to reality
  26. 26. Email: ton@onlinedialogue.com onlinedialogue.com Twitter: @TonW #elitecamp Van der Valk
  27. 27. Email: ton@onlinedialogue.com onlinedialogue.com Twitter: @TonW #elitecamp Van der Valk
  28. 28. Email: ton@onlinedialogue.com onlinedialogue.com Twitter: @TonW #elitecamp Van der Valk
  29. 29. Email: ton@onlinedialogue.com onlinedialogue.com Twitter: @TonW #elitecamp Van der Valk ü Unique website for each hotel ü 1 centralized team ü Paid by % of their turnover
  30. 30. Email: ton@onlinedialogue.com onlinedialogue.com Twitter: @TonW #elitecamp Online Dialogue ü Analytics & optimization partner They also have: Development partner Advertising partner
  31. 31. Email: ton@onlinedialogue.com onlinedialogue.com Twitter: @TonW #elitecamp My 3 focus points ü Optimize new bookingsengine ü Testing software costs too high ü Let them choose for Valk
  32. 32. Email: ton@onlinedialogue.com onlinedialogue.com Twitter: @TonW #elitecamp Step 1: data & control ü New Google Analytics main account ü Google Tag Manager
  33. 33. Email: ton@onlinedialogue.com onlinedialogue.com Twitter: @TonW #elitecamp Google tag manager & testing tools VWO GTM
  34. 34. Email: ton@onlinedialogue.com onlinedialogue.com Twitter: @TonW #elitecamp Data diving ü Small % will visit the (rooms) bookingsengine ü They navigate through the room pages
  35. 35. Email: ton@onlinedialogue.com onlinedialogue.com Twitter: @TonW #elitecamp Van der Valk
  36. 36. Email: ton@onlinedialogue.com onlinedialogue.com Twitter: @TonW #elitecamp Van der Valk
  37. 37. Email: ton@onlinedialogue.com onlinedialogue.com Twitter: @TonW #elitecamp My 3 focus points ü Optimize new bookingsengine ü Testing software costs too high ü Let them choose for Valk
  38. 38. Email: ton@onlinedialogue.com onlinedialogue.com Twitter: @TonW #elitecamp My 3 focus points ü Optimize new bookingsengine ü Learn to understand behaviour ü Let them choose for Valk
  39. 39. Email: ton@onlinedialogue.com onlinedialogue.com Twitter: @TonW #elitecamp G+ as virtual team communication
  40. 40. Email: ton@onlinedialogue.com onlinedialogue.com Twitter: @TonW #elitecamp Online Dialogue Yammer
  41. 41. Email: ton@onlinedialogue.com onlinedialogue.com Twitter: @TonW #elitecamp Bookingsengine optimization
  42. 42. Email: ton@onlinedialogue.com onlinedialogue.com Twitter: @TonW #elitecamp But first: statistics - history ü 4 out of 10 tests were successful ü With a 90% significance level ü Purchase cycle = 3 days ü Thousands room bookings per week
  43. 43. Email: ton@onlinedialogue.com onlinedialogue.com Twitter: @TonW #elitecamp Power & Significance ü 10 wrong winners every 100 tests ü Only +-65% of these winners were uncovered ü 40 winners = 30 real winners, so ü 45 winning ideas, 55 losing ideas ü Lucky: 75% of the winners is real
  44. 44. Email: ton@onlinedialogue.com onlinedialogue.com Twitter: @TonW #elitecamp Cookie deletion & cross device ü 90% significance is ok ü We can stay at this low power level ü 2 weeks max test-time in this case (to reduce sample pollution) ü Only test 1 type of device at the same time
  45. 45. Email: ton@onlinedialogue.com onlinedialogue.com Twitter: @TonW #elitecamp Ever been to Vegas?
  46. 46. Email: ton@onlinedialogue.com onlinedialogue.com Twitter: @TonW #elitecamp Automate the non learning stuff!
  47. 47. Email: ton@onlinedialogue.com onlinedialogue.com Twitter: @TonW #elitecamp Testing costs time Use it to LEARN
  48. 48. Email: ton@onlinedialogue.com onlinedialogue.com Twitter: @TonW #elitecamp Learn ü If a new functionality should be build ü What message really impacts behavior ü What we should feed the bandit Always Re-test before you decide on these 2:
  49. 49. Email: ton@onlinedialogue.com onlinedialogue.com Twitter: @TonW #elitecamp Pretty high “click back” numbers
  50. 50. Email: ton@onlinedialogue.com onlinedialogue.com Twitter: @TonW #elitecamp No room available
  51. 51. Email: ton@onlinedialogue.com onlinedialogue.com Twitter: @TonW #elitecamp Just measure it
  52. 52. Email: ton@onlinedialogue.com onlinedialogue.com Twitter: @TonW #elitecamp No room available §  15% sees a no room available page §  Conversion rate of them is 50% lower §  Low number of conversions per week §  2 weeks testing – low power warning §  Pontentially high impact
  53. 53. Email: ton@onlinedialogue.com onlinedialogue.com Twitter: @TonW #elitecamp Variation
  54. 54. Email: ton@onlinedialogue.com onlinedialogue.com Twitter: @TonW #elitecamp Test protocol ü Put the control live a week before test ü Always use a “copy control” as control ü After the test period – new traffic to control – for again 1 week (should be: 1 purchase cycle minimum)
  55. 55. Email: ton@onlinedialogue.com onlinedialogue.com Twitter: @TonW #elitecamp Like this
  56. 56. Email: ton@onlinedialogue.com onlinedialogue.com Twitter: @TonW #elitecamp Results ü No significant uplift ü Only 30% clicks ü 55% mouse overs ü The others bounce very fast from the page ü Without them: significant uplift!
  57. 57. Email: ton@onlinedialogue.com onlinedialogue.com Twitter: @TonW #elitecamp Actions ü Too little value for a re-test / new test ü Functionality: show other rooms ü Test with the dates on the widget
  58. 58. Email: ton@onlinedialogue.com onlinedialogue.com Twitter: @TonW #elitecamp Booking.com - scarcity
  59. 59. Email: ton@onlinedialogue.com onlinedialogue.com Twitter: @TonW #elitecamp First – let’s measure
  60. 60. Email: ton@onlinedialogue.com onlinedialogue.com Twitter: @TonW #elitecamp Scarcity data §  30% sees rooms with scarcity §  Conversion rate of them is the same §  A lot of conversions per week §  2 weeks testing – pretty OK power
  61. 61. Email: ton@onlinedialogue.com onlinedialogue.com Twitter: @TonW #elitecamp The code
  62. 62. Email: ton@onlinedialogue.com onlinedialogue.com Twitter: @TonW #elitecamp The results Significant uplift
  63. 63. Email: ton@onlinedialogue.com onlinedialogue.com Twitter: @TonW #elitecamp YES! Re-test
  64. 64. Email: ton@onlinedialogue.com onlinedialogue.com Twitter: @TonW #elitecamp Re-test +
  65. 65. Email: ton@onlinedialogue.com onlinedialogue.com Twitter: @TonW #elitecamp It’s so easy to code
  66. 66. Email: ton@onlinedialogue.com onlinedialogue.com Twitter: @TonW #elitecamp The results ü Yes, significant result again ü The other 2 messages are also significant ü But : the combination is even significant better than the scarcity version itself!
  67. 67. Email: ton@onlinedialogue.com onlinedialogue.com Twitter: @TonW #elitecamp Actions ü Build scarcity in the CMS system ü Re-test the combination of messaging ü Keep an eye on customer life time value development from new guests through scarcity
  68. 68. Email: ton@onlinedialogue.com onlinedialogue.com Twitter: @TonW #elitecamp Bonus Test story
  69. 69. Email: ton@onlinedialogue.com onlinedialogue.com Twitter: @TonW #elitecamp Bonus Test story
  70. 70. Email: ton@onlinedialogue.com onlinedialogue.com Twitter: @TonW #elitecamp Results Significant worse
  71. 71. Email: ton@onlinedialogue.com onlinedialogue.com Twitter: @TonW #elitecamp Follow up test
  72. 72. Email: ton@onlinedialogue.com onlinedialogue.com Twitter: @TonW #elitecamp Follow up test
  73. 73. Email: ton@onlinedialogue.com onlinedialogue.com Twitter: @TonW #elitecamp Follow up test
  74. 74. Email: ton@onlinedialogue.com onlinedialogue.com Twitter: @TonW #elitecamp Suddenly
  75. 75. Email: ton@onlinedialogue.com onlinedialogue.com Twitter: @TonW #elitecamp My 3 focus points ü Optimize new bookingsengine ü Learn to understand behaviour ü Let them choose for Valk
  76. 76. Email: ton@onlinedialogue.com onlinedialogue.com Twitter: @TonW #elitecamp My 3 focus points ü Test, learn and build proces ü Learn to understand behaviour ü Let them choose for Valk
  77. 77. Email: ton@onlinedialogue.com onlinedialogue.com Twitter: @TonW #elitecamp A/B-testguide ü Team communication ü Use the right aproach ü Store your knowledge
  78. 78. Email: ton@onlinedialogue.com onlinedialogue.com Twitter: @TonW #elitecamp So – the 3 things you should remember ü Learn to code ü Rethink your optimization strategy ü Make sure everyone knows the drill
  79. 79. Email: ton@onlinedialogue.com onlinedialogue.com Twitter: @TonW #elitecamp Test waiting
  80. 80. Email: ton@onlinedialogue.com onlinedialogue.com Twitter: @TonW #elitecamp Test waiting
  81. 81. Email: ton@onlinedialogue.com onlinedialogue.com Twitter: @TonW #elitecamp Test waiting
  82. 82. ONLINE D IALOGU E G u i d i n g y o u t o d i g i t a l m a t u r i t y To n W e s s e l i n g – G r o w t h H a c k e r # e l i t e c a m p E s t o n i a 2 0 1 4 / 0 6 / 1 4 My D aily Job

×