Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Conversion 2016 - Martijn Scheijbeler - What we learned from running 200 tests a year?

332 views

Published on

What we learned from running 200 tests a year?

Published in: Marketing
  • D0WNL0AD FULL ▶ ▶ ▶ ▶ http://1lite.top/01qJCT ◀ ◀ ◀ ◀
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Be the first to like this

Conversion 2016 - Martijn Scheijbeler - What we learned from running 200 tests a year?

  1. 1. What we learned from running 200 tests a year? Martijn Scheijbeler / Director of Marketing / martijn@thenextweb.com / @MartijnSch
  2. 2. Emerce Conversion - @MartijnSch Title
  3. 3. Martijn Scheijbeler Marketing Director at The Next Web Former Lead SEO & Analytics martijn@thenextweb.com @MartijnSch
  4. 4. Process PROCESS Learnings LEARNINGS
  5. 5. How it all
 Started @MartijnSch THE START
  6. 6. Why we feel
 The need for testing? @MartijnSch
  7. 7. “So what would we have to do to grow x10?” Boris (CEO) @MartijnSch “I have no idea what I’m doing”
  8. 8. INDEX 20,000 140,000 6,000 visitors startups listed square meters 7,500,000 unique visitors INDEX.CO Conference
  9. 9. Why am I showing you this slide? @MartijnSch
  10. 10. MEDIA News E-commerce EVENTS Events Conferences WORK SPACES TQ INTELLIGENCE Index
  11. 11. But how did we get to
 200 A/B Tests a Year? @MartijnSch
  12. 12. Testing Figures Data from 2015 2,5CRO Team in 2015 38.000.000Sessions with Tests 0CRO Team in 2014 5-7Tests per week PricelessAmounts of fun 35%Traffic v.s. Tests @MartijnSch
  13. 13. Quantity v.s. Quality? @MartijnSch
  14. 14. “If you double the number of experiments you’re going to double your inventiveness” - Jeff Bezos @MartijnSch
  15. 15. What makes it different: Publishers v.s. E-commerce? @MartijnSch
  16. 16. The opportunity is 350-400 tests a year. As ROI per test is low we need to run a lot to have a positive return. Efficiency <> Costs @MartijnSch
  17. 17. How is our (marketing) team
 Structured? @MartijnSch
  18. 18. SEO CRO/ANALYTICS TNW DEALS INDEX.CO CONTENT GROWTH @MartijnSch Front-end Developers
  19. 19. “Better ask for forgiveness
 than permission”- Boris Veldhuijzen van Zanten @MartijnSch
  20. 20. F*CK it, we’ll do it live It’s not rocket science “Innovate through in-house solutions” @MartijnSch
  21. 21. We failed, not once but also not twice… @MartijnSch
  22. 22. SO WE STARTED TESTING WITH GOOGLE ANALYTICS CONTENT EXPERIMENTS GOOGLE TAG MANAGER IS MAKING OUR LIVES EASIER + SCALABLE
  23. 23. WE DIDN’T COMMUNICATE ENOUGH, WE SHOULD HAVE! MOVING MOST OF OUR COMMUNICATION TO SLACK
  24. 24. What’s our
 Process @MartijnSch PROCESS
  25. 25. We needed to learn,
 Knowledge building? @MartijnSch
  26. 26. Thanks for all your support! We learn from you!
  27. 27. FIND THE PERFECT PROCESS TO FIX EVERYTHING > ALWAYS WHY? MORE EFFICIENCY: THE MORE TESTS, THE HIGHER THE INVENTIVENESS
  28. 28. What are (y)our Problems? @MartijnSch
  29. 29. “We can’t create any content” “No management approval” “Whatever b*llshit reason” @MartijnSch
  30. 30. “Our legal department is your
 Editorial team”- Google Employee @MartijnSch
  31. 31. How did we overcome our fails? @MartijnSch
  32. 32. We didn’t move to any ‘real’ CRO tool but decided to iterate on our own: GTM <> JS. GTM @MartijnSch Download: Github.com/MartijnSch/CRO
  33. 33. Building our own solutions for saving testing documentation + analysis of tests. Innovate @MartijnSch
  34. 34. We moved most of our communication towards Slack. Making the teams more involved in the whole process. Always communicate @MartijnSch
  35. 35. We hired two front-end developers, to make sure in the long-run we can test bigger things. Front-end Dev @MartijnSch
  36. 36. Improvements 5-7Tests per week 2-4Tests per week 30-40Tests per year 20%More efficient @MartijnSch
  37. 37. What’s in our Future? @MartijnSch
  38. 38. CONTINUOUS IMPROVEMENT WHO’S ABLE TO DEPLOY WINNING VARIANTS WITHIN HOURS?
  39. 39. MOVING TESTING TO OUR BACK-END COLLECTING DATA + SEGMENTATION MORE EASILY + LESS FLICKERING
  40. 40. GO BIG, BIGGER, BIGGEST! TESTING SMALL DOESN’T WORK (THAT WELL), GO BIGGER!
  41. 41. TLDR: Process 1. Have the right company vision 2. Have the right team structure 3. Have access to top level 4. Create partnerships with like minded 5. You’ll have trouble as well, nobody’s perfect 6. Don’t be afraid to invent the wheel again 7. Always focus on innovation @MartijnSch
  42. 42. What tests
 did we learn from @MartijnSch LEARNINGS
  43. 43. MEDIA News eCommerce EVENTS Events Conferences WORK SPACES TQ INTELLIGENCE Index
  44. 44. Sharing @MartijnSch
  45. 45. Sharing brings in more loyalty, more engagement and hopefully more traffic. Share buttons
  46. 46. • Variant increase clicks with 16% • Had a significance + power level of > 99% • We ran the test for 1x 7+ days • Targetting over 200.000 users Winner: Variant Original: Variant
  47. 47. Engagement @MartijnSch
  48. 48. Engagement is important, focus on getting more clicks on the related stories in the sidebar. Rankings!
  49. 49. • Variant increase clicks with 36% • Had a significance + power level of > 99% • We ran the test for 7+ days • Targetting over 300.000 users Winner: Variant Original: Variant:
  50. 50. Engagement is important, focus on getting more clicks on the related stories below the article. Related Stories
  51. 51. • Variant increase clicks with 36% • Had a significance + power level of > 99% • We ran the test for 2x 14+ days • Targetting over 500.000 users Winner: Variant Original:Variant
  52. 52. So what if we remove the whole sidebar, wouldn’t people be more focused on the stuff that is out there? Big, bigger
  53. 53. • Variant decreased clicks with 17% + 14% • Had a significance + power level of > 99% • We ran the test for 14+ days • Targetting over 420.000 users Winner: Original
  54. 54. TLDR: Test learnings 1. Know what you’re testing 2. Focus on your user segments 3. Iterate on what you’re already testing 4. Focus on high significance 5. Test big, bigger, biggest! @MartijnSch
  55. 55. Now you know how to run
 200 A/B Tests a Year? @MartijnSch
  56. 56. But did we get to
 200 tests? @MartijnSch
  57. 57. “Why didn’t we run 200 tests?” “What went wrong?” “Do I need to start crying?” Boris (CEO)
  58. 58. “Well, if this guy wouldn’t have stuck his fingers into a lawn mower, YES!” Missing out on ±20 tests in 2015. The Lawn Mower
  59. 59. Take Aways: 1. Focus on running X tests a year 2. Improve the process 3. Have the right (free) tools 4. Have the right people 5. Document/Communicate everything 6. Know what the future looks like 7. Have fun! @MartijnSch
  60. 60. Emerce Conversion - @MartijnSch Title
  61. 61. See you at Conference?
  62. 62. Thanks! Have fun! @MartijnSch
  63. 63. LET’S HAVE BEER AT THE ‘BORREL’ OR COFFEE IN THE NEAR FUTURE!
  64. 64. Do you have any Questions? @MartijnSch

×