Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

How a year of SEO split testing changed how I thought SEO worked

733 views

Published on

I spent a year running all the split tests from DistilledODN, a split testing platform. Here's how an entire year of testing changed how I thought and worked.

Published in: Marketing

How a year of SEO split testing changed how I thought SEO worked

  1. 1. HOW A YEAR OF SEO SPLIT TESTING CHANGED HOW I THOUGHT SEO WORKED DOMINIC WOODMAN
  2. 2. Hmm that did nothing Crushing failure Ooh traffic went up
  3. 3. All those links I’m building.
  4. 4. Hmm that did nothing Crushing failure Ooh traffic went up
  5. 5. How does SEO testing work & what do you need?
  6. 6. Getting started & learning how to think about testing
  7. 7. More than you wanted to know about titles and metas.
  8. 8. Testing resource heavy investments
  9. 9. Structured data & dangerous assumptions
  10. 10. Risky business...
  11. 11. How testing changes relationships
  12. 12. Conclusions
  13. 13. How does SEO testing work & what do you need?
  14. 14. SEO split testing is template basedHere’s our example website Home Animals Cats Dogs Unicorns Badgers
  15. 15. CatsCats We want to test two templates for our animals pages
  16. 16. Cats Dogs Unicorns Badgers CRO test - Different users, see different templates on the same pages 1 2 Cats BadgersDogs Unicorns
  17. 17. Cats Dogs Unicorns Badgers SEO test - different users see the same templates on different pages 1 2 Cats Dogs Unicorns Badgers Cats Dogs Unicorns Badgers
  18. 18. Then we measure the change in traffic, e.g. 3% daily increase
  19. 19. LIABILITY
  20. 20. Rank Search visibility
  21. 21. Total additional sessions
  22. 22. The fans show margin of error
  23. 23. The test is significant when the shaded area crosses 0
  24. 24. In this case it happened here:
  25. 25. There are two graphs.
  26. 26. Control Variant
  27. 27. Black is organic traffic for the variant
  28. 28. Start date for the test
  29. 29. Take all the data to left and use that to make the model
  30. 30. The model of the variant is the blue line
  31. 31. We overlay them
  32. 32. Compare the model to the variant after the start date
  33. 33. In this case the black line is higher, it’s a positive test
  34. 34. A good model, fits with the variant before the start date
  35. 35. However this was my model. They don’t fit well.
  36. 36. How much traffic do you need for SEO split testing? Rule of thumb: you need roughly 1000 organic sessions a day to the group of pages you’re testing. #INBOUND2018 @dom_woodman
  37. 37. That model you just saw, only had ~60 sessions a day.
  38. 38. Getting started & learning how to think about testing
  39. 39. 1
  40. 40. Some fluctuation but null.
  41. 41. 2
  42. 42. After
  43. 43. 10.8% decrease
  44. 44. 3
  45. 45. <img src="product1.jpg" alt="Home 6 Piece Towel Bale - Geo Grey / Zest">
  46. 46. A model so perfectly null I thought I had forgotten to launch it
  47. 47. 4&5
  48. 48. Thought leader. PHD in AI. Unbearably smug.
  49. 49. Before:
  50. 50. After:
  51. 51. A stunning 27% drop.
  52. 52. A 20% drop.
  53. 53. Can’t write titles to save his life.
  54. 54. Get a testing framework You need to be able to iterate and learn from tests. Having a shared framework, will help with this. #INBOUND2018 @dom_woodman
  55. 55. You don’t need the why Knowing the why is great (find it if you can), but you don’t need it to take action. #INBOUND2018 @dom_woodman
  56. 56. More than you wanted to know about titles and metas.
  57. 57. 6
  58. 58. 16
  59. 59. 1. Reviews 2. Safety Information
  60. 60. Before
  61. 61. After
  62. 62. 10% drop
  63. 63. 7
  64. 64. Before
  65. 65. After
  66. 66. 14.7% drop
  67. 67. Fast forward… more tests happen.
  68. 68. Title tests usually change traffic between ~5 - 15% This stayed equivalent across site size and industry. #INBOUND2018 @dom_woodman
  69. 69. 56% of all title tag changes were negative Around 37% were null and only 6% were actually positive. It’s really hard to write a good title. #INBOUND2018 @dom_woodman
  70. 70. Writing titles is just really really hard.
  71. 71. You can’t stop testing titles. Titles exist in the context of the rest of a SERP, when everyone copies your good title format (and they will) you’ll have to mix it up again. #INBOUND2018 @dom_woodman
  72. 72. Title/meta effects typically appear in 2-4 days. It’s also easy to validate the changes have been picked up because you can scrape it from the SERPS. #INBOUND2018 @dom_woodman
  73. 73. Heavy investment SEO changes
  74. 74. 8
  75. 75. We can render Javascript!
  76. 76. Javascript EnabledJavascript Disabled Control
  77. 77. Javascript EnabledJavascript Disabled Variant
  78. 78. 6.2% increase.
  79. 79. 5% increase
  80. 80. Visible content on the initial page load matters Although there is a lot subtlety to how Google renders JS, we haven’t covered here. #INBOUND2018 @dom_woodman
  81. 81. 9
  82. 82. No effect/negative effect
  83. 83. Increase of 3.1%
  84. 84. SEO content sometimes helps… :( Damn it. #INBOUND2018 @dom_woodman
  85. 85. The same changes have different effects on different sites. This is the big one. There really isn’t best practice. #INBOUND2018 @dom_woodman
  86. 86. (Assuming it was topically similar). We’ve saved clients a lot of money, by showing they could reduce content without an effect. Reducing content on non-article pages was often null. #INBOUND2018 @dom_woodman
  87. 87. Structured data & dangerous assumptions
  88. 88. 10
  89. 89. Adding structured data to category pages
  90. 90. Increase of 11%
  91. 91. 11
  92. 92. Increase of 11%
  93. 93. $$$$$$ - 150K additional sessions a month. How did your last year go?
  94. 94. 12, 13, 14
  95. 95. Null.
  96. 96. Null.
  97. 97. Null.
  98. 98. The same changes have different effects on different sites. This is the big one. There really isn’t best practice. #INBOUND2018 @dom_woodman
  99. 99. The same changes have different effects on different sites. This is the big one. There really isn’t best practice.
  100. 100. Structured data has an effect outside of rich snippets Occasionally we got big 10-15% wins on important templates, it varied wildly and was mostly null (never negative). #INBOUND2018 @dom_woodman
  101. 101. Periodically re-challenge your beliefs I was lucky to test a successful site first. In a different order, I might’ve given up on my hypothesis, before finding the right site. #INBOUND2018 @dom_woodman
  102. 102. 15
  103. 103. Increase of 16%
  104. 104. 16
  105. 105. 5 star much value very marketer
  106. 106. 0.0% change. Again impressively null.
  107. 107. If you don’t have intent, bells and whistles fail. 5 star rich snippets increased traffic by 16% on the right site with the right intent. When the intent wasn’t there, they appeared, but did nothing. #INBOUND2018 @dom_woodman
  108. 108. Risky business...
  109. 109. 17
  110. 110. <time itemprop="dateModified" datetime="{{date}}"> Updated on: {{date}} </time>
  111. 111. 8% increase
  112. 112. Freshness does matter. But I wouldn’t recommend faking last modified dates... #INBOUND2018 @dom_woodman
  113. 113. Agile testing can help you take larger risks. If you can measure and quickly roll out/roll back a test, you can try things you might not normally feel comfortable doing. #INBOUND2018 @dom_woodman
  114. 114. How testing changes relationships
  115. 115. 18
  116. 116. BUSINESS Synergy Dynamic
  117. 117. Increase of 12%
  118. 118. Increase of 12%
  119. 119. Null. Go figure.
  120. 120. You don’t need to argue, when you can test. When you have solid testing framework and can build things quickly and easily, testing is easier than arguing and removes arguments from a relationship. #INBOUND2018 @dom_woodman
  121. 121. Also null.
  122. 122. Happiness isn’t just up and to the right Instead the focus turns to other metrics like how many tests are we running, how can we improve the testing process. #INBOUND2018 @dom_woodman
  123. 123. A negative test you rolled back is a bullet dodged Negative tests feel like wasted time. Once you realise without testing you probably would’ve rolled out, it feels a lot better. #INBOUND2018 @dom_woodman
  124. 124. You’re about to be forced out of silos. You’re going to need to coordinate and work tightly with product and QA teams if you’re not already. #INBOUND2018 @dom_woodman
  125. 125. Conclusions
  126. 126. Hmm that did nothing Crushing failure Ooh traffic went up
  127. 127. The same changes have different effects on different sites. Really can’t emphasize this one enough. #INBOUND2018 @dom_woodman
  128. 128. Get a testing framework. Most tests fail or are null. Having a framework will help you move faster and find those wins. #INBOUND2018 @dom_woodman
  129. 129. Testing will improve your relationships. You’ll have to spend less time arguing and it creates a culture of curiosity. #INBOUND2018 @dom_woodman
  130. 130. Periodically re-challenge your beliefs You probably have some beliefs about what works or doesn’t work which are wrong from blind chance. Re-test them. #INBOUND2018 @dom_woodman
  131. 131. Making changes to sections of pages on templates ● Making SEO changes with tag manager ● Cloud flare edge workers ● DistilledODN General useful posts on testing frameworks & velocity ● Hypothesis framework ● Running a weekly growth meeting Do it yourself How does split testing work? ● How does SEO split testing work? Examples ● Pinterest - Demystifying SEO with experiments ● Etsy - SEO title tag testing Measuring SEO split tests ● Google’s original causal impact paper ● A DIY tool for measuring SEO split tests ● A walkthrough of the R Causal Impact library
  132. 132. bit.ly/inbound-odn
  133. 133. bit.ly/inbound18-slides @dom_woodman

×