Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

19 Lessons I learned from a year of SEO split testing

22,480 views

Published on

Last year I got a new job and spent the year running all the tests we've done on DistilledODN (an SEO split testing platform).

It's changed my perspective, taught me a huge amount and I'd like to take people through all the different lessons I've learned (19 of them in fact).

That's everything from: What sort of effect do basic SEO changes? Why is changing your title tags possibly a really risky move? How and when has structured data helped? How important is freshness (and can you fake it)? Does testing change your relationship with a client? Should you put emoji's in everything...

Published in: Marketing
  • We run our own split testing, albeit not with ODN, although we've had discussions to do so. One of the things I've learned in the process of growing our organic traffic - sometimes by more than 100% in a year on some big domains is that straight forward A/B testing can provide very bad results. Why? Because A/B testing assumes that the factors that impact an outcome will fit nicely into a straight multiplicative and/or additive model. Our experimentation demonstrates, that simply isn't true. SEO operating model, IMO, requires more of a systems engineering approach. What do I mean by this and why is it important? Well think of SEO a bit like aerodynamics (since that was the field I initially came from, its a good example for me). Some of the performance measures you assume work at one state, simply won't work in another. In the aerodynamics world, the type of engine or airfoil you put onto an aircraft that flies into different regions of the atmosphere and at different speeds mean that the things that worked at one level, won't work at another. That's why when you test the same change on two websites, the impacts can often be different. Its the same reason why the same medicine can have opposite effects on two people or why training two dogs in the exact same method can have different outcomes. A/B makes the false assumption of equality where none exists. If there is one common takeaway I've seen from reading Dominic's slides that we've learned - its that its much easier to have things go negative than positive - and that is particularly true with websites that tend to rank more highly than others. Small changes (particularly negative ones) have have disastrous impacts.
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Huge learnings! Thanks for sharing Dominic!
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Great deck. Is there a video of this talk?
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

19 Lessons I learned from a year of SEO split testing

  1. 1. A YEAR OF SEO SPLIT TESTING CHANGED HOW I THOUGHT SEO WORKED @dom_woodman
  2. 2. bit.ly/testing-year
  3. 3. Hmm that did nothing Crushing failure Ooh traffic went up
  4. 4. All those links I’m building.
  5. 5. Hmm that did nothing Crushing failure Ooh traffic went up
  6. 6. How does SEO testing work & what do you need?
  7. 7. Getting started & learning how to think about testing
  8. 8. More than you wanted to know about titles and metas.
  9. 9. Testing resource heavy investments
  10. 10. Structured data & dangerous assumptions
  11. 11. Risky business...
  12. 12. How testing changes relationships
  13. 13. Conclusions
  14. 14. How does SEO testing work & what do you need?
  15. 15. SEO split testing is template basedHere’s our example website Home Animals Cats Dogs Unicorns Badgers
  16. 16. CatsCats We want to test two templates for our animals pages
  17. 17. Cats Dogs Unicorns Badgers CRO test - Different users, see different templates on the same pages 1
  18. 18. Cats Dogs Unicorns Badgers CRO test - Different users, see different templates on the same pages 1 2 Cats BadgersDogs Unicorns
  19. 19. Cats Dogs SEO test - different users see the same templates on different pages 1 2 Cats Dogs Cats Dogs
  20. 20. Cats Dogs Unicorns Badgers SEO test - different users see the same templates on different pages 1 2 Cats Dogs Unicorns Badgers Cats Dogs Unicorns Badgers
  21. 21. Then we measure the change in traffic, e.g. 3% daily increase
  22. 22. LIABILITY
  23. 23. Rank Search visibility
  24. 24. Total additional sessions
  25. 25. The fans show margin of error
  26. 26. The test is significant when the shaded area crosses 0
  27. 27. In this case it happened here:
  28. 28. There are two graphs.
  29. 29. Control Variant
  30. 30. Black is organic traffic for the variant
  31. 31. Start date for the test
  32. 32. Take all the data to left and use that to make the model
  33. 33. The model of the variant is the blue line
  34. 34. We overlay them
  35. 35. Compare the model to the variant after the start date
  36. 36. In this case the black line is higher, it’s a positive test
  37. 37. A good model, fits with the variant before the start date
  38. 38. However this was my model. They don’t fit well.
  39. 39. How much traffic do you need for SEO split testing? Rule of thumb: you need roughly 1000 organic sessions a day to the group of pages you’re testing. @dom_woodman
  40. 40. That model you just saw, only had ~60 sessions a day.
  41. 41. Getting started & learning how to think about testing
  42. 42. 1
  43. 43. Some fluctuation but null.
  44. 44. 2
  45. 45. Before
  46. 46. After
  47. 47. 10.8% decrease
  48. 48. 3
  49. 49. <img src="product1.jpg" alt="Home 6 Piece Towel Bale - Geo Grey / Zest">
  50. 50. A model so perfectly null I thought I had forgotten to launch it
  51. 51. 4&5
  52. 52. Thought leader. PHD in AI. Unbearably smug.
  53. 53. Before:
  54. 54. After:
  55. 55. A stunning 27% drop.
  56. 56. A 20% drop.
  57. 57. Can’t write titles for shit.
  58. 58. Get a testing framework You need to be able to iterate and learn from tests. Having a shared framework, will help with this. @dom_woodman
  59. 59. You don’t need the why Knowing the why is great (find it if you can), but you don’t need it to take action. @dom_woodman
  60. 60. More than you wanted to know about titles and metas.
  61. 61. 6
  62. 62. 16
  63. 63. 1. Reviews 2. Safety Information
  64. 64. Before
  65. 65. After
  66. 66. 10% drop
  67. 67. 7
  68. 68. Before
  69. 69. After
  70. 70. 14.7% drop
  71. 71. Fast forward… more tests happen.
  72. 72. Title tests usually change traffic between ~5 - 15% This stayed equivalent across site size and industry. @dom_woodman
  73. 73. 56% of all title tag changes were negative Around 37% were null and only 6% were actually positive. It’s really hard to write a good title. @dom_woodman
  74. 74. Writing titles is just really really hard.
  75. 75. You can’t stop testing titles. Titles exist in the context of the rest of a SERP, when everyone copies your good title format (and they will) you’ll have to mix it up again. @dom_woodman
  76. 76. Title/meta effects typically appear in 2-4 days. It’s also easy to validate the changes have been picked up because you can scrape it from the SERPS. @dom_woodman
  77. 77. Heavy investment SEO changes
  78. 78. 8
  79. 79. We can render Javascript!
  80. 80. Javascript EnabledJavascript Disabled Control
  81. 81. Javascript EnabledJavascript Disabled Variant
  82. 82. 6.2% increase.
  83. 83. 5% increase
  84. 84. Visible content on the initial page load matters Although there is a lot subtlety to how Google renders JS, we haven’t covered here. @dom_woodman
  85. 85. 9
  86. 86. No effect/negative effect
  87. 87. Increase of 3.1%
  88. 88. SEO content sometimes helps… :( Damn it. @dom_woodman
  89. 89. The same changes have different effects on different sites. This is the big one. There really isn’t best practice. @dom_woodman
  90. 90. (Assuming it was topically similar). We’ve saved clients a lot of money, by showing they could reduce content without an effect. Reducing content on non-article pages was often null. @dom_woodman
  91. 91. Structured data & dangerous assumptions
  92. 92. 10
  93. 93. Adding structured data to category pages
  94. 94. Increase of 11%
  95. 95. 11
  96. 96. Increase of 11%
  97. 97. $$$$$$ - 150K additional sessions a month. How did your last year go?
  98. 98. 12, 13, 14
  99. 99. Null.
  100. 100. Null.
  101. 101. Null.
  102. 102. The same changes have different effects on different sites. This is the big one. There really isn’t best practice. @dom_woodman
  103. 103. The same changes have different effects on different sites. This is the big one. There really isn’t best practice. @dom_woodman THIS TIME I ACTUALLY LEARNED IT
  104. 104. Structured data has an effect outside of rich snippets Occasionally we got big 10-15% wins on important templates, it varied wildly and was mostly null (never negative). @dom_woodman
  105. 105. Periodically re-challenge your beliefs I was lucky to test a successful site first. In a different order, I might’ve given up on my hypothesis, before finding the right site. @dom_woodman
  106. 106. 15
  107. 107. Increase of 16%
  108. 108. 16
  109. 109. 5 star much value very marketer so dollars
  110. 110. 0.0% change. Again impressively null.
  111. 111. If you don’t have intent, bells and whistles fail. 5 star rich snippets increased traffic by 16% on the right site with the right intent. When the intent wasn’t there, they appeared, but did nothing. @dom_woodman
  112. 112. Risky business...
  113. 113. 17
  114. 114. <time itemprop="dateModified" datetime="{{date}}"> Updated on: {{date}} </time>
  115. 115. 8% increase
  116. 116. Freshness does matter. But I wouldn’t recommend faking last modified dates... @dom_woodman
  117. 117. Agile testing can help you take larger risks. If you can measure and quickly roll out/roll back a test, you can try things you might not normally feel comfortable doing. @dom_woodman
  118. 118. How testing changes relationships
  119. 119. 18
  120. 120. BUSINESS Synergy Dynamic
  121. 121. Increase of 12%
  122. 122. Increase of 12%
  123. 123. Null. Go figure.
  124. 124. You don’t need to argue, when you can test. When you have solid testing framework and can build things quickly and easily, testing is easier than arguing and removes arguments from a relationship. @dom_woodman
  125. 125. Also null.
  126. 126. Happiness isn’t just up and to the right Instead the focus turns to other metrics like how many tests are we running, how can we improve the testing process. @dom_woodman
  127. 127. A negative test you rolled back is a bullet dodged Negative tests feel like wasted time. Once you realise without testing you probably would’ve rolled out, it feels a lot better. @dom_woodman
  128. 128. You’re about to be forced out of silos. You’re going to need to coordinate and work tightly with product and QA teams if you’re not already. @dom_woodman
  129. 129. Conclusions
  130. 130. Hmm that did nothing Crushing failure Ooh traffic went up
  131. 131. The same changes have different effects on different sites. Really can’t emphasize this one enough. @dom_woodman
  132. 132. Get a testing framework. Most tests fail or are null. Having a framework will help you move faster and find those wins. @dom_woodman
  133. 133. Testing will improve your relationships. You’ll have to spend less time arguing and it creates a culture of curiosity. @dom_woodman
  134. 134. Periodically re-challenge your beliefs You probably have some beliefs about what works or doesn’t work which are wrong from blind chance. Re-test them. @dom_woodman
  135. 135. Making changes to sections of pages on templates ● Making SEO changes with tag manager ● Cloud flare edge workers ● DistilledODN General useful posts on testing frameworks & velocity ● Hypothesis framework ● Running a weekly growth meeting Do it yourself How does split testing work? ● How does SEO split testing work? Examples ● Pinterest - Demystifying SEO with experiments ● Etsy - SEO title tag testing Measuring SEO split tests ● Google’s original causal impact paper ● A DIY tool for measuring SEO split tests ● A walkthrough of the R Causal Impact library
  136. 136. bit.ly/distilled-odn
  137. 137. bit.ly/testing-year @dom_woodman

×