Last year I got a new job and spent the year running all the tests we've done on DistilledODN (an SEO split testing platform). It's changed my perspective, taught me a huge amount and I'd like to take people through all the different lessons I've learned (19 of them in fact). That's everything from: What sort of effect do basic SEO changes? Why is changing your title tags possibly a really risky move? How and when has structured data helped? How important is freshness (and can you fake it)? Does testing change your relationship with a client? Should you put emoji's in everything...

1. A YEAR OF SEO SPLIT
TESTING CHANGED
HOW I THOUGHT
SEO WORKED
@dom_woodman

2. bit.ly/testing-year

5. Hmm that did
nothing
Crushing
failure
Ooh traffic went up

7. All those links I’m building.

9. Hmm that did
nothing
Crushing
failure
Ooh traffic went up

15. How does SEO testing
work & what do you need?

16. Getting started & learning
how to think about testing

17. More than you wanted to know
about titles and metas.

18. Testing resource
heavy investments

19. Structured data &
dangerous assumptions

20. Risky business...

21. How testing changes
relationships

22. Conclusions

23. How does SEO testing
work & what do you need?

24. SEO split testing is template basedHere’s our example website
Home
Animals
Cats Dogs Unicorns Badgers

25. CatsCats
We want to test two templates for our animals pages

26. Cats Dogs Unicorns Badgers
CRO test - Different users, see different templates on the same pages
1

27. Cats Dogs Unicorns Badgers
CRO test - Different users, see different templates on the same pages
1
2
Cats BadgersDogs Unicorns

28. Cats Dogs
SEO test - different users see the same templates on different pages
1
2
Cats Dogs
Cats Dogs

29. Cats Dogs Unicorns Badgers
SEO test - different users see the same templates on different pages
1
2
Cats Dogs Unicorns Badgers
Cats Dogs Unicorns Badgers

30. Then we measure the change in traffic, e.g. 3% daily increase

32. LIABILITY

37. Rank
Search visibility

40. Total additional sessions

41. The fans show margin of error

42. The test is significant when the shaded area crosses 0

43. In this case it happened here:

47. There are two
graphs.

50. Control Variant

51. Black is organic traffic for the variant

52. Start date for the test

53. Take all the data to left and use that to make the model

54. The model of the variant is the blue line

55. We overlay them

56. Compare the model to the variant after the start date

57. In this case the black line is higher, it’s a positive test

58. A good model, fits with the variant before the start date

59. However this was my model. They don’t fit well.

60. How much traffic do you
need for SEO split testing?
Rule of thumb: you need roughly 1000 organic sessions a day to
the group of pages you’re testing.
@dom_woodman

61. That model you
just saw, only had
~60 sessions a day.

62. Getting started & learning
how to think about testing

63. 1

67. Some fluctuation but null.

69. 2

70. Before

71. After

72. 10.8% decrease

74. 3

76. <img src="product1.jpg" alt="Home 6 Piece Towel Bale - Geo Grey /
Zest">

77. A model so perfectly null I thought I had forgotten to launch it

79. 4&5

80. Thought leader.
PHD in AI.
Unbearably smug.

82. Before:

83. After:

84. A stunning 27% drop.

86. A 20% drop.

87. Can’t write
titles for shit.

88. Get a testing framework
You need to be able to iterate and learn from tests. Having a
shared framework, will help with this.
@dom_woodman

89. You don’t need the why
Knowing the why is great (find it if you can), but you don’t need
it to take action.
@dom_woodman

90. More than you wanted to know
about titles and metas.

91. 6

92. 16

93. 1. Reviews
2. Safety Information

94. Before

95. After

96. 10% drop

97. 7

99. Before

100. After

101. 14.7% drop

102. Fast forward… more
tests happen.

103. Title tests usually change
traffic between ~5 - 15%
This stayed equivalent across site size and industry.
@dom_woodman

104. 56% of all title tag
changes were negative
Around 37% were null and only 6% were actually positive. It’s really
hard to write a good title.
@dom_woodman

105. Writing titles
is just really
really hard.

106. You can’t stop testing
titles.
Titles exist in the context of the rest of a SERP, when everyone
copies your good title format (and they will) you’ll have to mix it
up again.
@dom_woodman

107. Title/meta effects typically
appear in 2-4 days.
It’s also easy to validate the changes have been picked up
because you can scrape it from the SERPS.
@dom_woodman

108. Heavy investment
SEO changes

109. 8

110. We can render Javascript!

111. Javascript EnabledJavascript Disabled
Control

112. Javascript EnabledJavascript Disabled
Variant

113. 6.2% increase.

114. 5% increase

115. Visible content on the
initial page load matters
Although there is a lot subtlety to how Google renders JS, we
haven’t covered here.
@dom_woodman

116. 9

118. No effect/negative effect

119. Increase of 3.1%

120. SEO content sometimes
helps… :(
Damn it.
@dom_woodman

121. The same changes have
different effects on
different sites.
This is the big one. There really isn’t best practice.
@dom_woodman

122. (Assuming it was topically similar). We’ve saved clients a lot of
money, by showing they could reduce content without an effect.
Reducing content on
non-article pages was
often null.
@dom_woodman

123. Structured data &
dangerous assumptions

124. 10

127. Adding structured data to category pages

128. Increase of 11%

129. 11

130. Increase of 11%

131. $$$$$$ - 150K additional
sessions a month.
How did your last year go?

133. 12, 13,
14

135. Null.

136. Null.

137. Null.

138. The same changes have
different effects on
different sites.
This is the big one. There really isn’t best practice.
@dom_woodman

139. The same changes have
different effects on
different sites.
This is the big one. There really isn’t best practice.
@dom_woodman
THIS TIME I
ACTUALLY
LEARNED IT

140. Structured data has an
effect outside of rich
snippets
Occasionally we got big 10-15% wins on important templates, it
varied wildly and was mostly null (never negative).
@dom_woodman

141. Periodically re-challenge
your beliefs
I was lucky to test a successful site first. In a different order, I
might’ve given up on my hypothesis, before finding the right site.
@dom_woodman

142. 15

144. Increase of 16%

145. 16

147. 5 star
much value
very marketer
so dollars

149. 0.0% change. Again impressively null.

150. If you don’t have intent,
bells and whistles fail.
5 star rich snippets increased traffic by 16% on the right site with the
right intent. When the intent wasn’t there, they appeared, but did
nothing.
@dom_woodman

151. Risky business...

153. 17

156. <time itemprop="dateModified" datetime="{{date}}">
Updated on: {{date}}
</time>

157. 8% increase

158. Freshness does matter.
But I wouldn’t recommend faking last modified dates...
@dom_woodman

159. Agile testing can help
you take larger risks.
If you can measure and quickly roll out/roll back a test, you can try
things you might not normally feel comfortable doing.
@dom_woodman

160. How testing changes
relationships

161. 18

162. BUSINESS
Synergy
Dynamic

166. Increase of 12%

167. Increase of 12%

168. Null. Go figure.

169. You don’t need to argue,
when you can test.
When you have solid testing framework and can build things quickly
and easily, testing is easier than arguing and removes arguments
from a relationship.
@dom_woodman

172. Also null.

174. Happiness isn’t just up
and to the right
Instead the focus turns to other metrics like how many tests are we
running, how can we improve the testing process.
@dom_woodman

175. A negative test you rolled
back is a bullet dodged
Negative tests feel like wasted time. Once you realise without
testing you probably would’ve rolled out, it feels a lot better.
@dom_woodman

176. You’re about to be
forced out of silos.
You’re going to need to coordinate and work tightly with product
and QA teams if you’re not already.
@dom_woodman

177. Conclusions

178. Hmm that did
nothing
Crushing
failure
Ooh traffic went up

179. The same changes have
different effects on
different sites.
Really can’t emphasize this one enough.
@dom_woodman

180. Get a testing framework.
Most tests fail or are null.
Having a framework will help you move faster and find those wins.
@dom_woodman

181. Testing will improve your
relationships.
You’ll have to spend less time arguing and it creates a culture of
curiosity.
@dom_woodman

182. Periodically re-challenge
your beliefs
You probably have some beliefs about what works or doesn’t work
which are wrong from blind chance. Re-test them.
@dom_woodman

183. Making changes to sections of pages
on templates
● Making SEO changes with tag
manager
● Cloud flare edge workers
● DistilledODN
General useful posts on testing
frameworks & velocity
● Hypothesis framework
● Running a weekly growth meeting
Do it yourself
How does split testing work?
● How does SEO split testing work?
Examples
● Pinterest - Demystifying SEO with
experiments
● Etsy - SEO title tag testing
Measuring SEO split tests
● Google’s original causal impact
paper
● A DIY tool for measuring SEO split
tests
● A walkthrough of the R Causal
Impact library

184. bit.ly/distilled-odn

185. bit.ly/testing-year
@dom_woodman