This document discusses the importance of split testing in SEO. It provides examples of split tests conducted by various large websites like Pinterest and Etsy. These examples show that traditional SEO best practices sometimes do not work, while other untested changes can significantly improve organic traffic. The key lessons are that search engine algorithms and ranking signals have become more complex, user-centric and interconnected, making testing critical for effective SEO. Hypothesis-based testing ensures changes are targeted and helps avoid wasting time on ineffective optimizations.

1. SEO TESTS ON BIG SITES & SMALL
WHAT ETSY, PINTEREST AND OTHERS CAN TEACH US
Tom Anthony

2. WOW OPENINGNBED
2
NBED SOME WOWNESS
A COUPLE OF IDEAS BELOW
FIRST DEMONSTRATES IT CAN BE DANGEROUS TO TRY SOMETHING WITHOUT KNOWING
SECOND IS LIKE PEOPLE WHO BLOG STUFF AND CONVINCE OTHERS - BUT ARE WILDLY WRONG

3. 3

4. @TomAnthonySEO #SMX
THIS IS WHY MEDICINE
HAS CONTROLLED STUDIES

5. @TomAnthonySEO #SMX
INTRO ODN
WE RUN TESTS NBED

6. @TomAnthonySEO #SMX
USERS ARE SPLIT INTO TWO GROUPS
‘TRADITIONAL’ CRO A/B TESTING
Visitors A & B:
control version
of a page
Visitors C & D:
test version
of the same page

7. @TomAnthonySEO #SMX
SPLIT A GROUP OF SIMILAR PAGES INTO TWO GROUPS
SEO SPLIT-TESTING
Pages 1, 2, 3:
Control Layout
Pages 4, 5, 6:
Test Layout
All users see the
same as each other

8. @TomAnthonySEO #SMX
IMPORTANT - GOOGLEBOT IS TREATED NO DIFFERENTLY
SEO SPLIT-TESTING
Pages 1, 2, 3:
Control Layout
Pages 4, 5, 6:
Test Layout
Googlebot sees
the same as users

9. @TomAnthonySEO #SMX
LASTLY - MEASURE WHICH SET OF PAGES GOOGLE PREFERS
SEO SPLIT-TESTING

10. @TomAnthonySEO #SMX
A 12% UPLIFT FOR THIS UK RETAILER
ADDING SCHEMA.ORG PRODUCT REVIEWS

11. @TomAnthonySEO #SMX
Early on we found many SEO ‘best practice’ changes don’t help.

12. @TomAnthonySEO #SMX
FROM ALTAVISTA TO PRESENT DAY.
SEO TIMELINE
1
Pre-search engine, big list of sites.
Dark Ages

13. @TomAnthonySEO #SMX
FROM ALTAVISTA TO PRESENT DAY.
SEO TIMELINE
1
Pre-search engine, big list of sites.
2
Pre-Google. Keyword stuffing reigns.
Dark Ages
Middle Ages

14. @TomAnthonySEO #SMX
FROM ALTAVISTA TO PRESENT DAY.
SEO TIMELINE
1
Pre-search engine, big list of sites.
2
Pre-Google. Keyword stuffing reigns.
3
Google arrives. PageRank. Links.
Dark Ages
Middle Ages
Renaissance

15. @TomAnthonySEO #SMX
FROM ALTAVISTA TO PRESENT DAY.
SEO TIMELINE
1
Pre-search engine, big list of sites.
2
Pre-Google. Keyword stuffing reigns.
3
Google arrives. PageRank. Links.
Dark Ages
Middle Ages
Renaissance
Ranking signals become external.
Knowing about your site is not enough.

16. @TomAnthonySEO #SMX
FROM ALTAVISTA TO PRESENT DAY.
SEO TIMELINE
1
Pre-search engine, big list of sites.
2
Pre-Google. Keyword stuffing reigns.
3
Google arrives. PageRank. Links.
4
WMT/GSC. Link indexes. Rank tracking.
Dark Ages
Middle Ages
Renaissance
Industrial Revolution

17. @TomAnthonySEO #SMX
FROM ALTAVISTA TO PRESENT DAY.
SEO TIMELINE
1 Dark Ages
2 Middle Ages
Pre-Google. Keyword stuffing reigns.
3 Renaissance
Google arrives. PageRank. Links.
4 Industrial Revolution
Pre-search engine, big list of sites.
WMT/GSC. Link indexes. Rank tracking.
The rise of data…
Cannot be effective without
these external data sources.

18. @TomAnthonySEO #SMX
FROM ALTAVISTA TO PRESENT DAY.
SEO TIMELINE
1
Pre-search engine, big list of sites.
2
Pre-Google. Keyword stuffing reigns.
3
Google arrives. PageRank. Links.
4
Dark Ages
Middle Ages
Renaissance
Industrial Revolution
WMT/GSC. Link indexes. Rank tracking.
The Google algo
becomes more complex:
• Machine Learning
• Explosion of Signals
• User Signals

19. @TomAnthonySEO #SMX
SHEER SCALE AND THE INTRODUCTION OF MACHINE LEARNING HAVE CHANGED IT…
GOOGLE’S ALGO IS NO LONGER PREDICTABLE
GOOGLE ALGO USED TO BE LOTS OF
INDEPENDENT LEVERS
BUT NOW IT IS A COMPLEX SYSTEM.
IMPOSSIBLE TO PREDICT.
Keyword in H1
# of Links
Domain Age
+
+
+
Relevancy Score
Site Speed
Social Signals
Domain Age
Domain Trust
User Signals
Text above fold
Structured Data
# of Links
Ad Density
Keyword in Title

20. @TomAnthonySEO #SMX
FROM ALTAVISTA TO PRESENT DAY.
SEO TIMELINE
1
Pre-search engine, big list of sites.
2
Pre-Google. Keyword stuffing reigns.
3
Google arrives. PageRank. Links.
4
Dark Ages
Middle Ages
Renaissance
Industrial Revolution
WMT/GSC. Link indexes. Rank tracking.
So, we have data, but it is hard
to understand how to turn
that data into decisions.

21. @TomAnthonySEO #SMX
FROM ALTAVISTA TO PRESENT DAY.
SEO TIMELINE
1
Pre-search engine, big list of sites.
2
Pre-Google. Keyword stuffing reigns.
3
Google arrives. PageRank. Links.
4
5 Information Age
Data-driven SEO via Split-Testing.
Dark Ages
Middle Ages
Renaissance
Industrial Revolution
WMT/GSC. Link indexes. Rank tracking.

22. @TomAnthonySEO #SMX
FROM ALTAVISTA TO PRESENT DAY.
SEO TIMELINE
1
Pre-search engine, big list of sites.
2
Pre-Google. Keyword stuffing reigns.
3
Google arrives. PageRank. Links.
4
5 Information Age
Data-driven SEO via Split-Testing.
Dark Ages
Middle Ages
Renaissance
Industrial Revolution
WMT/GSC. Link indexes. Rank tracking.
Since 2014, we’ve been
headed into this stage.

23. @TomAnthonySEO #SMX
LOTS OF DATA - BUT HARD TO KNOW
THE BEST DECISIONS TO MAKE WITH IT
So many companies have started split-testing…

24. @TomAnthonySEO #SMX
A
GUIDE TO
SPLIT-TESTING
THE NEXT
ERA
OF SEO

25. @TomAnthonySEO #SMX

26. USER SIGNALS ARE RANKING SIGNALS1
2 RANKING SIGNALS ARE NO LONGER LINEAR
3 DATA TO HYPOTHESIS TO ACTION

27. USER SIGNALS ARE RANKING SIGNALS1

28. @TomAnthonySEO #SMX
https://medium.com/@Pinterest_Engineering/demystifying-seo-with-experiments-a183b325cf4c

29. @TomAnthonySEO #SMX
OBSERVATION
Search Console detected many
duplicate title tags on board pages:
e.g. “{board_name} on Pinterest”
Test #1

30. @TomAnthonySEO #SMX
ION
tected many
n board pages:
on Pinterest”
TEST IDEA
Reduce duplicate title tags by including
Pin counts in the title tag.
e.g. “{board_name} on Pinterest | ({number} Pins)
Test #1

31. @TomAnthonySEO #SMX
RESULT
No statistically significant
improvement in traffic.
EA
tags by including
e title tag.
est | ({number} Pins)
Test #1

32. @TomAnthonySEO #SMX
OBSERVATION
“Pinterest is filled with tens of billions
of Pins, but the text descriptions
can be lacking”
Test #2

33. @TomAnthonySEO #SMX
ION
tens of billions
descriptions
king”
TEST IDEA
Add a better description, written by
users of other Pins of the same image.
Test #2

34. @TomAnthonySEO #SMX
RESULT
A 30% increase
in organic traffic.
EA
tion, written by
the same image.
Test #2

35. @TomAnthonySEO #SMX
RESEARCH PAPER FROM GOOGLE IN 2010
GOOGLE IS USING USER SIGNALS
http://static.googleusercontent.com/media/research.google.com/en//pubs/archive/36363.pdf

36. @TomAnthonySEO #SMX
GOOGLE IS USING USER SIGNALS
“Another direction for future work is to incorporate active
learning in order to gather a more representative sample
of user preferences.”
RESEARCH PAPER FROM GOOGLE IN 2010

37. @TomAnthonySEO #SMX
BLOG POST FROM DAN PETROVIC — 3YEARS AGO
GOOGLE IS USING USER SIGNALS
https://moz.com/blog/user-behaviour-data-as-a-ranking-signal

38. @TomAnthonySEO #SMX
EXPERIMENT BY RAND FISHKIN
SHORT TERM CTR AFFECTS RANKINGS
https://twitter.com/randfish/status/612730826615230464

39. Reduce duplicate title tags by including
Pin counts in the title tag.
Test #1
No statistically significant
improvement in traffic.
Why?
Add a better description, written by
users of other Pins of the same image.
Test #2
A 30% increase
in organic traffic.
Why?

40. Reduce duplicate title tags by including
Pin counts in the title tag.
Test #1
No statistically significant
improvement in traffic.
Titles matter as they help users by
making it clear what a page is about.
The change didn’t help users.
Add a better description, written by
users of other Pins of the same image.
Test #2
A 30% increase
in organic traffic.
Better descriptions help users and bots
better understand contents of a pin.
The change helped users.

41. Understand why you are seeing warnings.
Try to address the underlying issue.
TAKEAWAY
Doing this is more likely to have a real impact.

42. Users are unpredictable, and so you
must test whenever they are involved.
TAKEAWAY
Google is using user signals as a ranking factor.

43. 2 RANKING SIGNALS ARE NO LONGER LINEAR

44. “By running a large number of experiments, we
found some well-known strategies for SEO didn’t
work for us, while certain tactics we weren’t
confident about worked like a charm.”

45. Best Practices weren’t having an impact.
But they were wasting time and resource trying them.
TAKEAWAY

46. @TomAnthonySEO #SMX
TEST RESULT: ALT ATTRIBUTES HAD NO IMPACT
WE HAVE FOUND THE SAME MANY TIMES OVERALT ATTRIBUTE TEST FROM LASTYEAR

47. @TomAnthonySEO #SMX
https://codeascraft.com/2017/01/25/optimizing-meta-descriptions-h1s-and-title-tags-lessons-from-multivariate-seo-testing-at-etsy/

48. @TomAnthonySEO #SMX

49. Shorter title tags, consisting only of the target search keyword, appear
more relevant/enticing to search users.
“
”
Shorter title tags can improve CTRs.
TAKEAWAY

50. H1 tags are still important and
still effect SEO performance, but…
TAKEAWAY
We found that an H1 change can have a statistically significant
impact on organic search traffic.
“
”

51. @TomAnthonySEO #SMX
Title uplift: 5%
H1 test uplift: 5%
Uplift from both: 7%
TESTS SHOW THAT UPLIFTS ARE NOT INDEPENDENTEXAMPLE NUMBERS TO ILLUSTRATE

52. All SEO changes affect one another.
TAKEAWAY
Don’t think one change won’t impact others.

53. 3 DATA TO HYPOTHESIS TO ACTION

54. Some of our earliest DistilledODN tests…

55. @TomAnthonySEO #SMX

56. @TomAnthonySEO #SMX

57. @TomAnthonySEO #SMX
BUT BUT BUT… BEST PRACTICE SAID USP IS A GOOD IDEA?!
NOTHING HAPPENED

59. Before
After
query search volume
which tv should i buy 250
query search volume
which tv to buy 1,300

60. @TomAnthonySEO #SMX
MAYBE EVEN A BIT WORSE
NO REAL IMPACT

62. <img src="product1.jpg"
alt="Home 6 Piece Towel Bale - Geo Grey / Zest">

63. @TomAnthonySEO #SMX
WE’VE NEVER HAD AN IMPACT FROM ALT ATTRIBUTE
NADA… NOTHING… NO IMPACT…

65. “Stand back guys,
I got this.”

66. Before
After

67. @TomAnthonySEO #SMX
WORST TEST RESULT WE EVER HAD
OH GOD…

68. COULD BE OUR
RESULTS
AREN’T ACCURATE
?

69. @TomAnthonySEO #SMX
THE RESULTS ARE CONSISTENT
THE RESULT WAS ACCURATE — WE SUCK

70. @TomAnthonySEO #SMX
Most SEO changes have a no or a negative impact.

71. @TomAnthonySEO #SMX
IT LEADS TO MORE EFFECTIVE TESTING
CRO EXPERTS KNOW - YOU NEED A HYPOTHESIS
dis.tl/cro-hypothesis

72. @TomAnthonySEO #SMX
A HYPOTHESISES SHOULD BE SPECIFIC
Changing <abc> will have an impact on <xyz>
& thus improve organic search performance.
Based on <evidence>, we hypothesise…

73. @TomAnthonySEO #SMX
HYPOTHESIS EXAMPLE
Changing <title formats, by…> will improve <CTR>
& thus improve organic search performance.
Based on <survey data>, we hypothesise…

74. @TomAnthonySEO #SMX
CRO HYPOTHESISES FRAMEWORK
https://conversion.com/blog/introducing-hypothesis-framework/

76. @TomAnthonySEO #SMX
EXAMPLE OF HYPOTHESIS APPROACH

77. CATEGORY PAGE HTML ISVERBOSE,
AND CONTAINS A LOT OF JUNK.
ADDING A LIST OF PRODUCTS AS
STRUCTURED MARKUP WILL MAKE
THE INTENT CLEARER.
HYPOTHESIS

78. @TomAnthonySEO #SMX
ADD STRUCTURED MARKUP

79. @TomAnthonySEO #SMX
AN 11% UPLIFT IN ORGANIC TRAFFICOUR HYPOTHESIS WAS ACCURATE

80. @TomAnthonySEO #SMX
REPEATED IN ANOTHER REGION - 11% AGAINWE SEE THIS OFTEN IN DIFFERENT REGIONS

81. #StateOfSearch@TomAnthonySEO

82. #StateOfSearch@TomAnthonySEO

86. THE HYPOTHESIS DID NOT
FIT THESE OTHER SITES.
!

87. Hypothseses ensure you understand why
you are making a change.
A hypothesis approach leads to more effective changes,
and helps avoid wasting time on ineffective changes.
DATAPOINT
TAKEAWAY

88. @TomAnthonySEO #SMX
HAVE A HYPOTHESIS - YOU WILL MAKE BETTER RECOMMENDATIONS

89. SUMMARY

90. @TomAnthonySEO #SMX
TO BE COMPETITIVE TESTING IS BECOMING A NECESSITY
WE ARE IN A NEW ERA OF SEO
1
Pre-search engine, big list of sites.
2
Pre-Google. Keyword stuffing reigns.
3
Google arrives. PageRank. Links.
4
5 Information Age
Data-driven SEO via Split-Testing.
Dark Ages
Middle Ages
Renaissance
Industrial Revolution
WMT/GSC. Link indexes. Rank tracking.
Just as tactics
from here…
…wouldn’t work
well here…

91. @TomAnthonySEO #SMX
TO BE COMPETITIVE TESTING IS BECOMING A NECESSITY
WE ARE IN A NEW ERA OF SEO
1
Pre-search engine, big list of sites.
2
Pre-Google. Keyword stuffing reigns.
3
Google arrives. PageRank. Links.
4
5 Information Age
Data-driven SEO via Split-Testing.
Dark Ages
Middle Ages
Renaissance
Industrial Revolution
WMT/GSC. Link indexes. Rank tracking.
Just as tactics
from here…
…wouldn’t work
well here…
… being competitive
here means testing!

92. SEO must be user-centric,
which makes testing necessary.
TAKEAWAY
Google includes user signals. Users are unpredictable.

93. TAKEAWAY
Realise you are not at
optimised as you think you are.
Many SEO tests get uplifts from updating
titles, meta descriptions & H1s etc.

94. Ranking signals are no longer ‘summed’
together.They effect one another.
TAKEAWAY
This also means that what worked for someone else,
may not work for you.

95. TAKEAWAY
Have a hypothesis for what
you expect from any SEO change.
You will make more effective recommendations.

96. Testing allows you to
stop chasing ranking factors.
TAKEAWAY

97. @TomAnthonySEO #SMX
FREE TOOL TO DO AN ANALYSIS OF TESTS
BLOG POST - HOW YOU CAN TEST
https://www.distilled.net/resources/do-it-yourself-seo-split-testing-tool-with-causal-impact/

98. @TomAnthonySEO #SMX
SMX MUNICH 2017 DECK
MORE TEST EXAMPLES
https://www.slideshare.net/TomAnthony/next-era-of-seo-a-guide-to-seo-splittesting
A
GUIDE TO
SPLIT-TESTING
THE NEXT
ERA
OF SEO

99. DANKE!
@TomAnthonySEO
http://dis.tl/smx-testing-2018