The document discusses an SEO expert's approach to making recommendations by forming hypotheses and testing them. It notes that many SEO recommendations have little impact and presents a framework for developing more effective recommendations through hypothesis-driven testing. This involves specifically hypothesizing how a change will impact metrics, gathering relevant data, measuring the results of split tests, and iterating based on learnings. Examples are provided of hypotheses tested around structured data, date annotations, and information architecture that led to measurable improvements. The document advocates for this approach to yield more impactful results than recommendations made without testing.

1. SEO BY HYPOTHESIS
Recommend → Hypothesise & Test
@TomAnthonySEO

2. Cooking nuts in mud
Replicated it, but devil in details
Google black box - can’t always replicate
#dmsso@TomAnthonySEOwww.madeinindia.tv

3. recommendation!
WE GET IDEAS FROM ALL OVER…
#dmsso@TomAnthonySEO
blog
post
worked
before
colleague
google
guidelines
new
idea
tried for
another client

4. FROM THERE, WE MAKE RECOMMENDATIONS
“Add ‘Free Shipping’ to title tags.”
#dmsso@TomAnthonySEO

5. DO WE UNDERSTAND
THE IMPACT
OF THESE CHANGES
?

6. WE SURVEYED 2000+ SEOS…
future of tv advertising
Which URL ranks higher for the search:

7. MORE EXPERIENCED SEOS DO BETTER…
30%
40%
50%
60%
70%
Laypeople <= 3 yrs SEO > 3 yrs SEO

8. … BUT WE DON’T DO GREAT
30%
40%
50%
60%
70%
Laypeople <= 3 yrs SEO > 3 yrs SEO Coin flip

9. WE CANNOT PREDICT
WHAT RANKS
WITH ANY ACCURACY
!

10. WE’VE BEEN A/B TESTING…
NBED - we have started testing
#dmsso@TomAnthonySEO

11. RECOMMENDATION → TEST
“Add ‘Free Shipping’ to title tags.”
#dmsso@TomAnthonySEO
“Let’s test ‘Free Shipping’ in title tags.”

14. NOTHING HAPPENED
#dmsso@TomAnthonySEO

16. Before
After
query search volume
which tv should i buy 250
query search volume
which tv to buy 1,300

17. THAT MADE THINGS WORSE
#dmsso@TomAnthonySEO

19. <img src="product1.jpg"
alt="Home 6 Piece Towel Bale - Geo Grey / Zest">

20. NADA… NOTHING… NO IMPACT…
#dmsso@TomAnthonySEO

22. “Stand back guys,
I got this.”

23. Before
After

24. OH GOD…
#dmsso@TomAnthonySEO

25. COULD BE OUR
RESULTS
AREN’T ACCURATE
?

26. NOPE, THE PLATFORM IS SOLID. WE SUCK.
#dmsso@TomAnthonySEO

27. HARDER THAN ANTICIPATED TO PICK WINNERS
#dmsso@TomAnthonySEO

28. CAN WE MAKE MORE
EFFECTIVE
RECOMMENDATIONS
?

29. LAST YEAR: RECOMMENDED TESTING

30. FOLLOW THE CRO EXPERTS - dis.tl/cro-hypothesis

31. MEASURE
THE RESULT
GATHER
SOME DATA
BUILD A
HYPOTHESIS

32. BUILD A
HYPOTHESIS

33. IT REQUIRES BEING SPECIFIC
Changing <abc> will have an impact on <xyz>
& thus improve organic search performance.
Based on <evidence>, we hypothesise…
#dmsso@TomAnthonySEO

34. FOR EXAMPLE..
Changing <title formats, by…> will improve <ctr>
& thus improve organic search performance.
Based on <survey data>, we hypothesise…
#dmsso@TomAnthonySEO

35. HYPOTHESIS BENEFITS
Made to be more consistent.
It ensures there is clarity.
#dmsso@TomAnthonySEO
To support it we need data.

36. GATHER
SOME DATA

37. Control
Influence
We have the same
data as Google.
Google has data
that we don’t.

38. Control
Influence
Keyword
targeting
Internal
links
We have the same
data as Google.
Google has data
that we don’t.

39. Control
Influence
Website
‘quality’
We have the same
data as Google.
Google has data
that we don’t.

40. Control
Influence
Usage
Data
We have the same
data as Google.
Google has data
that we don’t.

41. Control
Influence
External
Links
We have the same
data as Google.
Google has data
that we don’t.

42. Control
Influence
External
Links
Usage
Data
Keyword
targeting
Internal
links
Website
‘quality’
We have the same
data as Google.
Google has data
that we don’t.

43. Control
Influence
External
Links
Usage
Data
Keyword
targeting
Internal
links
Website
‘quality’
Test &
Model
We have the same
data as Google.
Google has data
that we don’t.

44. Control
Influence
External
Links
Usage
Data
Keyword
targeting
Internal
links
Website
‘quality’
Survey &
Study
We have the same
data as Google.
Google has data
that we don’t.

45. Control
Influence
External
Links
Usage
Data
Keyword
targeting
Internal
links
Website
‘quality’
Analyse
Better
We have the same
data as Google.
Google has data
that we don’t.

46. Internal
links
Analyse
Better

47. INFORMATION ARCHITECTURE
#dmsso@TomAnthonySEO

48. INFORMATION ARCHITECTURE
#dmsso@TomAnthonySEO
• Few sites are this simple.
• Can’t measure impact of
changes.

49. CURRENT BEST APPROACH
#dmsso@TomAnthonySEO
Calculate internal PageRank
Use interactive visualisations

50. VISUALISATIONS - USEFUL, BUT RARELY
Credit: Paul Shapiro

51. INTERNAL PAGERANK - POWERFUL, BUT QUIRKY
Credit: Paul Shapiro

52. PEOPLE SEE REAL RESULTS FROM BIG CHANGES TO IA
See Alex’s fascinating Mozcon talk [PDF]

53. BUT REAL-WORLD
CHANGES ARE:
HARD TO MAKE &
HARD TO UNDO
!

54. OUR INTUITION
IS REALLY BAD.
!

55. WE ASKED PEOPLE, GIVEN A SITE…

56. ADD A NEW PAGE - LINKED FROM HOMEPAGE

57. IT LINKS TO SAME PAGES AS HOMEPAGE

58. WHAT HAPPENS TO PAGERANK?
PageRank
Change?

59. WILL ASKED ON TWITTER…
#dmsso@TomAnthonySEO

60. …3/5 PEOPLE GOT IT WRONG
NOPE
#dmsso@TomAnthonySEO

61. GIVEN THAT,
HOW CAN WE
HYPOTHESISE
THE IMPACT?
?

62. WEIGHTED
INTERNAL PAGERANK

63. CRAWL THE SITE

64. FETCH & NORMALISE MOZRANK
Raw MozRank Normalised MozRank
3.67E-13 1.0%
3.35E-11 94.2%
1.71E-13 0.5%
1.64E-13 0.5%
1.59E-13 0.4%
3.28E-13 0.9%
6.88E-14 0.2%
2.45E-13 0.7%
7.12E-14 0.2%
3.12E-13 0.9%
1.67E-13 0.5%
#dmsso@TomAnthonySEO

65. OVERLAY MOZRANK ON TO INTERNAL PR ALGO
8.2
5.3 4.3 4.8

66. WHAT HAPPENS TO PAGERANK?
PageRank
Change?

67. MODEL EXACTLY WHAT WOULD HAPPEN
8.05
5.3 4.3 4.8
4.05

68. FIND THE WHOLE PROCESS HERE
http://dis.tl/ia-pagerank
#dmsso@TomAnthonySEO

69. Control
Influence
External
Links
Usage
Data
Keyword
targeting
Internal
links
Website
‘quality’
Survey &
Study
We have the same
data as Google.
Google has data
that we don’t.

70. Website
‘quality’
Survey
& Study
Usage
Data

71. USAGE DATA: WE CAN’T GUESS
#dmsso@TomAnthonySEO
It is impossible to guess what
people prefer.

72. SERP TURKEY
#dmsso@TomAnthonySEO

73. SERP TURKEY

74. SERP TURKEY

75. PANDA & QUALITY

76. • How do you communicate it to clients / bosses?
• How do you benchmark it against competitors?
• How do you figure out if proposed changes improve it?
QUALITY?

77. Gather human rater
information
Google employs
thousands of human
quality raters to answer
questionnaires about
many kinds of website

78. Train ML models
Google uses the human
questionnaires as training
data for ML models of
“quality”
Gather human rater
information
Google employs
thousands of human
quality raters to answer
questionnaires about
many kinds of website

79. 2011
Panda released
The Panda quality
algorithm starts being
used as a batch process
modifying the regular core
algorithm
Train ML models
Google uses the human
questionnaires as training
data for ML models of
“quality”
Gather human rater
information
Google employs
thousands of human
quality raters to answer
questionnaires about
many kinds of website

80. 2011
Panda released
The Panda quality
algorithm starts being
used as a batch process
modifying the regular core
algorithm
Train ML models
Google uses the human
questionnaires as training
data for ML models of
“quality”
Gather human rater
information
Google employs
thousands of human
quality raters to answer
questionnaires about
many kinds of website
2016
Panda goes real-time
“Quality” becomes a first-
class ranking factor in the
core algorithm

81. QUALITY - RANKING FACTOR IS A PROXY
#dmsso@TomAnthonySEO
Google use Machine Learning to
predict what we can just ask
people.
Actually
measure:
ML PROXY
FOR QUALITY
Want to
measure:
QUALITY

82. 2011: WILL SUGGESTS RUNNING PANDA SURVEYS
#dmsso@TomAnthonySEO

83. PANDA SURVEY
Client site 1 Client site 2
Would you trust information from this website? 72% 64%

84. PANDA SURVEY
Client site 1 Client site 2 Key competitor
Would you trust information from this website? 72% 64% 81%

85. PANDA SURVEY
Client site 1 Client site 2 Key competitor
Would you trust information from this website? 72% 64% 81%
Is this website written by experts? 50% 46% 65%

86. PANDA SURVEY
Client site 1 Client site 2 Key competitor
Would you trust information from this website? 72% 64% 81%
Is this website written by experts? 50% 46% 65%
Would you give this site your credit card details? 29% 21% 43%
Are there any noticeable errors on this page? 6% 4% 1%
Does this page provide original content or info? 76% 72% 85%
Would you recognize this site as an authority? 44% 33% 58%
Does this website contain insightful analysis? 72% 62% 81%
Would you consider bookmarking pages on this site? 44% 38% 56%
Are there excessive adverts on this website? 2% 2% 8%
Could pages from this site appear in print? 54% 54% 59%

87. MEASURE
THE RESULT

88. DECK - HOW YOU CAN DO SPLIT-TESTING
A
GUIDE TO
SPLIT-TESTING
THE NEXT
ERA
OF SEO
dis.tl/diy-ab-test

89. MEASURING EXAMPLE

90. HYPOTHESIS
CATEGORY PAGE HTML IS VERBOSE,
AND CONTAINS A LOT OF CRUFT.
ADDING A LIST OF PRODUCTS AS
STRUCTURED MARKUP WILL MAKE
THE INTENT CLEARER.

91. ADD STRUCTURED MARKUP
#dmsso@TomAnthonySEO

92. AN 11% UPLIFT
#dmsso@TomAnthonySEO

93. REPEATED IN ANOTHER REGION - 11% AGAIN
#dmsso@TomAnthonySEO

94. #dmsso@TomAnthonySEO

95. #dmsso@TomAnthonySEO

96. #dmsso@TomAnthonySEO

97. #dmsso@TomAnthonySEO

98. #dmsso@TomAnthonySEO

99. THE HYPOTHESIS DID NOT
FIT THESE OTHER SITES.
!

100. FINAL EXAMPLE - FRESHNESS
#dmsso@TomAnthonySEO

101. COMPETITORS - MORE FREQUENT DATE ANNOTATIONS
#dmsso@TomAnthonySEO

102. GOOGLE IDENTIFIES FRESHNESS AS
IMPORTANT IN THESE RESULTS
REGULARLY MAKING SMALL UPDATES
FOR THIS CLIENT’S CONTENT WILL HELP
GOOGLE RECOGNISE FRESHNESS.
HYPOTHESIS

103. ADDED & UPDATED DATE MODIFIED
#dmsso@TomAnthonySEO
<time itemprop="dateModified"
datetime="{{date}}">
Updated on: {{date}}
</time>

104. AN 8% UPLIFT
#dmsso@TomAnthonySEO

105. RECAP &
TAKEAWAYS

106. THE MAJORITY OF SEO
RECOMMENDATIONS
HAVE LITTLE IMPACT
OBSERVATION

107. NULL/NEGATIVE CHANGES SLOW YOU DOWN
#dmsso@TomAnthonySEO

108. EVEN EXPERTS STRUGGLE
TO PREDICT THE IMPACT
OF CHANGES
OBSERVATION
ME INCLUDED!

109. PAGERANK IS ITERATIVE & NOT INTUITIVE
NOPE

110. IN ISOLATION RANKINGS ARE HARD TO PREDICT
30%
40%
50%
60%
70%
Laypeople <= 3 yrs SEO > 3 yrs SEO Coin flip

111. HYPOTHESIS
FRAMEWORK

112. MORE IMPACT & QUICKER RESULTS
#dmsso@TomAnthonySEO
RUN TESTSBUILD A HYPOTHESISGATHER DATA

113. BEGIN TESTING, START HERE…
distilled.net/diy-splittester

114. BEGIN HYPOTHESISING, START HERE…
#dmsso@TomAnthonySEO
http://dis.tl/pandaQ
http://dis.tl/serpturkey
http://dis.tl/ia-pagerank

116. THANK YOU!
@TomAnthonySEO @WillCritchlow @Dom_Woodman

117. distilledodn.com