Computers might already be better at your job than you are. Are you ready to partner with them to keep your job? This presentation shows you how hard our jobs have become, gives you the results of some of our testing, and outlines a plan to keep our jobs.

1. Lessons from SEO split-testing
Including evidence that we don’t know what we’re talking about
@willcritchlow

2. We used to have a pretty good
understanding of ranking factors

3. My mental model for ~2009 ranking factors had
three different modes:

4. One in the
hyper-competitive
head
My mental model for ~2009 ranking factors had
three different modes:
One in the
competitive
mid-tail
...and
one
in
the
long-tail

5. One in the
hyper-competitive
head

6. Tons of perfectly on-topic
pages to choose from
One in the
hyper-competitive
head

7. So pick only perfectly-on-topic pages
One in the
hyper-competitive
head

8. (*) Page authority, but the
domain inevitably factors into
that calculation. This is why
so many homepages ranked
One in the
hyper-competitive
head
...and rank by authority (*)

9. This resulted in a mix
of homepages of
mid-size sites, and
inner pages on huge
sites
One in the
hyper-competitive
head

10. But the general way
to move up was
through increased
authority
One in the
hyper-competitive
head

11. Kind of search
result
Pages ranking To move up...
Head Homepages of mid-size
sites and inner pages of
massive sites. All
perfectly-targeted.
Improve authority.
Mid-tail
Long-tail

12. One in the
hyper-competitive
head
One in the
competitive
mid-tail

13. Wealth of ROUGHLY
on-topic pages to
choose from
One in the
competitive
mid-tail

14. PERFECTLY on-topic
could do well even on
a relatively weak site
One in the
competitive
mid-tail

15. Rank the roughly
on-topic pages by
authority x “on-topicness”
One in the
competitive
mid-tail

16. Move up with better
targeting or more
authority
One in the
competitive
mid-tail

17. Kind of search
result
Pages ranking To move up...
Head Homepages of mid-size
sites and inner pages of
massive sites. All
perfectly-targeted.
Improve authority.
Mid-tail Perfectly on-topic pages
on relatively weak sites
plus roughly on-topic on
bigger sites.
Improve targeting or
authority.
Long-tail

18. One in the
hyper-competitive
head
One in the
competitive
mid-tail
...and
one
in
the
long-tail

19. In the long-tail, a site
of arbitrary weakness
could rank if it was the
most relevant
...and
one
in
the
long-tail

20. Otherwise, massive
sites rank with
off-topic pages that
mention something
similar
...and
one
in
the
long-tail

21. Generally, move up
with better targeting
...and
one
in
the
long-tail

22. Kind of search
result
Pages ranking To move up...
Head Homepages of mid-size
sites and inner pages of
massive sites. All
perfectly-targeted.
Improve authority.
Mid-tail Perfectly on-topic pages
on relatively weak sites
plus roughly on-topic on
bigger sites.
Improve targeting or
authority.
Long-tail Arbitrarily-weak on-topic
pages and
roughly-targeted deep
pages on massive sites.
Improve targeting.

23. Kind of search
result
Pages ranking To move up...
Head Homepages of mid-size
sites and inner pages of
massive sites. All
perfectly-targeted.
Improve authority.
Mid-tail Perfectly on-topic pages
on relatively weak sites
plus roughly on-topic on
bigger sites.
Improve targeting or
authority.
Long-tail Arbitrarily-weak on-topic
pages and
roughly-targeted deep
pages on massive sites.
Improve targeting.
So that was
~2009

24. It’s not so simple any more.
Google is harder to understand these days.

25. PageRank
(the first algorithm to
use the link structure of
the web)
We know how we got to ~2009...

26. Information
retrieval
PageRank

27. Information
retrieval
PageRank Original research

28. Information
retrieval
PageRank Original research TWEAKS
...with growing complexity in subsequent years

29. Particularly this comment from a user called Kevin Lacker (@lacker):

30. I was thinking about it like it was a
math puzzle and if I just thought
really hard it would all make sense.
-- Kevin Lacker (@lacker)

31. Hey why don't you take the square
root?
-- Amit Singhal according to Kevin Lacker (@lacker)

32. oh... am I allowed to write code that
doesn't make any sense?
-- Kevin Lacker (@lacker)

33. -- Amit Singhal according to Kevin Lacker (@lacker)
Multiply by 2 if it helps, add 5,
whatever, just make things work
and we can make it make sense
later.

34. No, but I’m still pretty good at this
You’re thinking this to yourself right now.

51. I can beat you with a simple
machine
Well. It turns out all we needed was a coin to flip. You’re all fired.

55. It’s only going to get worse under Sundar Pichai

56. So we need to test.
Here’s how SEO split tests work

60. You might have seen @TomAnthonySEO tweeting
about the platform we’ve built to make this easy

62. Excuse a brief diversion into geeky
details

63. Instead of comparing the performance of the control pages directly with the variant pages, we build a
forecast of what’s called the counterfactual which is an estimate of what would have happened if we hadn’t
made the change. We use the control group to make a counterfactual forecast that takes into account
seasonality and site-wide changes.
The black line on the chart above is the actual organic traffic to the variant pages. The blue line is the
counterfactual.
More: Distilled blog post and free forecasting tool

64. It’s easiest to analyse the results by looking at the cumulative difference over time between the actual
organic traffic and the counterfactual.
The pale blue area is the 95% confidence interval.
We can see a (statistically) zero effect for an initial time while Google crawls and indexes the test,
followed by steady growth. A couple of weeks in, the confidence interval goes above zero and we have a
winning test.
More: Distilled blog

65. It’s easiest to analyse the results by looking at the cumulative difference over time between the actual
organic traffic and the counterfactual.
The pale blue area is the 95% confidence interval.
We can see a (statistically) zero effect for an initial time while Google crawls and indexes the test,
followed by steady growth. A couple of weeks in, the confidence interval goes above zero and we have a
winning test.
More: Distilled blog
Hashtag winning

66. Further reading for those interested:
● Predicting the present with Bayesian structural time series [PDF]
● Inferring causal impact using Bayesian structural time series [PDF]
● CausalImpact R package
● Finding the ROI of title tag changes
More: Distilled blog

67. 1. Adding structured data
2. Adding ALT attributes
3. Setting exact match title tags
4. Using JS to show content
5. Removing SEO category text

68. Thanks Dom
@dom_woodman

69. 1. Adding structured data
2. Adding ALT attributes
3. Setting exact match title tags
4. Using JS to show content
5. Removing SEO category text

70. Category pages have lots of images and not much text

71. Adding structured data to category pages

72. Organic sessions increased by 11%

73. 1. Adding structured data
2. Adding ALT attributes
3. Setting exact match title tags
4. Using JS to show content
5. Removing SEO category text

74. Established wisdom would say ALT tags on images are good for SEO

75. We found adding ALT tags to images had no effect

76. 1. Adding structured data
2. Adding ALT attributes
3. Setting exact match title tags
4. Using JS to show content
5. Removing SEO category text

77. Title tag before: Which TV should I buy? - Argos
Title tag after: Which TV to buy? - Argos
What happens when you match title tags to the greatest search volume?

78. Organic sessions decreased by an average of 8%

79. 1. Adding structured data
2. Adding ALT attributes
3. Setting exact match title tags
4. Using JS to show content
5. Removing SEO category text

80. We can render Javascript!

81. What happens if your content is only visible with Javascript?
Javascript EnabledJavascript Disabled

82. Making it visible increased organic sessions by ~ 6.2%

83. Read more on our blog: early results from split-testing JS for SEO

84. 1. Adding structured data
2. Adding ALT attributes
3. Setting exact match title tags
4. Using JS to show content
5. Removing SEO category text

85. How does SEO text on category pages perform?

86. E-commerce site number 1 ~ 3.1% increase in organic sessions

87. E-commerce site number 2 - No effect/negative effect

88. This is why we have been investing so much in split-testing
Check out odn.distilled.net if you haven’t already. The team will be happy to
demo for you.
We served ~5 billion requests last quarter and recently published
everything from response times to our +£100k / month split test.

89. But I’m also seeing more subtle impacts on my recommendations:
● You can recommend small tweaks and see the benefits compound
● You can test wild hypotheses with unknown upsides
● You can try things that might have a downside (more focused targeting, less copy, etc.)
And that’s even before you get the benefits of testing clickthrough rate, and the benefits of pretty charts
to show the boss highlighting the impact of your work!
More: blog post

90. Oh, and one more thing

91. Can we build a better machine than
a coin flip?
Seems like a low bar

96. The goal is a winning combination
of human and machine
Human + computer beats computer (for now)

97. Questions: @willcritchlow

98. ● Mobius strip
● Confusion
● Signal box
● Cigar
● Discontinuity
● Confidence
● Burt Totaro
● Sundar Pichai
● John Giannandrea
● Chuck Norris
● Jeff Dean
● Fencing
● Keyboard
Image credits
● Go
● Robot
● Leopard print sofa
● Leopard
● Bug
● Lego robots
● Iron Man
● Leeds