The final great presentation at MKGO3 in Milton Keynes recently. This one went right over my head but, if you're cleverer than me you will learn something useful

1. Knowing ranking factors won’t
be enough
How to avoid losing your job to a robot
@willcritchlow - MKGO 2018

2. I’m going to tell you about a robot
that understands ranking factors
better than any of you
...but before I get to that, let’s look at a bit of history...

3. The other day I searched:

4. Unsurprisingly, I got
an answer

5. But it got me thinking
about how, in 2009,
the results would
have looked more like
this.

6. In 2009, it would have
looked more like this.
With every title
containing the
keyphrase.

7. In 2009, it would have
looked more like this.
With every title
containing the
keyphrase.
Most at the beginning.

8. OK. Maybe wikipedia
would have been #1.

9. We used to have a pretty good
understanding of ranking factors

10. My mental model for ~2009 ranking factors had
three different modes:

11. One in the
hyper-competitive
head
My mental model for ~2009 ranking factors had
three different modes:
One in the
competitive
mid-tail
...and
one
in
the
long-tail

12. One in the
hyper-competitive
head

13. Tons of perfectly on-topic
pages to choose from
One in the
hyper-competitive
head

14. So pick only perfectly-on-topic pages
One in the
hyper-competitive
head

15. (*) Page authority, but the
domain inevitably factors into
that calculation. This is why
so many homepages ranked
One in the
hyper-competitive
head
...and rank by authority (*)

16. This resulted in a mix
of homepages of
mid-size sites, and
inner pages on huge
sites
One in the
hyper-competitive
head

17. But the general way
to move up was
through increased
authority
One in the
hyper-competitive
head

18. Kind of search
result
Pages ranking To move up...
Head Homepages of mid-size
sites and inner pages of
massive sites. All
perfectly-targeted.
Improve authority.
Mid-tail
Long-tail

19. One in the
hyper-competitive
head
One in the
competitive
mid-tail

20. Wealth of ROUGHLY
on-topic pages to
choose from
One in the
competitive
mid-tail

21. PERFECTLY on-topic
could do well even on
a relatively weak site
One in the
competitive
mid-tail

22. Rank the roughly
on-topic pages by
authority x “on-topicness”
One in the
competitive
mid-tail

23. Move up with better
targeting or more
authority
One in the
competitive
mid-tail

24. Kind of search
result
Pages ranking To move up...
Head Homepages of mid-size
sites and inner pages of
massive sites. All
perfectly-targeted.
Improve authority.
Mid-tail Perfectly on-topic pages
on relatively weak sites
plus roughly on-topic on
bigger sites.
Improve targeting or
authority.
Long-tail

25. One in the
hyper-competitive
head
One in the
competitive
mid-tail
...and
one
in
the
long-tail

26. In the long-tail, a site
of arbitrary weakness
could rank if it was the
most relevant
...and
one
in
the
long-tail

27. Otherwise, massive
sites rank with
off-topic pages that
mention something
similar
...and
one
in
the
long-tail

28. Generally, move up
with better targeting
...and
one
in
the
long-tail

29. Kind of search
result
Pages ranking To move up...
Head Homepages of mid-size
sites and inner pages of
massive sites. All
perfectly-targeted.
Improve authority.
Mid-tail Perfectly on-topic pages
on relatively weak sites
plus roughly on-topic on
bigger sites.
Improve targeting or
authority.
Long-tail Arbitrarily-weak on-topic
pages and
roughly-targeted deep
pages on massive sites.
Improve targeting.

30. Kind of search
result
Pages ranking To move up...
Head Homepages of mid-size
sites and inner pages of
massive sites. All
perfectly-targeted.
Improve authority.
Mid-tail Perfectly on-topic pages
on relatively weak sites
plus roughly on-topic on
bigger sites.
Improve targeting or
authority.
Long-tail Arbitrarily-weak on-topic
pages and
roughly-targeted deep
pages on massive sites.
Improve targeting.
So that was
~2009

31. It’s not so simple any more.
Google is harder to understand these days.

32. PageRank
(the first algorithm to
use the link structure of
the web)
We know how we got to ~2009...

33. Information
retrieval
PageRank

34. Information
retrieval
PageRank Original research

35. Information
retrieval
PageRank Original research TWEAKS
...with growing complexity in subsequent years

36. Particularly this comment from a user called Kevin Lacker (@lacker):

37. I was thinking about it like it was a
math puzzle and if I just thought
really hard it would all make sense.
-- Kevin Lacker (@lacker)

38. Hey why don't you take the square
root?
-- Amit Singhal according to Kevin Lacker (@lacker)

39. oh... am I allowed to write code that
doesn't make any sense?
-- Kevin Lacker (@lacker)

40. -- Amit Singhal according to Kevin Lacker (@lacker)
Multiply by 2 if it helps, add 5,
whatever, just make things work
and we can make it make sense
later.

41. Why does this make the algorithm so
hard to understand?

42. 3 big reasons:
High-
dimension
Non-linear
Discontinuous

43. High-
dimension
Non-linear
Discontinuous

44. High-
dimension
Non-linear
Discontinuous

45. High-
dimension
Non-linear
Discontinuous

46. You might know what any one of
the levers does, but they can
interact with each other in complex
ways
This is what a high-dimensional function looks like

47. High-
dimension
Non-linear
Discontinuous

48. We sell custom cigar humidors. Our
custom cigar humidors are handmade. If
you’re thinking of buying a custom cigar
humidor, please contact our custom
cigar humidor specialists at
custom.cigar.humidors@example.com
What this needs is another mention of [cigar humidors]

49. With no mentions of [cigar] or [humidor] this
page would be unlikely to rank
And yet you can clearly go too far, and have the effect turn negative.
This is called nonlinearity.
The cigar example is taken directly from Google’s quality guidelines.

50. High-
dimension
Non-linear
Discontinuous

51. Discontinuities are steps in the
function
Think about so-called “over-optimization” tipping points

52. Let’s put all this together
into a practical example:

53. Think about category pages:
Do you recommend removing “SEO text”?
We’ve tested it, so we know the answer.

54. If you said “yes”, congratulations
(+3.1% organic sessions in a split-test)

55. Unless you’re responsible for this site
No effect / possible negative effect

56. No, but I’m still pretty good at this
You’re thinking this to yourself right now.

73. I promised to tell you about a robot
that is better than even
experienced SEOs...
Well. It turns out all we needed was a coin to flip. You’re all fired.

76. It’s only going to get worse under Sundar Pichai

77. Sundar takes over as CEO
Singhal leaves
Giannandrea takes over
Jeff Dean does Jeff Dean things

78. The original Google Translate was
the result of the work of hundreds of
engineers over 10 years.

79. Director of Translate, Macduff
Hughes said that it sounded to him
as if maybe they could pull off a
neural-network-based replacement
in three years.

80. Jeff Dean said “we can do it by the
end of the year, if we put our minds
to it”.

81. Hughes: “I’m not going to be the one
to say Jeff Dean can’t deliver speed.”

82. A month later, the work of a team of
3 engineers was tested against the
existing system. The improvement
was roughly equivalent to the
improvement of the old system over
the previous 10 years.

83. Hughes sent his team an email. All
projects on the old system were to be
suspended immediately.
[Read the whole story ]

84. Background reading:(backchannel, bloomberg)

85. How to avoid losing your job to a
robot
This is what you promised, Will.

86. Let’s start by
understanding
some robot
weaknesses

87. What’s this?

88. Ooh. Ooh.
I know this one.
-- robot

89. “It’s a leopard. I’m like 99% sure.”

90. Computers are better than humans at
classification, but struggle with adversaries
Read more about this here -- Cheetah, Leopard, Jaguar

91. We don’t fully understand all ML mistakes
See: adversarial AI

92. And when you’re trying to fool the machine...
See: adversarial AI

93. And when you’re trying to fool the machine...
See: adversarial AI

94. You get some really wild examples
See: adversarial AI

95. Lesson:
We expect adversarial abilities to
take a step backwards
They will remain good at classifying bad links but will be likely to fall
prey to weird outcomes in adversarial situations

96. We’re going to see new kinds of
bugs

97. Rules of ML [PDF] outlines engineering lessons
from getting ML into production at Google

98. That document also has a section on trying to
understand what the machines are doing

99. But human explainability may not
even be possible
Not every concept a neural network uses fits neatly into a concept for
which we have a word. It’s not clear this is a weakness per se, but...

100. ...this means that engineers won’t
always know more than we do
about why a page does or doesn’t
rank
The big knowledge gap of the future is data - clickthrough rates,
bounce rates etc.

101. So how do we fight
back?

102. Michael Lewis’ latest book is
about Kahneman and Tversky
spelling.
It recounts a story about a piece
of medical software that existed
in the 1960s.

103. It was designed to encapsulate
how a range of doctors
diagnosed stomach cancer from
x-rays.

104. It proceeded to outperform those
same doctors despite only
containing their expertise.
Real people have biases, and fool
themselves.
Encapsulate your own expert
knowledge.

105. At Distilled, we use a
methodology we call the
balanced digital scorecard.
This encapsulates our beliefs
about how to build a
high-performing business.
Applying it helps avoid our own
biases.

106. Also, while we are talking about
books, The Checklist Manifesto is
an important part of avoiding the
same cognitive biases.

107. Focus on consulting skills
I’ve written a few things about
this (DistilledU module, writing
better business documents, using
split-tests to consult better).
Use case studies and creativity.
Computers are better at
diagnosis than cure.
This means: getting things done,
convincing organizations,
applying general knowledge,
learning new things.

108. We are going to need to be
better than ever at debugging
things.
I wrote about debugging skills for
non-developers here.
A lot of the story of enterprise
consulting is going to be about
figuring out why things have
gone wrong in the face of sparse
or incorrect information from
Google.

110. Disregard expert surveys
Firstly, there are all the problems
outlined in the search result pairs
study - both in the ability of
experts to understand factors,
and in your ability to use the
information even if they do.
Secondly, they are broken with
another bias called the “law of
small numbers” from Lewis’ book.
PS - I say this as a participant in
many of them
Me

111. Equally, building your digital
strategy on what Google tells you
to do will become an even worse
idea than it already is.

112. Check out Tom Capper’s presentation on how
engineers’ statements can be misleading

113. ...and remember the confounding split-tests
It’s already not always as simple as “feature X is good”
Which all means we may need to be more independent-minded and do
more of our own research

114. This is why we have been investing so much in split-testing
Check out odn.distilled.net if you haven’t already. The team will be happy to
demo for you.
We served ~5 billion requests last quarter and recently published
everything from response times to our +£100k / month split test.

115. Let’s recap
1. Even in a world of 200+ “classical” ranking factors, humans were bad at
understanding the algorithm

116. Let’s recap
1. Even in a world of 200+ “classical” ranking factors, humans were bad at
understanding the algorithm
2. Machine learning will make this worse, and is accelerating under Sundar

117. Let’s recap
1. Even in a world of 200+ “classical” ranking factors, humans were bad at
understanding the algorithm
2. Machine learning will make this worse, and is accelerating under Sundar
3. There are things computers remain bad at, and rankings will become more
opaque even to Google engineers

118. Let’s recap
1. Even in a world of 200+ “classical” ranking factors, humans were bad at
understanding the algorithm
2. Machine learning will make this worse, and is accelerating under Sundar
3. There are things computers remain bad at, and rankings will become more
opaque even to Google engineers
4. We remain relevant by:
a. Using methodologies and checklists to capture human capabilities and
avoid our biases

119. Let’s recap
1. Even in a world of 200+ “classical” ranking factors, humans were bad at
understanding the algorithm
2. Machine learning will make this worse, and is accelerating under Sundar
3. There are things computers remain bad at, and rankings will become more
opaque even to Google engineers
4. We remain relevant by:
a. Using methodologies and checklists to capture human capabilities and
avoid our biases
b. Becoming great consultants and change agents

120. Let’s recap
1. Even in a world of 200+ “classical” ranking factors, humans were bad at
understanding the algorithm
2. Machine learning will make this worse, and is accelerating under Sundar
3. There are things computers remain bad at, and rankings will become more
opaque even to Google engineers
4. We remain relevant by:
a. Using methodologies and checklists to capture human capabilities and
avoid our biases
b. Becoming great consultants and change agents
c. Debugging the heck out of everything

121. Let’s recap
1. Even in a world of 200+ “classical” ranking factors, humans were bad at
understanding the algorithm
2. Machine learning will make this worse, and is accelerating under Sundar
3. There are things computers remain bad at, and rankings will become more
opaque even to Google engineers
4. We remain relevant by:
a. Using methodologies and checklists to capture human capabilities and
avoid our biases
b. Becoming great consultants and change agents
c. Debugging the heck out of everything
d. Avoiding being misled by experts or Google

122. Let’s recap
1. Even in a world of 200+ “classical” ranking factors, humans were bad at
understanding the algorithm
2. Machine learning will make this worse, and is accelerating under Sundar
3. There are things computers remain bad at, and rankings will become more
opaque even to Google engineers
4. We remain relevant by:
a. Using methodologies and checklists to capture human capabilities and
avoid our biases
b. Becoming great consultants and change agents
c. Debugging the heck out of everything
d. Avoiding being misled by experts or Google
Testing!

123. Questions: @willcritchlow

128. ● Mobius strip
● Confusion
● Signal box
● Cigar
● Discontinuity
● Confidence
● Burt Totaro
● Sundar Pichai
● John Giannandrea
● Chuck Norris
● Jeff Dean
● Fencing
● Keyboard
Image credits
● Go
● Robot
● Leopard print sofa
● Leopard
● Bug
● Lego robots
● Iron Man
● Roundabout