The Evolution of SEO: Split-Testing for Search, by Rob Ousbey

The Evolution of SEO:
Split-Testing For Search
@RobOusbey

Everything Changes
When you’re newer to an industry / an organization
/ a department, you have the opportunity to ask
questions like “why do we do this thing?” and “why do
we do it this way?” and you don’t put up with the
answer “because that’s the way we’ve always done it..”

@RobOusbey @Distilled
Everything Changes

SEO is Changing
15 years ago, the ranking factors we talked about
were almost exclusively around term-relevance and
the link-based authority of the site & page.

SEO is Changing
Is the page
relevant to the
term?
Is the page/site
authoritative?

SEO is Changing
The ‘ranking factors’ were determined and tweaked
by a team of humans inside Google.

SEO is Changing
In response, we ran correlation studies to try to
reverse engineer what they were doing.

Google’s Goal:
Portal to the Internet
Google’s aspiration was to be your jumping-off point
onto the rest of the web. If you clicked through to a
site and had a good internet experience, it reflected
well on them. And if you had a bad experience, it
reflected poorly on them for sending you to that site.

Malware & viruses
Paywall / registration-wall on landing page
Out of stock products
Out of date information
Pages that make it hard to find information
Pages that are full of advertising
Slow Website
Bad User Experiences

Measuring User Satisfaction
They tried assessing the users’ satisfaction with
results by directly asking us what we thought.

Google Has More and More Data
But Google ended up being able to collect more
data about users and their paths than anyone ever
imagined.

Google Has More and More Data
(Though we don’t know which of these data sources
they’ve ever used as direct or indirect inputs.)

Google is Comfortable with Machine Learning
https://www.wired.com/2016/06/how-google-is-remaking-itself-as-a-machine-learning-first-company/
They also become much more amenable to using
machine learning to process all of this data. This has
better allowed user satisfaction to be measured
directly. A good example would be if a site ranked
number one, but everyone who clicked on it then
clicked back to the results and selected something
else, then maybe it shouldn’t be ranking there.

What We’re Seeing Now

Contemporary SEO
Page 1 Page 2+
For any search, most of the clicks are on page 1,
rather than page 2 and beyond.

Contemporary SEO
Page 1 Page 2+
Very little
user data
Loads of
user data
So there’s a lot more user data available to Google
about the engagement with page 1. And there’s very
little data about the deeper results.

Contemporary SEO
Page 1 Page 2+
Some
correlation
with links
Data via: distilled.net
On pages 2+, where there’s not much user data, we
see a pretty reasonable correlation between links
and rankings.

Contemporary SEO
Page 1 Page 2+
Some
correlation
with links
Less
correlation
with links
But at the bottom of page 1, the correlation is
weaker.

Contemporary SEO
Page 1 Page 2+
Some
correlation
with links
Weakest
correlation
with links
And at the top of the first page, there’s barely any
correlation between rankings & links!
Less
correlation
with links

Contemporary SEO
Page 1
And…
this correlation is getting
weaker every year.
Less
correlation
with links
Weakest
correlation
with links

Contemporary SEO
SearchVolume
Very little
user data
Loads of
user data
When we think about the head & long tail of
keywords, we know that there is much more user
data available for head terms than then long tail.

Contemporary SEO
SearchVolume
Rankings more based on
user engagement metrics
Rankings are
more link-based
And again, we see that the long tail rankings are
more link based. High volume head terms are more
dictated by user engagement, brand strength, etc.

Contemporary SEO
Larry Kim’s research into click-through-rate’s influence on rankings:
moz.com/blog/higher-organic-click-through-conversion-rates-rankings

Contemporary SEO
Brian Dean’s conclusions about the impact of bounce rate on rankings:
backlinko.com/search-engine-ranking

Contemporary SEO
Searchmetrics’ analysis of time on site’s correlation with rankings:
www.searchmetrics.com/ranking-factors

Contemporary SEO
Google have even been somewhat open about the
fact that they are at least looking at these kinds of
metrics.

Contemporary SEO
“The team ran through various data points, like
what percent of users clicked through a
picture-link and then quickly clicked back (a
bad sign), or whether there was a significant
increase in the time until they made their first
interaction with the results (also bad).”
https://www.cnbc.com/2018/09/17/google-tests-changes-to-its-search-algorithm-how-search-works.html

User Metrics as Ranking Factors
When we see a site like this, with pop-ups and ads
obscuring the content, we would expect to see
search visibility declining.

Which is exactly what’s happened.

This site added a registration form before you could
see any of the content.

And they have been losing search visibility.

@RobOusbey
Forbes is on another level.

@RobOusbey
I would imagine that people are clicking back away
from Forbes, to get back to the search results. Or
possibly even avoiding clicking on a Forbes result in
the first place.

SEO is Changing
700
new linking sites,
every day
And even though they’re still earning a lot of links….

SEO is Changing
35%
y/y decline
… they’ve lost 35% search visibility over the last year.

SEO is Changing
https://moz.com/blog/how-often-does-google-update-its-algorithm
It’s still not a cliché to say that SEO is changing.

The SEO Process
But I don’t think our standard SEO processes have
kept up with all of this.

The SEO Process
Traditionally, we’d read a bunch of articles with ideas
for things we needed to do.

The SEO Process
2018 20162014
Or watch these “so-called experts” on stage at
conferences, talking about what worked for them.

The SEO Process
Then you come up with a bunch of ideas.

The SEO Process
And – hopefully – get them all implemented.

The SEO Process
Outcome:
14% traffic increase
With any luck, traffic goes up!

The SEO Process
+2% +2% +2% +2% +2%
Outcome:
+2% +2%
But do you know if each change contributed an
equal amount?

The SEO Process
0% 0% 0% 0% 0%
Outcome:
+8% +6%
Or did only some of the changes have an impact?

The SEO Process
0% 0% -1% -3% -4%
Outcome:
+12% +10%
Or – worse still – what if some of the changes
actually harmed traffic? (But the success of the other
changes masked this effect.)

The SEO Process
0% 0% -1% -3% -4%
Outcome:
+12% +10%
If you’d known what the impact of each change was going to be, you would have only applied the successful ones! You’d have
saved time & effort in implementation, and had MORE traffic!
The SEO process of the future would let us isolate every single change, and measure the impact of them separately.

Testing the Impact of SEO Changes
Before
500k visits
/month
How do we test the impact on any one change to a
website? Here’s how you might go about it…

Before
500k visits
/month

Before
500k visits
/month
After
600k visits
/month
You could look at how much traffic you had before
and after making the change to your site.

But that is nonsense.

If your site is very seasonal, this won’t work.

If your industry has a general trend, it won’t work.

If there are any UX or general SERP changes during
the period of your test, it will throw off your data.

We need to test against a parallel universe
You really need to be able to compare two parallel
universes: one where we made the change, and one
where we didn’t do anything.

We need to test against a parallel universe

Animals
Dogs UnicornsCats Badgers
Let’s use this as the section of a demo site.

H1 Badgers
Old Design
Intro Copy
Image

Badgers
H1 Badgers
Old Design
Intro Copy
Image
H1
Proposed Design
Video
Bullet List

Animals
Dogs UnicornsCats Badgers
We’ll keep half the site as it is at the moment…

Animals
Cats Dogs Unicorns Badgers
… and apply our proposed changes to the other half
of the site.

Control group
Variant group
1000 -
5000 -
4000 -
3000 -
2000 -
6000 -
7000 -
Before the change, traffic to each group will be able
the same.

Control group
Variant group
1000 -
5000 -
4000 -
3000 -
2000 -
6000 -
7000 -
Test start date
Then we roll those changes out to the variant group.

Control group
Variant group
1000 -
5000 -
4000 -
3000 -
2000 -
6000 -
7000 -
Test start date
Then we wait for Google to crawl/index the updated
pages.

Control group
Variant group
Test start date
1000 -
5000 -
4000 -
3000 -
2000 -
6000 -
7000 -
We’re not looking for the change in traffic to the variant group,
but to see if there’s a gap between the variant and control group.

1000 -
5000 -
4000 -
3000 -
2000 -
6000 -
7000 -
In this case, traffic to the variant group moved above
the control group, so we know the change was an
effective one.

Positive Test
Roll out to
whole site
There are three potential outcomes from a test.

Positive Test
Roll out to
whole site
Revert the
changes
Negative Test

Positive Test
Roll out to
whole site
Null Test
Make an
informed decision
Revert the
changes
Negative Test

Split pages
into equal groups
Apply changes to
variant group
Create predictive
model for traffic to
variant group
Analyze impact of
the changes
This is the process we are running.

Distilled has developed a tool to do all of this for you.
We call it our “Optimization Delivery Network”:
DistilledODN.com

What We’ve Learned

I’m going to share some examples from clients who
are under NDA, but I’m going to use BestBuy.com as
the site to take screenshots from for these demos.

Testing SEO Best Practices
Image Alt Tags

HYPOTHESIS: ALT attributes on images are good for SEO

Control:
Product page images had no alt text
Image Alt Text Test

Control:
Variant:
Product name added as image alt text
Image Alt Text Test

Control:
Variant:
Product name added as image alt text
Result:
no change in traffic
Image Alt Text Test

“Best practices”
are not so clear cut.

Revisiting Old SEO Practices
2

HYPOTHESIS: Text content on category pages is goodCategory Page SEO Content Test

Category Page SEO Content Test

Control:
Category pages still have ‘SEO content’

Control:
Variant:
‘SEO content’ removed from these pages

Control:
Variant:
Result:
no change in traffic

Control:
Variant:
Result:
3.1% increase in traffic

Control:
Variant:
Result:
3.8% decrease in traffic

No change has the same
impact on any two sites

Structured data before Structured data after
Structured Markup Tests

Control:
Category pages had no structured data

Control:
Variant:
Structured data added to pages

Control:
Variant:
Result:
11% increase in traffic

$1.9 million in annual revenue
from the additional traffic to these category pages.

Your Competitors Hate This! !
These nerds found one
weird trick that can add
11% organic traffic to any
website’s category page!
LEARN THE TRICK

Control:
Variant:
Result:
No change in traffic

Periodically challenge
your beliefs.

Learning About SEO Practicalities
4

Title Tag / Meta Description Tests
Before:
After:

Control:
Title tags remain focusses on main topic
Variant:
Title tag mentions other topics covered on page
Result:
10% decrease in traffic

6%: Positive Tests
37%: Null Tests 57%: Negative Tests

Typical results: 5% - 15%

Typical timeline: 2-4 days

Google can discover and
& index changes quickly

Learning the Risk of Testing
5

Buy Dresses
Buy Hats
Buy Belts
More Title Tag Tests
Online
Online
Online

Removed
“online”

Removed
“online”
Re-added
“online”

The effects of changes
are reversible

Javascript Enabled Javascript Disabled
Page rendering before
Javascript Visibility Test

MAY 2014
“our indexing system has been rendering a
substantial number of web pages more like an
average user’s browser with JavaScript turned on."
OCT 2015
“We are generally able to render and understand
your web pages like modern browsers.”

Control:
Pages still require JS to render

Control:
Variant:
Content is loaded directly in HTML

Control:
Variant:
Content is loaded directly in HTML
Result:
6.2% increase in traffic

Google is not the most
reliable source of truth

Fresh Content Tests
<time itemprop="dateModified" datetime="2016-06-28 00:16:00">
Updated on: June 28th, 2016
</time>

Fresh Content Tests
<time itemprop="dateModified" datetime="{{ today }}">
Updated on: {{ today }}
</time>

Control:
Pages had honest dateModified dates
Fresh Content Test

Control:
Variant:
Pages had dateModified set to today’s date
Fresh Content Test

Control:
Variant:
Pages had dateModified set to today’s date
Result:
8% increase in traffic
Fresh Content Test

Getting Started

Do It Yourself
Read one of our very early posts about how to run
these SEO split tests, here on the Moz Blog.

Free Data Analysis Software
Read how to do everything yourself, using our data
analysis tool, here on the Distilled blog.

Interdisciplinary Team
Analytics SEO Stats
Front EndBack End

DistilledODN.com
Did I mention we have a tool that does all this for you?
DistilledODN.com

Let’s Do This

Split-testing is becoming more necessary
SearchVolume
Rankings more based on
user engagement metrics

Split-testing is becoming more established

Split-testing makes you more credible
0% 0% -1% -3% -4%+12% +10%

Split-testing is already accessible

DISTILLED.NET
SEE ALL OUR FREE VIDEOS
AND BLOG POSTS:
CHECK OUT OUR
SEO A/B TESTING PLATFORM:
DISTILLEDODN.COM
@RobOusbey

Photo Credits
Film Rolls: Colby Stopa on Visual hunt / CC BY
Face Icons: Icon made by Freepik from www.flaticon.com
Google and the Google logo are registered trademarks of Google LLC, used with permission.

The Evolution of SEO: Split-Testing for Search, by Rob Ousbey

The Evolution of SEO: Split-Testing for Search, by Rob Ousbey

Recommended

More Related Content

Recently uploaded

Recently uploaded(20)

Featured

Featured(20)

The Evolution of SEO: Split-Testing for Search, by Rob Ousbey