SEO split testing involves testing changes to page templates to determine their impact on organic traffic. For example, an animal template could be tested by replacing images with videos on some pages while leaving others unchanged. These variant and control pages are tracked over time. If the variant pages outperform controls, showing the change improved rankings, the change is adopted site-wide. By comparing variant to control page performance, the impact of the change can be isolated from external factors like seasonality that would affect both groups.

1. What is SEO split testing?

2. One quick thing: This is deliberately a simple example with a basic explanation of the maths that we use. In
reality, the maths is a lot more complicated and based on this research by Google: Inferring causal impact
using Bayesian structural time series.
The purpose of this presentation isn’t to teach or explain the maths behind the ODN, it’s to, hopefully,
explain the core concepts in a simple way, that allows you to imagine applying this methodology to websites
with a lot more than 4 sub-category pages :)
If you want to dig into the testing methodology in detail, then you can visit:
https://odn.distilled.net/learn-more/faqs/

3. Imagine this example website
It has two categories (animals and
countries)
It has 8 sub-category pages (cats,
dogs, Scotland etc.)
All of the animals sub-category
pages use the same template and all
of the countries sub-category pages
use another template

4. SEO split testing is centered on the concept of
testing changes to page templates.
In the animals sub-category example to the left
you can see that although the content of each
page is different, they all follow the same
template. They have an H1 at the top of the
page, a block of intro copy then a featured
image.
A group of pages that share the same template
can be used for SEO split-testing
Cats Dogs Unicorns Badgers
CatsH1
Intro copy
Featured image

5. We could create an animal template test

6. Or a countries template test

7. But you can’t mix templates

8. For demonstration purposes, imagine that we
wanted to test replacing the image with a video
and removing the intro copy from the animals
sub-category template.
Once the new template design is finalised, the
next step is to decide which of the animal
sub-category URLs remain on the control
template and which are changed to the variant
template.
CatsH1
Intro copy
Featured image
CatsH1
Featured video
Old design
Proposed design

9. Cats
Dogs Unicorns
Badgers Cats Badgers
Dogs Unicorns
Distilled ODN
Distilled’s ODN platform uses
advanced maths to decide which
URLs should remain on the
control template and which
should get the variant template.
For simplicity in this example, you
can think of this as selecting URLs
at random to be on each template.
Picking control and variant URLs

10. The site looks like this during the test
Home
Animals
Cats Dogs
Countries
Scotland England Ireland WalesUnicorns Badgers

11. Measuring the results
1000 -
5000 -
4000 -
3000 -
2000 -
6000 -
7000 -
Control
group
Variant
group
Before the test begins, we compare
how organic traffic to the control group
of pages and the variant group of pages
has performed historically.

12. Measuring the results
1000 -
5000 -
4000 -
3000 -
2000 -
6000 -
7000 -
Control
group
Variant
group
We then roll out the template change to X% of
pages within the site section. To keep things
simple, let’s assume this is 50% of pages
Test start date

13. Measuring the results
1000 -
5000 -
4000 -
3000 -
2000 -
6000 -
7000 -
Control
group
Variant
group
During the first week or two, we don’t
expect to see much change. Google needs
time to crawl the pages that have the new
template and to and decide whether the
change positive or negative.
Test start date

14. Measuring the results
1000 -
5000 -
4000 -
3000 -
2000 -
6000 -
7000 -
Control
group
Variant
group
Over time we notice that the variant pages start to
outperform the control pages.
We can declare the test a success and recommend
that the changes be rolled out to 100% of pages.
Test start date

15. By having a control group of
pages that have the same
intent/theme/template we can
exclude external factors like
seasonality because the control
group of pages would also be
impacted.
The analysis isn’t looking at the
trend of the traffic; it’s looking at
the difference in performance
between the control group and
the variant.
How can we be sure it wasn’t...
Seasonality
Google rolled out an update
Competitors’ performance decreases
Backlinks to your site
TV campaigns
Branding/direct traffic
Other macro factors

16. Positive changes look like this
1000 -
5000 -
4000 -
3000 -
2000 -
6000 -
7000 -
Control
group
Variant
group
Test start date
Variant group
outperforms the control
group

17. Negative changes look like this
1000 -
5000 -
4000 -
3000 -
2000 -
6000 -
7000 -
Control
group
Variant
group
Test start date
Control group
outperforms the variant
group

18. Seasonality would look like this
1000 -
5000 -
4000 -
3000 -
2000 -
6000 -
7000 -
Control
group
Variant
groupTest start date

19. Seasonality would look like this
1000 -
5000 -
4000 -
3000 -
2000 -
6000 -
7000 -
Control
group
Variant
groupTest start date
Although there is an upward trend, the difference in
performance between the control pages and the
variant pages is the same as before the test began.
This would be declared as a neutral test.

20. Want to know more?

21. Find out more at odn.distilled.net