SEO focused A/B Testing or Split-Testing is fast becoming an important new technique for digital marketers. This deck explains why it is important, and how you can do it.

1. A
GUIDE TO
SPLIT-TESTING
THE NEXT
ERA
OF SEO

2. STANFORD STUDY
Frank George

3. Risk Aversity Performance
Cautious Fantastic.
Recommend for awards.
Cautious Terrible.
Close to getting fired.
FIREFIGHTER TRAITS
Frank
George

4. Students were now told:
We made it up.
Entirely fictional!
!

5. How should a firefight treat risk?
They should be cautious!
They should embrace risk.
It is a dangerous job!
STUDENTS NOW ASKED:
Frank
George

6. People cannot let
go of old beliefs.
See what they
want to see.
!

7. Have you recommended
ALT attributes on images
for SEO reasons?
?

8. ALT ATTRIBUTES GOOD FOR SEO

9. TEST RESULT: ALT ATTRIBUTES HAD NO IMPACT

10. Google algorithm
is too complex.
We are
too biased.
!

11. Blog Post

12. Blog Post

13. Blog PostBlog Post

14. HOW YOU CAN TEST — FREE TOOL

15. ‘TRADITIONAL’ CRO A/B TESTING IS USER ORIENTED
Visitors A & B:
control version
of a page
Visitors C & D:
test version
of the same page

16. SEO A/B TESTING IS PAGE ORIENTED
Pages 1, 2, 3:
Control Layout
Pages 4, 5, 6:
Test Layout
All users see the
same as each other

17. SEO A/B TESTING INCLUDES GOOGLEBOT
Googlebot sees
the same as users
Pages 1, 2, 3:
Control Layout
Pages 4, 5, 6:
Test Layout

18. OVERVIEW

19. SEO SHOULD BE INFORMED
BY DATA & TESTS
1

20. 2009: ALGORITHM IS PRETTY WELL UNDERSTOOD
No Panda
No Penguin
No RankBrain
No Machine Learning
We know what to do.

21. AMIT SINGHAL - HEAD OF SEARCH, GOOGLE

22. See this fascinating comment from a user called Kevin Lacker (@lacker):

23. I was thinking about it like it was a
math puzzle and if I just thought
really hard it would all make sense.
Kevin Lacker

24. Hey why don't you take
the square root?
Amit Singhal according to Kevin Lacker

25. I was like, why a square root, it
doesn't make much sense
Kevin Lacker

26. Multiply by 2 if it helps, add 5,
whatever, just make things work and
we can make it make sense later.
Amit Singhal according to Kevin Lacker

27. THE ALGORITHM WAS MOSTLY HANDMADE

28. ALGORITHM IS TOO COMPLEX TO VISUALISE

29. EARLY 2016: AMIT SINGHAL RETIRES

30. JOHN GIANNANDREA TAKES OVER

31. PREVIOUSLY HEAD OF MACHINE LEARNING

32. IMPACT OF MACHINE LEARNING

33. IMPACT OF MACHINE LEARNING
Translate
10 YEARS
RESEARCH
2 MONTHS
MACHINE LEARNING
Imagine the same step change in the search algo…

34. IMPACT OF MACHINE LEARNING

35. It will be extremely hard
to understand many rankings.
Even for Google.
!

36. No, but I’m still
pretty good at this
You’re thinking this to yourself right now.

37. How good are we?
?

38. Google Information Needs Study (NBED LINK)
http://adage.com/article/media/future-tv/303508/
SEARCH: future of tv advertising

39. Google Information Needs Study (NBED LINK)
http://contentmarketinginstitute.com/2016/04/television-advertising-native/
SEARCH: future of tv advertising

40. DA PA LRDs Rank
http://dis.tl/rankq-1a
91 47 2 ?
http://dis.tl/rankq-1b
86 57 9 ?
SEARCH: future of tv advertising

41. DA PA LRDs Rank
http://dis.tl/rankq-1a
91 47 2 3
http://dis.tl/rankq-1b
86 57 9 8
SEARCH: future of tv advertising

42. https://www.jensinkler.com/fix-your-kettlebell-swing/
SEARCH: kettlebell technique

43. http://www.mensfitness.com/training/pro-tips/perfect-kettlebell-swing
SEARCH: kettlebell technique

44. DA PA LRDs Rank
http://dis.tl/rankq-2a
41 42 12 ?
http://dis.tl/rankq-2b
76 44 20 ?
SEARCH: kettlebell technique

45. DA PA LRDs Rank
http://dis.tl/rankq-2a
41 42 12 4
http://dis.tl/rankq-2b
76 44 20 6
SEARCH: kettlebell technique

46. http://www.macworld.co.uk/how-to/iphone/how-to-back-up-iphone-ipad-backup-3595444/
SEARCH: do I need to backup my iphone

47. http://www.idownloadblog.com/2013/08/28/not-enough-icloud-storage-iphone-cannot-be-backed-up/
SEARCH: do I need to backup my iphone

48. DA PA LRDs Rank
http://dis.tl/rankq-3a
77 59 16 ?
http://dis.tl/rankq-3b
67 34 3 ?
SEARCH: do I need to backup my iphone

49. DA PA LRDs Rank
http://dis.tl/rankq-3a
77 59 16 7
http://dis.tl/rankq-3b
67 34 3 5
SEARCH: do I need to backup my iphone

50. How do we compare?
?

51. WE SURVEYED 2000 PEOPLE FOR 10 QUERIES

52. % OF UK SEARCH RESULT PAIRS PREDICTED CORRECTLY
30%
40%
50%
60%
70%
Laypeople

53. % OF UK SEARCH RESULT PAIRS PREDICTED CORRECTLY
30%
40%
50%
60%
70%
Laypeople <= 3 yrs SEO

54. % OF UK SEARCH RESULT PAIRS PREDICTED CORRECTLY
30%
40%
50%
60%
70%
Laypeople <= 3 yrs SEO > 3 yrs SEO

55. % OF UK SEARCH RESULT PAIRS PREDICTED CORRECTLY
30%
40%
50%
60%
70%
Laypeople <= 3 yrs SEO > 3 yrs SEO Coin flip

56. % OF UK SEARCH RESULT PAIRS PREDICTED CORRECTLY
30%
40%
50%
60%
70%
Laypeople <= 3 yrs SEO > 3 yrs SEO Coin flip DeepRank

57. TESTING - STOP GUESSING WHAT WORKED
Testing the same type of change on different sites can have differing effects.

58. THE CASE FOR TESTING
1) Predicting effect in algo is really hard (no best practice)

59. THE CASE FOR TESTING
1) Predicting effect in algo is really hard (no best practice)
2) Some sites respond differently to others for same change

60. THE CASE FOR TESTING
1) Predicting effect in algo is really hard (no best practice)
2) Some sites respond differently to others for same change
3) Machine Learning impact on algo will make this worse

61. THE CASE FOR TESTING
1) Predicting effect in algo is really hard (no best practice)
2) Some sites respond differently to others for same change
3) Machine Learning impact on algo will make this worse
4) Optimise further than previously possible

62. 2
SOME RESULTS FROM
OUR SEO TESTS

63. DistilledODN
Your Server Your Home Page
CDN
Distilled ODN
Your Optimized
Home Page

64. 2 IMPORTANT GRAPHS. THIS IS THE FIRST.

65. BLACK IS THE VARIANT, TREAT BLUE AS THE CONTROL

66. BLACK (ACTUAL) HIGHER THAN BLUE (CONTROL) - POSITIVE

67. THIS IS THE START DATE

68. SECOND GRAPH: TOTAL ADDITIONAL SESSIONS

69. SHADED AREA SHOWS CONFIDENCE INTERVAL

70. TEST IS SIGNIFICANT WHEN THE FAN CROSSES 0

71. IN THIS CASE IT HAPPENED HERE:

72. HERE THEY ARE TOGETHER.

73. ALT ATTRIBUTES
1

74. HYPOTHESIS: ALT attributes on images are good for SEO

75. RESULT: ALT attributes had no effect

76. SEO ‘CONTENT’
2

77. HYPOTHESIS: Text content on category pages is sort of good?

78. RESULT: No real impact when removed

79. RESULT: A 3.1% improvement in organic performance

80. Not every site
is the same.
Identical changes can
have different results.
!

81. JAVASCRIPT
RENDERING
3

82. HYPOTHESIS: Javascript impacts Google’s crawling / indexation
MAY 2014
“our indexing system has been rendering a substantial number of web pages more like an
average user’s browser with JavaScript turned on.
OCT 2015
“We are generally able to render and understand your web pages like modern browsers.”

83. Javascript EnabledJavascript Disabled
SITUATION: Javascript needed to render product list

84. RESULT: CSS fallback led to a 6.2% uplift

85. TITLES:
HIGHEST SEARCH VOLUME
4

86. Title tag before: Which TV should I buy? - Argos
Title tag after: Which TV to buy? - Argos
HYPOTHESIS: Title tags should match greatest search volume term

87. RESULT: Organic sessions decreased by average 8%

88. STRUCTURED MARKUP
ON CATEGORY PAGES
5

89. SITUATION: Category pages have lots of images but little text

90. HYPOTHESIS: Structured Markup per product will help

91. RESULT: Organic sessions increase by 11%

92. 3
HOW YOU CAN RUN
TESTS YOURSELVES
(+ A FREE TOOL)

93. 3 STEPS TO DIY SEO SPLIT-TESTS
1. Create two buckets of pages.
2. Make a change to all pages in one bucket.
3. Analyse which bucket performs better.

94. 1. CREATE TWO BUCKETS OF PAGES.
A good DIY approach is using GA segments. Easy to make (see link).
Suggestion: Create Segments by category (e.g. blog tag, product category).

95. At least 1,000 organic sessions
per day to each bucket
!

96. 2. MAKE A CHANGE TO ALL PAGES IN ONE BUCKET.
Control Variant
This will be your variant bucket.
Utilise your CMS where possible. Ask Dev for scalable approach.

97. 3. ANALYSE WHICH BUCKET PERFORMS BETTER.
distilled.net/diy-splittester

98. Typically takes 2 - 3 weeks
to reach significance.
!

99. READ THIS:
Blog Post

100. THEN READ THIS:
Blog Post

101. 4
SUMMARY

102. SPLIT-TESTING IS BECOMING MORE NECESSARY
30%
40%
50%
60%
70%
Laypeople <= 3 yrs SEO > 3 yrs SEO Coin flip

103. SPLIT-TESTING IS BECOMING MORE ESTABLISHED

104. SPLIT-TESTING MAKES YOU MORE CREDIBLE

105. SPLIT-TESTING IS ALREADY ACCESSIBLE

106. Step change in your
SEO approach.
Start now.
!

107. WWW.DISTILLEDODN.COM
CHECK OUT OUR SEO
A/B TESTING PLATFORM:
@TomAnthonySEO
@Dom_Woodman
@WillCritchlow
DIS.TL/SMXDE17
GET THE DECK: