SEO has always sat at the intersection between being a science and an art. We all love to try out new ideas and try to understand what makes the search engines tick, but it can be frustrating to have to cut through the guesswork and speculation just to figure out what Google really wants from us. Even worse, we still find ourselves making SEO changes, seeing uplifts, but then not knowing which changes actually had any impact. Fortunately, new software and better technologies now make it possible to run proper SEO-focused tests and, for the first time, actually measure the impact that each SEO change has on our site. Rob will share these techniques, discuss some of the experiments that Distilled has been running, reveal the unexpected things they’ve learned along the way, and share how you can start running experiments yourself.

1. The Evolution of SEO:
Split-Testing For Search
@RobOusbey

2. Have you recommended
ALT attributes on images
for SEO reasons?
?

3. ALT ATTRIBUTES ARE GOOD FOR SEO

4. TEST RESULT: ALT ATTRIBUTES HAD NO IMPACT

5. Google algorithm
is too complex.
We are
too biased.
!

6. Blog Post

7. Blog Post

8. Blog PostBlog Post

9. HOW YOU CAN TEST — FREE TOOL

10. ‘TRADITIONAL’ CRO A/B TESTING IS USER ORIENTED
Visitors A & B:
control version
of a page
Visitors C & D:
test version
of the same page

11. SEO A/B TESTING IS PAGE ORIENTED
Pages 1, 2, 3:
Control Layout
Pages 4, 5, 6:
Test Layout
All users see the
same as each other

12. SEO A/B TESTING INCLUDES GOOGLEBOT
Googlebot sees
the same as users
Pages 1, 2, 3:
Control Layout
Pages 4, 5, 6:
Test Layout

13. OVERVIEW

14. SEO SHOULD BE INFORMED
BY DATA & TESTS
1

15. 2009: ALGORITHM IS PRETTY WELL UNDERSTOOD
No Panda
No Penguin
No RankBrain
No Machine Learning
We know what to do.

16. AMIT SINGHAL - HEAD OF SEARCH, GOOGLE

17. See this fascinating comment from a user called Kevin Lacker (@lacker):

18. I was thinking about it like it was a
math puzzle and if I just thought really
hard it would all make sense.
Kevin Lacker

19. Hey why don't you take
the square root?
Amit Singhal according to Kevin Lacker

20. I was like, why a square root, it doesn't
make much sense
Kevin Lacker

21. Multiply by 2 if it helps, add 5,
whatever, just make things work and
we can make it make sense later.
Amit Singhal according to Kevin Lacker

22. THE ALGORITHM WAS MOSTLY HANDMADE

23. ALGORITHM IS TOO COMPLEX TO VISUALISE

24. EARLY 2016: AMIT SINGHAL RETIRES

25. JOHN GIANNANDREA TAKES OVER

26. PREVIOUSLY HEAD OF MACHINE LEARNING

27. IMPACT OF MACHINE LEARNING

28. IMPACT OF MACHINE LEARNING
Translate
10 YEARS
RESEARCH
2 MONTHS
MACHINE LEARNING
Imagine the same step change in the search algo…

29. IMPACT OF MACHINE LEARNING

30. It will be extremely hard
to understand many rankings.
Even for Google.
!

31. No, but I’m still
pretty good at this
You’re thinking this to yourself right now.

32. How good are we?
?

33. Google Information Needs Study (NBED LINK)
http://adage.com/article/media/future-tv/303508/
SEARCH: future of tv advertising

34. Google Information Needs Study (NBED LINK)
http://contentmarketinginstitute.com/2016/04/television-advertising-native/
SEARCH: future of tv advertising

35. DA PA LRDs Rank
http://dis.tl/rankq-1a
91 47 2 ?
http://dis.tl/rankq-1b
86 57 9 ?
SEARCH: future of tv advertising

36. DA PA LRDs Rank
http://dis.tl/rankq-1a
91 47 2 3
http://dis.tl/rankq-1b
86 57 9 8
SEARCH: future of tv advertising

37. https://www.jensinkler.com/fix-your-kettlebell-swing/
SEARCH: kettlebell technique

38. http://www.mensfitness.com/training/pro-tips/perfect-kettlebell-swing
SEARCH: kettlebell technique

39. DA PA LRDs Rank
http://dis.tl/rankq-2a
41 42 12 ?
http://dis.tl/rankq-2b
76 44 20 ?
SEARCH: kettlebell technique

40. DA PA LRDs Rank
http://dis.tl/rankq-2a
41 42 12 4
http://dis.tl/rankq-2b
76 44 20 6
SEARCH: kettlebell technique

41. http://www.macworld.co.uk/how-to/iphone/how-to-back-up-iphone-ipad-backup-3595444/
SEARCH: do I need to backup my iphone

42. http://www.idownloadblog.com/2013/08/28/not-enough-icloud-storage-iphone-cannot-be-backed-up/
SEARCH: do I need to backup my iphone

43. DA PA LRDs Rank
http://dis.tl/rankq-3a
77 59 16 ?
http://dis.tl/rankq-3b
67 34 3 ?
SEARCH: do I need to backup my iphone

44. DA PA LRDs Rank
http://dis.tl/rankq-3a
77 59 16 7
http://dis.tl/rankq-3b
67 34 3 5
SEARCH: do I need to backup my iphone

45. How do we compare?
?

46. WE SURVEYED 2000 PEOPLE FOR 10 QUERIES

47. % OF SERP PAIRS PREDICTED CORRECTLY
30%
40%
50%
60%
70%
Laypeople

48. % OF SERP PAIRS PREDICTED CORRECTLY
30%
40%
50%
60%
70%
Laypeople <= 3 yrs SEO

49. % OF SERP PAIRS PREDICTED CORRECTLY
30%
40%
50%
60%
70%
Laypeople <= 3 yrs SEO > 3 yrs SEO

50. % OF SERP PAIRS PREDICTED CORRECTLY
30%
40%
50%
60%
70%
Laypeople <= 3 yrs SEO > 3 yrs SEO Coin flip

51. % OF SERP PAIRS PREDICTED CORRECTLY
30%
40%
50%
60%
70%
Laypeople <= 3 yrs SEO > 3 yrs SEO Coin flip DeepRank

52. TESTING - STOP GUESSING WHAT WORKED
Testing the same type of change on different sites can have differing effects.

53. THE CASE FOR TESTING
1) Predicting effect in algo is really hard (no best practice)

54. THE CASE FOR TESTING
1) Predicting effect in algo is really hard (no best practice)
2) Some sites respond differently to others for same change

55. THE CASE FOR TESTING
1) Predicting effect in algo is really hard (no best practice)
2) Some sites respond differently to others for same change
3) Machine Learning impact on algo will make this worse

56. THE CASE FOR TESTING
1) Predicting effect in algo is really hard (no best practice)
2) Some sites respond differently to others for same change
3) Machine Learning impact on algo will make this worse
4) Optimise further than previously possible

57. 2
SOME RESULTS FROM
OUR SEO TESTS

58. DistilledODN
Your Server Your Home Page
CDN
Distilled ODN
Your Optimized
Home Page

59. TWO IMPORTANT GRAPHS. THIS IS THE FIRST.

60. BLACK IS THE VARIANT, BLUE IS THE CONTROL

61. VARIANT HIGHER THAN CONTROL – POSITIVE RESULT

62. THIS IS THE START DATE

63. SECOND GRAPH: TOTAL ADDITIONAL SESSIONS

64. SHADED AREA SHOWS CONFIDENCE INTERVAL

65. TEST IS SIGNIFICANT WHEN THE FAN CROSSES 0

66. IN THIS CASE IT HAPPENED HERE:

67. HERE THEY ARE TOGETHER.

68. ALT ATTRIBUTES
1

69. HYPOTHESIS: ALT attributes on images are good for SEO

70. RESULT: ALT attributes had no effect

71. SEO ‘CONTENT’
2

72. HYPOTHESIS: Text content on category pages is sort of good?

73. RESULT: No real impact when removed

74. RESULT: A 3.1% improvement in organic performance

75. Not every site
is the same.
Identical changes can
have different results.
!

76. JAVASCRIPT
RENDERING
3

77. HYPOTHESIS: Javascript impacts Google’s crawling / indexation
MAY 2014
“our indexing system has been rendering a substantial number of web pages more like an average
user’s browser with JavaScript turned on.
OCT 2015
“We are generally able to render and understand your web pages like modern browsers.”

78. Javascript EnabledJavascript Disabled
SITUATION: Javascript needed to render product list

79. RESULT: CSS fallback led to a 6.2% uplift

80. TITLES:
HIGHEST SEARCH VOLUME
4

81. Title tag before: Which TV should I buy? - Argos
Title tag after: Which TV to buy? - Argos
HYPOTHESIS: Title tags should match greatest search volume term

82. RESULT: Organic sessions decreased by average 8%

83. STRUCTURED MARKUP
ON CATEGORY PAGES
5

84. SITUATION: Category pages have lots of images but little text

85. HYPOTHESIS: Structured Markup per product will help

86. RESULT: Organic sessions increase by 11%

87. 3
HOW YOU CAN RUN
TESTS YOURSELVES
(+ A FREE TOOL)

88. 3 STEPS TO DIY SEO SPLIT-TESTS
1. Create two buckets of pages.
2. Make a change to all pages in one bucket.
3. Analyse which bucket performs better.

89. 1. CREATE TWO BUCKETS OF PAGES.
A good DIY approach is using GA segments. Easy to make (see link).
Suggestion: Create Segments by category (e.g. blog tag, product category).

90. At least 1,000 organic sessions
per day to each bucket
!

91. 2. MAKE A CHANGE TO ALL PAGES IN ONE BUCKET.
Control Variant
This will be your variant bucket.
Utilise your CMS where possible. Ask Dev for scalable approach.

92. 3. ANALYZE WHICH BUCKET PERFORMS BETTER.
distilled.net/diy-splittester

93. Typically takes 2 - 3 weeks
to reach significance.
!

94. READ THIS:
Blog Post

95. THEN READ THIS:
Blog Post

96. 4
SUMMARY

97. SPLIT-TESTING IS BECOMING MORE NECESSARY
30%
40%
50%
60%
70%
Laypeople <= 3 yrs SEO > 3 yrs SEO Coin flip

98. SPLIT-TESTING IS BECOMING MORE ESTABLISHED

99. SPLIT-TESTING MAKES YOU MORE CREDIBLE

100. SPLIT-TESTING IS ALREADY ACCESSIBLE

101. This is the next evolution
in your SEO approach.
Start now.
!

102. DISTILLED.NET
SEE ALL OUR FREE VIDEOS
AND BLOG POSTS:
@RobOusbey
@WillCritchlow
@ TomAnthonySEODISTILLEDODN.COM
CHECK OUT OUR SEO
A/B TESTING PLATFORM: