This document discusses optimizing a website's crawl budget for a mobile-first index. It recommends auditing the mobile site, understanding how mobile crawlers view it, and optimizing things like pruning content and internal redirects. It then discusses growing organic revenue as a case study, noting one site migrated to mobile-first and saw a 46% increase in organic traffic over 8 weeks after crawl budget optimization.

1. Crawl Budget Optimisation
In a Mobile-First Index
@Botify | @SearchMATH
https://www.slideshare.net/MarkThomas114

2. AUDIT
MOBILE
SITE
1
OPTIMISE
CRAWL
BUDGET
2
GROW
ORGANIC
REVENUE
3

3. MOBILE-
FIRST
Case Study Context
✔ ✔ ✔
CRAWL
BUDGET
2018

4. ORGANIC TRAFFIC
46% IN 8 WEEKS

5. Crawl Budget Optimisation

6. 46%
Yielding Traffic Growth

7. AUDIT
MOBILE
SITE
1
OPTIMISE
CRAWL
BUDGET
2
GROW
ORGANIC
REVENUE
3

8. MOBILE-FIRST
REFRESHER

9. W
E
B
S
E
A
R
C
H
Internet
Crawler
Scheduler
Parser Indexing SearchIndexURLs

10. C
R
A
W
L
B
U
D
G
E
T
Internet
Crawler
Scheduler
Parser Indexing SearchIndexURLs
CRAWL DEMAND
CRAWL RATE
LIMIT
CRAWL
BUDGET

11. We'll use the mobile version of
the content for indexing and
ranking, to better help our
– primarily mobile – users
find what they're looking for.
“
Mobile-First Index Explained

12. Internet
Crawler
Scheduler
W
E
B
C
R
A
W
L
I
N
G Parser Indexing SearchIndexURLs
BEFORE

13. M
O
B
I
L
E
F
I
R
S
T
BEFORE
Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

14. M
O
B
I
L
E
F
I
R
S
T
AFTER
Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like
Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1;
+http://www.google.com/bot.html)

15. Internet
Crawler
Scheduler
Parser Indexing SearchIndexURLs
M
O
B
I
L
E
F
I
R
S
T
AFTER

16. we transition sites slowly to
ensure a good experience for
site owners and users.
“
Mobile-First Index Transition

17. We evaluate each site
individually on its readiness for
mobile-first indexing based on
the best practices and transition
the site when the site is ready.
“
Mobile-First Index Transition

18. Mobile-First Transition
M
O
B
I
L
E
F
I
R
S
T
Index Index Index
Index
2018
JANUARY
2018
JULY

19. MONITOR
SERVER LOGS
M
O
B
I
L
E
F
I
R
S
T
MONITOR GSC
NOTIFICATIONS

20. MONITOR
SERVER LOGS
M
O
B
I
L
E
F
I
R
S
T
MONITOR GSC
NOTIFICATIONS

21. AUDIT
MOBILE
SITE
1
OPTIMISE
CRAWL
BUDGET
2
GROW
ORGANIC
REVENUE
3

22. STEP 1
UNDERSTAND
HOW MOBILE
CRAWLERS
VIEW YOUR
SITE

23. M
O
B
I
L
E
T
E
S
T
I
N
G

24. Recommended Validation Approach
M
O
B
I
L
E
T
E
S
T
I
N
G
MOBILE
FRIENDLY TEST
MOBILE
USABILITY GSC
CHROME DEV
TOOLS

25. M
O
B
I
L
E
T
E
S
T
I
N
G

26. M
O
B
I
L
E
T
E
S
T
I
N
G
Crawl with Smartphone UA

27. Internet
Crawler
Scheduler
Parser Indexing SearchIndexURLs
M
O
B
I
L
E
F
I
R
S
T
Test Crawl with
Google’s
Smartphone User
Agent

28. Links
Speed
Content
Metadata
Structured Data
Links
Speed
Content
Metadata
Structured Data
M
O
B
I
L
E
T
E
S
T
I
N
G
CRAWL
C
O
M
P
A
R
E

29. BE CAREFUL, THIS IS INSUFFICIENT
M
O
B
I
L
E
T
E
S
T
I
N
G

30. KPIS MOBILE DESKTOP DIFF
Avg. Load Times 474,0 707,8 -33,0%
Avg. Depth 34,0 4,9 591,0%
% Internal PageRank 91,6% 92,1% -0,6%
Avg. Follow Inlinks 22,5 24,3 -7,3%
Avg. Follow Outlinks 24,1 24,3 -0,71%
Avg. Anchors Variations 1,7 2,7 -36,9%
Avg. Dup. Titles 512,8 0,4 141222,2%
Avg. Dup. H1 386,2 2,3 16401,2%
Avg. Dup. Descriptions 55,4 0,5 11043,8%
Avg. Title Length 69,6 72,6 -4,2%
Avg. Words (Not Ignored) 147,7 222,1 -33,5%
Avg. % Unique N-Grams 10,9 14,8 -26,5%
Sum. Similar Pages (>90%) 133902 42500 215,1%
M
O
B
I
L
E
T
E
S
T
I
N
G
GO DEEPER: Analyse Structural KPIs

31. M
O
B
I
L
E
T
E
S
T
I
N
G
GO DEEPER: Analyse Structural KPIs

32. M
O
B
I
L
E
T
E
S
T
I
N
G
Crawl with JavaScript enabled

33. JavaScript is the browser’s behaviour language
M
O
B
I
L
E
T
E
S
T
I
N
G
What is JavaScript?
● Tells the browser what to display
● Tree of nodes, described by tags
● Nodes can contain:
○ Text
○ Metadata
○ Links
○ (Assets) - not relevant
○ (Style) - not relevant
● Uses HTML to tell the browser what to
display
● Allows a developer to dictate (‘Script’) a
browser’s behaviour after receiving a Web
Page
● Can make extra requests
● Can react to user interaction (clicks, mouse
movement, keyboard, etc…)
● Can store information on the user’s
computer
● Can modify the HTML
HTML is the browser’s display language

34. M
O
B
I
L
E
T
E
S
T
I
N
G
Simulate with JavaScript enabled

35. MIGRATEVALIDATEPLAN
M
O
B
I
L
E
T
E
S
T
I
N
G
Recommended Approach

36. AUDIT
MOBILE
SITE
1
OPTIMISE
CRAWL
BUDGET
2
GROW
ORGANIC
REVENUE
3

37. STEP 2
OPTIMISE
MOBILE
CRAWL
BUDGET

38. ❏ Prune Content
❏ Clean Internal Redirects
❏ Avoid Inconsistent Signals
❏ Use Robots.txt to block sections of your site you don’t want
crawled instead of page level noindex tags
❏ Improve your site structure
❏ 304 Status Code
❏ Use 410 to tell Google when content is permanently gone
C
R
A
W
L
B
U
D
G
E
T
Optimisation

39. AUDIT
MOBILE
SITE
1
OPTIMISE
CRAWL
BUDGET
2
GROW
ORGANIC
REVENUE
3

40. STEP 3
REVENUE
GROWTH
CASE
STUDY

41. G
R
O
W
R
E
V
E
N
U
E
Migrated to Mobile-First

42. Focus on profitability:
# Crawls vs # Visits
Do we have ?
1. few crawls vs. many visits
2. few crawls vs. few visits
3. many crawls vs. few visits
- low frequency ?
- thin content ?
- a spider trap ?
Allows us to focus on Active Page Ratio:
- nb. URLs visited / nb. URLs crawled
G
R
O
W
R
E
V
E
N
U
E
Identify and cut the waste

43. G
R
O
W
R
E
V
E
N
U
E
Analyse Architecture

44. The Results?

45. Crawl Budget Optimisation

46. March 15th
17.5% Indexable Pages Crawled
April 15th
33.4% Indexable Pages Crawled

47. 46%
Yielding Traffic Growth

48. THANK YOU!