Jori Ford presented on how to take control of a website's crawl budget. She explained that crawl budget is determined by crawl rate and demand. Crawl rate depends on factors like response time and server errors, while demand is based on popularity and freshness. She recommended quantifying a site's current crawl budget by analyzing log files to see what pages are crawled and how often. Then pages can be optimized by improving response time and removing errors. Non-essential pages should be pruned to conserve budget, while important pages can be better linked and updated to increase demand. Overall the key is understanding how Google allocates crawl resources and managing page quality and relevance accordingly.

1. Jori Ford | @chicagoseopro | #TechSEOBoost
#TechSEOBoost | @CatalystSEM
THANK YOU TO THIS YEAR’S SPONSORS
Crawl Budget Conqueror - Take Control of
Your Crawl Budget
Jori Ford, G2

2. Jori Ford | @chicagoseopro | #TechSEOBoost
Take Control of Your Crawl Budget!
Crawl Budget
Conqueror

3. Jori Ford | @chicagoseopro | #TechSEOBoost
Agenda
How Does Google See Crawl Budget?
Key Factors to Managing Crawl Budget
Ways to Quantify Crawl Budget on Your Site
Fine-Tuning & Crawl Controls
• Conservation - Tactics to help you maximize the crawl budget you have
• Optimization - Shifting crawl to places you want
• Expansion - Get More, Why Not!

4. Jori Ford | @chicagoseopro | #TechSEOBoost
How Does Google
See Crawl Budget?

5. Jori Ford | @chicagoseopro | #TechSEOBoost
Deconstructing Crawl Budget
Crawl Budget = Crawl Rate x Crawl Demand

6. Jori Ford | @chicagoseopro | #TechSEOBoost
Crawl Rate is
Variable
“If the site responds really
quickly for a while, the limit
goes up, meaning more
connections can be used to
crawl. If the site slows down
or responds with server
errors, the limit goes down and
Googlebot crawls less.”
-- Gary Illyes, Google

7. Jori Ford | @chicagoseopro | #TechSEOBoost
Crawl Rate Breakdown
= {Server Response Time x Time}
Error Rate
Result:
= {ttfb x duration}
%server error

8. Jori Ford | @chicagoseopro | #TechSEOBoost
Crawl Demand = Popularity x Freshness

9. Jori Ford | @chicagoseopro | #TechSEOBoost
Popular on the Internet =

10. Jori Ford | @chicagoseopro | #TechSEOBoost
Staleness aka
Freshness
In general, we try to do our crawling based
on what we think this page might be
changing or how often it might be changing.
So, if we think that something stays
the same for a longer period of
time, we might not crawl it for a
couple of months.
-- John Mueller, Google

11. Jori Ford | @chicagoseopro | #TechSEOBoost
Crawl Budget =
{Avg. ttfb x duration/%server error} x {(CTR) x (Avg Time
b/t Updates)}
Key Factors to Managing Crawl Budget:
• Time to First Byte
• Server Error Rate
• Click-Thru Rate (CTR)
• Freshness

12. Jori Ford | @chicagoseopro | #TechSEOBoost
Ways to Quantify Crawl Budget on Your Site

13. Jori Ford | @chicagoseopro | #TechSEOBoost
Quantifying Crawl Budget
on Your Site
Step 1: Determine how many pages you’re
expecting to be crawled.
Step 2: Figure out what pages Google
is crawling?
Step 3: How many are there? And with what
frequency are they crawled?
Step 4: Segment your data by page type.

14. Jori Ford | @chicagoseopro | #TechSEOBoost
Step 1: Determine # of Pages Expected
1. What are your money pages?
2. Group by Page Types or Templates
3. Match to XML Sitemaps (ONLY if
optimized)

15. Jori Ford | @chicagoseopro | #TechSEOBoost
Step 2: What Pages are Being
Crawled?
• Raw Server Crawl Logs
• Log File Analyzer Tools (Botify,
OnCrawl, Screaming Frog, Deep
Crawl)
• Custom Tracking (GTM, Cloud
Service Workers)

16. Jori Ford | @chicagoseopro | #TechSEOBoost
Step 3: # of Crawled Pages &
Frequency
• Excel Using Pivot Tables
• SQL Database Query
• Log File Analyzer Tools (Botify,
OnCrawl, Screaming Frog, Deep
Crawl)

17. Jori Ford | @chicagoseopro | #TechSEOBoost
Step 4: Segment
by Type
% Page Type
% Crawl Allocation
% Active vs. Inactive
% Not Crawled

18. Jori Ford | @chicagoseopro | #TechSEOBoost
Calculate Your Crawl
Budget
= (Avg # of Crawled URLs x Freq)
Time

19. Jori Ford | @chicagoseopro | #TechSEOBoost
Fine-Tuning & Crawl Controls

20. Jori Ford | @chicagoseopro | #TechSEOBoost
Conserve Your
Crawl Budget
• Prune Non-Money Pages
• De-Index Dupes
• Remove pages not in XML Sitemap(s)
• Removal or fixing of Erroring Pages

21. Jori Ford | @chicagoseopro | #TechSEOBoost
Optimize
Crawling
• Improve TTFB (time to first byte)
• Optimize XML Sitemaps
• Manage Facets, Filters, & Search Pages w/
NoIndex
• Handle parameters
• Internal Link Optimization
• Orphaned No More!
• Model Crawled/Active Pages

22. Jori Ford | @chicagoseopro | #TechSEOBoost
Expand Crawl Budget
Containers & Clustering Services
• Heroku Dynos
• AWS EC2
• Docker & Kubernetes
• Microsoft Azure
Increase Google’s crawl by auto-
scaling apps

23. Jori Ford | @chicagoseopro | #TechSEOBoost

24. Jori Ford | @chicagoseopro | #TechSEOBoost
Remember…
Google has limited resources
There are only a few major factors you can manage:
• Time to First Byte
• Server Error Rate
• Click-Thru Rate (CTR)
• Freshness
You can quantify your budget manually or with tools, but knowing it is critical
Conserve. Optimize. Expand

25. Jori Ford | @chicagoseopro | #TechSEOBoost
Thank You
–
@chicagoseopro
/in/chicagoseopro
www.joriford.com

26. Jori Ford | @chicagoseopro | #TechSEOBoost
Thanks for Viewing the Slideshare!
–
Watch the Recording: https://youtube.com/session-example
Or
Contact us today to discover how Catalyst can deliver unparalleled SEO
results for your business. https://www.catalystdigital.com/