SlideShare a Scribd company logo
1 of 14
Crawl Budget
Don’t stop the Bot
Overview
1. Introduction
2. Crawl Budget
3. Crawl Rate Limit
4. Crawl Frequency
5. Why you should care
6. Actions
Crawling
“Crawling is the entry point for sites
into Google's search results.”
Gary Illyes, Google Webmaster Central Blog
Google’s Crawl Budget
Google defines crawl budget as:
“The number of URLs Googlebot
can and wants to crawl.”
Google’s Crawl Budget
Crawl Rate Crawl Demand
How often the site is visited.
This is determined by:
• Crawl health
What is Good Health?
• Quick response time
• Limited server errors
Fast site = increased crawl rate
Slow site = Google takes it easy, slows
down, visits less.
Popularity
• How often URLs are visited from the web
• Removal of stale URLs to keep the SERPs
fresh
There are two elements that decide how much crawl budget a website
gets:
Crawl Rate + Crawl Demand = Crawl Budget
Factors affecting crawl budget
• Having many low-value-add URLs has a negative
impact on budget. These include:
• Faceted navigation & session identifiers
• On-site duplicate content
• Low quality & spam content
• Soft error pages e.g. 404s; 500s
• Hacked pages
• Infinite spaces & proxies
Source: from Google Webmaster blog
Why give a jot about the Bot:
Wasting server resources on pages like those mentioned will drain
crawl activity from pages that do actually have value.
This may cause a significant delay in discovering great content on
your or a client’s site.
Crawl Rate Limit
Crawl rate limit is designed to help Google not crawl your pages too
much and too fast where it hurts your server.
In other words, if the Bot thinks your site can’t cope, it will take it
easy. It will crawl slower and therefore reach less of your site.
Crawl Frequency
Crawl Frequency is the number of days per month that Googlebot
requests a URL. There is a clear relationship between traffic and
frequency of crawl, making this a critical SEO indicator.
Understanding what Google is crawling frequently is a good indicator of
what Google thinks is worthwhile, what it needs to keep fresh in the
index.
Understanding the characteristics of those pages can inform what you
might need to do to improve the remainder.
Source: Botify
Factors affecting crawl frequency
• Things that can reduce crawl frequency include:
• Site structure issues
• Duplicate content
• Publishing pages for which there’s no demand
• Publishing at a rate that is faster than what Google is ready to admit to
the index
Pages with more internal links are more likely to be crawled more frequently.
Source: Botify
Why give a jot about the Bot:
• Crawl budget is a precious resource. We need to use it wisely. It is SO
important on larger sites.
• More pages crawled = the more pages that may be indexed.
• More bot energy focussed on high value pages over poor low value content.
• New content is discovered quickly and easily, giving it a chance to rank
sooner
What we want to achieve:
A well-organized site in which the most important content is easily accessible
from the homepage and other important entry points.
A speedy site that represents healthy servers, so the Bot can get more content
over the same number of connections.
• Use Deepcrawl to identify thin pages
• Improve these pages where possible
• Consider combining with another page or removing
Thin
Pages
Duplicate
Content
Internal Linking
• Always do a ‘site:’ search for a variety of related keywords to your
new page / content
• Make sure your content is unique
• Make sure your content sits logically in the site’s hierarchy
• Remove orphaned pages – link to them from relevant pages,
ideally those that are indexed and do well
• Make sure important pages are linked to widely, from relevant
variations of optimised anchor text
• Make sure sitemaps are up-to-date
• Use search console to discover most linked to pages & make sure
these respond to priorities
What we need to do?
Webmaster Blog
Crawl Errors report in Search Console
Crawl Frequency from Botify
Further Reading

More Related Content

Recently uploaded

Relatório da OMS / Unicef / IBFAN sobre a situação do Código Internacional de...
Relatório da OMS / Unicef / IBFAN sobre a situação do Código Internacional de...Relatório da OMS / Unicef / IBFAN sobre a situação do Código Internacional de...
Relatório da OMS / Unicef / IBFAN sobre a situação do Código Internacional de...
Prof. Marcus Renato de Carvalho
 
Decentralized Physical Infrastructure (DePIN) Explained.pdf
Decentralized Physical Infrastructure (DePIN) Explained.pdfDecentralized Physical Infrastructure (DePIN) Explained.pdf
Decentralized Physical Infrastructure (DePIN) Explained.pdf
nehapardhi711
 
Affiliate Marketing Basic Guide For Beginners _20240521_091615_0000.pdf
Affiliate Marketing Basic Guide For Beginners _20240521_091615_0000.pdfAffiliate Marketing Basic Guide For Beginners _20240521_091615_0000.pdf
Affiliate Marketing Basic Guide For Beginners _20240521_091615_0000.pdf
atinukehassan87
 

Recently uploaded (20)

The ClearVoice Approach to Content Audits
The ClearVoice Approach to Content AuditsThe ClearVoice Approach to Content Audits
The ClearVoice Approach to Content Audits
 
Beyond the Basics: Enhanced Strategies for Next-Level Advertising
Beyond the Basics: Enhanced Strategies for Next-Level AdvertisingBeyond the Basics: Enhanced Strategies for Next-Level Advertising
Beyond the Basics: Enhanced Strategies for Next-Level Advertising
 
Report-on-Tech-Marketing-in-B2B-fintech-industry-2024-1.pdf
Report-on-Tech-Marketing-in-B2B-fintech-industry-2024-1.pdfReport-on-Tech-Marketing-in-B2B-fintech-industry-2024-1.pdf
Report-on-Tech-Marketing-in-B2B-fintech-industry-2024-1.pdf
 
Being a PMM with a multi-product portfolio - Product Marketing Summit
Being a PMM with a multi-product portfolio - Product Marketing SummitBeing a PMM with a multi-product portfolio - Product Marketing Summit
Being a PMM with a multi-product portfolio - Product Marketing Summit
 
Relatório da OMS / Unicef / IBFAN sobre a situação do Código Internacional de...
Relatório da OMS / Unicef / IBFAN sobre a situação do Código Internacional de...Relatório da OMS / Unicef / IBFAN sobre a situação do Código Internacional de...
Relatório da OMS / Unicef / IBFAN sobre a situação do Código Internacional de...
 
Using GA 4 to to Prove Value - Greg Jarboe - Aug 8, 2023.pptx
Using GA 4 to to Prove Value - Greg Jarboe - Aug 8, 2023.pptxUsing GA 4 to to Prove Value - Greg Jarboe - Aug 8, 2023.pptx
Using GA 4 to to Prove Value - Greg Jarboe - Aug 8, 2023.pptx
 
Content Segmentation for Organic Visibility
Content Segmentation for Organic VisibilityContent Segmentation for Organic Visibility
Content Segmentation for Organic Visibility
 
How Your Platform Can Achieve a 90% Demo-to-Customer CVR
How Your Platform Can Achieve a 90% Demo-to-Customer CVRHow Your Platform Can Achieve a 90% Demo-to-Customer CVR
How Your Platform Can Achieve a 90% Demo-to-Customer CVR
 
Jesse Saez: A Journey of Dedication and Success in the Automotive Industry
Jesse Saez: A Journey of Dedication and Success in the Automotive IndustryJesse Saez: A Journey of Dedication and Success in the Automotive Industry
Jesse Saez: A Journey of Dedication and Success in the Automotive Industry
 
SocialMedia Marketing Plan for TheSparksFoundation
SocialMedia Marketing Plan for TheSparksFoundationSocialMedia Marketing Plan for TheSparksFoundation
SocialMedia Marketing Plan for TheSparksFoundation
 
Decentralized Physical Infrastructure (DePIN) Explained.pdf
Decentralized Physical Infrastructure (DePIN) Explained.pdfDecentralized Physical Infrastructure (DePIN) Explained.pdf
Decentralized Physical Infrastructure (DePIN) Explained.pdf
 
BrightonSEO - Search Engine Omnipresence_ Why SEOs need to look beyond Google
BrightonSEO - Search Engine Omnipresence_ Why SEOs need to look beyond GoogleBrightonSEO - Search Engine Omnipresence_ Why SEOs need to look beyond Google
BrightonSEO - Search Engine Omnipresence_ Why SEOs need to look beyond Google
 
Affiliate Marketing Basic Guide For Beginners _20240521_091615_0000.pdf
Affiliate Marketing Basic Guide For Beginners _20240521_091615_0000.pdfAffiliate Marketing Basic Guide For Beginners _20240521_091615_0000.pdf
Affiliate Marketing Basic Guide For Beginners _20240521_091615_0000.pdf
 
All Over Conclusion Digital Marketing / Digital Marketing Benefits
All Over Conclusion Digital Marketing / Digital Marketing BenefitsAll Over Conclusion Digital Marketing / Digital Marketing Benefits
All Over Conclusion Digital Marketing / Digital Marketing Benefits
 
Why Digital Marketing Important for our Business.pdf
Why Digital Marketing Important for our Business.pdfWhy Digital Marketing Important for our Business.pdf
Why Digital Marketing Important for our Business.pdf
 
How to Track, Measure & Communicate SEO Results
How to Track, Measure & Communicate SEO ResultsHow to Track, Measure & Communicate SEO Results
How to Track, Measure & Communicate SEO Results
 
Japanese Sauna Hat Trends - Totonoete Inc.
Japanese Sauna Hat Trends - Totonoete Inc.Japanese Sauna Hat Trends - Totonoete Inc.
Japanese Sauna Hat Trends - Totonoete Inc.
 
Social Media Paid Ads Performance Report.pdf
Social Media Paid Ads Performance Report.pdfSocial Media Paid Ads Performance Report.pdf
Social Media Paid Ads Performance Report.pdf
 
SES London 2009 Beyond Linkbait Greg Jarboe.ppt
SES London 2009 Beyond Linkbait Greg Jarboe.pptSES London 2009 Beyond Linkbait Greg Jarboe.ppt
SES London 2009 Beyond Linkbait Greg Jarboe.ppt
 
Colgate Palmolive Human Resources Management .pdf
Colgate Palmolive Human Resources Management .pdfColgate Palmolive Human Resources Management .pdf
Colgate Palmolive Human Resources Management .pdf
 

Featured

Social Media Marketing Trends 2024 // The Global Indie Insights
Social Media Marketing Trends 2024 // The Global Indie InsightsSocial Media Marketing Trends 2024 // The Global Indie Insights
Social Media Marketing Trends 2024 // The Global Indie Insights
Kurio // The Social Media Age(ncy)
 
Good Stuff Happens in 1:1 Meetings: Why you need them and how to do them well
Good Stuff Happens in 1:1 Meetings: Why you need them and how to do them wellGood Stuff Happens in 1:1 Meetings: Why you need them and how to do them well
Good Stuff Happens in 1:1 Meetings: Why you need them and how to do them well
Saba Software
 
Introduction to C Programming Language
Introduction to C Programming LanguageIntroduction to C Programming Language
Introduction to C Programming Language
Simplilearn
 

Featured (20)

How to Prepare For a Successful Job Search for 2024
How to Prepare For a Successful Job Search for 2024How to Prepare For a Successful Job Search for 2024
How to Prepare For a Successful Job Search for 2024
 
Social Media Marketing Trends 2024 // The Global Indie Insights
Social Media Marketing Trends 2024 // The Global Indie InsightsSocial Media Marketing Trends 2024 // The Global Indie Insights
Social Media Marketing Trends 2024 // The Global Indie Insights
 
Trends In Paid Search: Navigating The Digital Landscape In 2024
Trends In Paid Search: Navigating The Digital Landscape In 2024Trends In Paid Search: Navigating The Digital Landscape In 2024
Trends In Paid Search: Navigating The Digital Landscape In 2024
 
5 Public speaking tips from TED - Visualized summary
5 Public speaking tips from TED - Visualized summary5 Public speaking tips from TED - Visualized summary
5 Public speaking tips from TED - Visualized summary
 
ChatGPT and the Future of Work - Clark Boyd
ChatGPT and the Future of Work - Clark Boyd ChatGPT and the Future of Work - Clark Boyd
ChatGPT and the Future of Work - Clark Boyd
 
Getting into the tech field. what next
Getting into the tech field. what next Getting into the tech field. what next
Getting into the tech field. what next
 
Google's Just Not That Into You: Understanding Core Updates & Search Intent
Google's Just Not That Into You: Understanding Core Updates & Search IntentGoogle's Just Not That Into You: Understanding Core Updates & Search Intent
Google's Just Not That Into You: Understanding Core Updates & Search Intent
 
How to have difficult conversations
How to have difficult conversations How to have difficult conversations
How to have difficult conversations
 
Introduction to Data Science
Introduction to Data ScienceIntroduction to Data Science
Introduction to Data Science
 
Time Management & Productivity - Best Practices
Time Management & Productivity -  Best PracticesTime Management & Productivity -  Best Practices
Time Management & Productivity - Best Practices
 
The six step guide to practical project management
The six step guide to practical project managementThe six step guide to practical project management
The six step guide to practical project management
 
Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...
Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...
Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...
 
Unlocking the Power of ChatGPT and AI in Testing - A Real-World Look, present...
Unlocking the Power of ChatGPT and AI in Testing - A Real-World Look, present...Unlocking the Power of ChatGPT and AI in Testing - A Real-World Look, present...
Unlocking the Power of ChatGPT and AI in Testing - A Real-World Look, present...
 
12 Ways to Increase Your Influence at Work
12 Ways to Increase Your Influence at Work12 Ways to Increase Your Influence at Work
12 Ways to Increase Your Influence at Work
 
ChatGPT webinar slides
ChatGPT webinar slidesChatGPT webinar slides
ChatGPT webinar slides
 
More than Just Lines on a Map: Best Practices for U.S Bike Routes
More than Just Lines on a Map: Best Practices for U.S Bike RoutesMore than Just Lines on a Map: Best Practices for U.S Bike Routes
More than Just Lines on a Map: Best Practices for U.S Bike Routes
 
Ride the Storm: Navigating Through Unstable Periods / Katerina Rudko (Belka G...
Ride the Storm: Navigating Through Unstable Periods / Katerina Rudko (Belka G...Ride the Storm: Navigating Through Unstable Periods / Katerina Rudko (Belka G...
Ride the Storm: Navigating Through Unstable Periods / Katerina Rudko (Belka G...
 
Barbie - Brand Strategy Presentation
Barbie - Brand Strategy PresentationBarbie - Brand Strategy Presentation
Barbie - Brand Strategy Presentation
 
Good Stuff Happens in 1:1 Meetings: Why you need them and how to do them well
Good Stuff Happens in 1:1 Meetings: Why you need them and how to do them wellGood Stuff Happens in 1:1 Meetings: Why you need them and how to do them well
Good Stuff Happens in 1:1 Meetings: Why you need them and how to do them well
 
Introduction to C Programming Language
Introduction to C Programming LanguageIntroduction to C Programming Language
Introduction to C Programming Language
 

Search Engine Crawl Budget - Don't Stop the Bot

  • 2. Overview 1. Introduction 2. Crawl Budget 3. Crawl Rate Limit 4. Crawl Frequency 5. Why you should care 6. Actions
  • 3. Crawling “Crawling is the entry point for sites into Google's search results.” Gary Illyes, Google Webmaster Central Blog
  • 4. Google’s Crawl Budget Google defines crawl budget as: “The number of URLs Googlebot can and wants to crawl.”
  • 5. Google’s Crawl Budget Crawl Rate Crawl Demand How often the site is visited. This is determined by: • Crawl health What is Good Health? • Quick response time • Limited server errors Fast site = increased crawl rate Slow site = Google takes it easy, slows down, visits less. Popularity • How often URLs are visited from the web • Removal of stale URLs to keep the SERPs fresh There are two elements that decide how much crawl budget a website gets: Crawl Rate + Crawl Demand = Crawl Budget
  • 6. Factors affecting crawl budget • Having many low-value-add URLs has a negative impact on budget. These include: • Faceted navigation & session identifiers • On-site duplicate content • Low quality & spam content • Soft error pages e.g. 404s; 500s • Hacked pages • Infinite spaces & proxies Source: from Google Webmaster blog
  • 7. Why give a jot about the Bot: Wasting server resources on pages like those mentioned will drain crawl activity from pages that do actually have value. This may cause a significant delay in discovering great content on your or a client’s site.
  • 8. Crawl Rate Limit Crawl rate limit is designed to help Google not crawl your pages too much and too fast where it hurts your server. In other words, if the Bot thinks your site can’t cope, it will take it easy. It will crawl slower and therefore reach less of your site.
  • 9. Crawl Frequency Crawl Frequency is the number of days per month that Googlebot requests a URL. There is a clear relationship between traffic and frequency of crawl, making this a critical SEO indicator. Understanding what Google is crawling frequently is a good indicator of what Google thinks is worthwhile, what it needs to keep fresh in the index. Understanding the characteristics of those pages can inform what you might need to do to improve the remainder. Source: Botify
  • 10. Factors affecting crawl frequency • Things that can reduce crawl frequency include: • Site structure issues • Duplicate content • Publishing pages for which there’s no demand • Publishing at a rate that is faster than what Google is ready to admit to the index Pages with more internal links are more likely to be crawled more frequently. Source: Botify
  • 11. Why give a jot about the Bot: • Crawl budget is a precious resource. We need to use it wisely. It is SO important on larger sites. • More pages crawled = the more pages that may be indexed. • More bot energy focussed on high value pages over poor low value content. • New content is discovered quickly and easily, giving it a chance to rank sooner
  • 12. What we want to achieve: A well-organized site in which the most important content is easily accessible from the homepage and other important entry points. A speedy site that represents healthy servers, so the Bot can get more content over the same number of connections.
  • 13. • Use Deepcrawl to identify thin pages • Improve these pages where possible • Consider combining with another page or removing Thin Pages Duplicate Content Internal Linking • Always do a ‘site:’ search for a variety of related keywords to your new page / content • Make sure your content is unique • Make sure your content sits logically in the site’s hierarchy • Remove orphaned pages – link to them from relevant pages, ideally those that are indexed and do well • Make sure important pages are linked to widely, from relevant variations of optimised anchor text • Make sure sitemaps are up-to-date • Use search console to discover most linked to pages & make sure these respond to priorities What we need to do?
  • 14. Webmaster Blog Crawl Errors report in Search Console Crawl Frequency from Botify Further Reading