SlideShare a Scribd company logo
1 of 42
Crawl Budget:
Everything you Need
to Know
https://www.slideshare.net/SallyR7
@salryam
Sally Raymer
HEYCAR
/in/sally-r-seo/
What is crawl budget and
why does it matter?
The bottom
line
● Updated guidelines recognise
medium and problematic sites
● Increasing the crawl rate to
better fit your site
● Reducing crawl bloat
How Google Interacts with
our Sites
Page is created
Site requests page is
crawled
Google Crawls the page
Indexing
Does the page contain authoritative, high
quality content that matches user intent?
Page is indexed Crawled but not indexed
Discovered not Indexed
Crawl Demand Crawl Capacity
● Eligible indexable URLs
● How often content is
created/ updated
● How much traffic you get
to folders/ pages on site
● How quickly content loads
● How many errors are on
site
JS
Rendering
Leveraging your tools
What is my crawl rate?
Create Actionable data
crawl requests/ 90
days = daily crawl
budget
indexable + errors
+ canonicals = no.
of requested pages
no. of requested
pages / daily crawl
budget = site ratio
Website
Daily crawl
budget
No. of
requested
crawls
Site ratio
No.
Indexed
Pages
% Indexed
pages vs
requests
example.com 20,000 4,826,000 x241 476,000 9.7%
Identifying Issues
Uncover the coverage
report
● Discovered not indexed
● Crawled not indexed
● Server errors
● Large number of canonical
tags
● Large number redirects
● Indexed -blocked by robots.txt
Tackle low traffic pages
Log File Analysis
How often is Google crawling?
What’s being crawled
in comparison to
organic traffic
What response codes are being crawled?
Define historic trends and volume
What paginated pages & parameters
are being crawled
What file types is Google crawling?
Images
Video
JS
What is not being crawled?
What commonalities do crawled pages share?
Managing Crawl Behaviour
Trying to Index All Pages
Use Internal links to prioritise
● Capitalise on link equity
● Review link hierarchy regularly
● Strengthen content hubs
● Be mindful of where links are pointing
Dynamically populate sitemaps
● Qualify all 200 canonicals
● Break your sitemaps up
● Don’t forget images and videos
● Minimise manual management
Use robots.txt
● Prevent Google from crawling
specific taxonomies
● Highlight top directories
● Link to sitemaps
● Remember: Blocked URLS can
be indexed
Assessing Deindexation
Three Questions to ask yourself
1.What are our KPIs?
2.How do we define user intent?
3.What is our web structure?
Credit: https://www.screamingfrog.co.uk/site-architecture-crawl-visualisations/
Low (low traffic) quality content
If it doesn’t work, can we cut it?
● Are they high intent pages?
● Do they have quality backlinks?
If the answer is no, what is the value in holding on to pages
that aren’t working?
Duplication & Cannibalisation
● Canonicals ➡️ Do we need both sets of content
● Bad taxonomy ➡️ Clean up the site structure
● Internal competition ➡️ Use GSC & keyword mapping
Faceted navigation &
parameters
● Do not deindex all parameters as a rule
● Are top level parameters valuable?
● Are static alternatives available?
● Is link equity and information architecture protected?
Redirects & 404s
● Redirects used for OOS products
● Historical buildup of redirects or 404’s
● Embedded links will continue to be crawled
● Optimise the user journey not just site health
Picking an Approach to Deindexation
Noindex directive - if the page serves a purpose
in the user journey, the noindex meta tag is
probably the best method but does not take effect
immediately.
Blocking via robots.txt - this is a great way of
keeping down crawl requests but it doesn’t stop
indexation.
410 - Removing URLs that do not play a part in the
user journey or provide value is a great way of
dealing with crawl bloat.
Improving Demand and Capacity
Ensure Content is Accessible
If you have a large JS web app,
ensuring that Google does not
have to allocate more server
resources than essential to
render important content.
Credit: https://nextjs.org/learn/basics/data-fetching/two-forms
Updating & Creating New Content
Regularly
Creating fresh content that meets
Google E-E-A-T quality rater
guidelines will help increase crawl
demand for your site.
Improving Page Speed and Performance
● Invest in servers
● Improve page performance metrics
● Limit errors - redirect chains or loops
Crawl Budget: A Happy Ending
Identifying crawl
rate
Optimising the
crawl budget
Identifying crawl
issues
1.
2. 3.
THANK YOU
Slideshare.Net/SallyRaymer2
@salraym /sally-r-seo/
Some great resources:
Google Documentation: Managing Crawl Budget
Verifying Google Bot for log file analysis
Screaming Frog’s guide to log file analysis
How to identify keyword cannibalisation
SEJ internal link optimisation checklist for enterprise sites
Mckinsey Consumer Decision Journey
Mckinsey customer touchpoints

More Related Content

What's hot

Why your tech optimisations are still sat in the backlog
Why your tech optimisations are still sat in the backlogWhy your tech optimisations are still sat in the backlog
Why your tech optimisations are still sat in the backlogVicky481083
 
Improving Crawling and Indexing using Real-Time Log File Insights
Improving Crawling and Indexing using Real-Time Log File InsightsImproving Crawling and Indexing using Real-Time Log File Insights
Improving Crawling and Indexing using Real-Time Log File InsightsSteven van Vessum
 
Content Design & its Role in SEO and Accessibility [BrightonSEO Spring 2023]
Content Design & its Role in SEO and Accessibility [BrightonSEO Spring 2023]Content Design & its Role in SEO and Accessibility [BrightonSEO Spring 2023]
Content Design & its Role in SEO and Accessibility [BrightonSEO Spring 2023]Chloe Smith
 
Internal Linking - The Topic Clustering Way edited.pptx
Internal Linking - The Topic Clustering Way edited.pptxInternal Linking - The Topic Clustering Way edited.pptx
Internal Linking - The Topic Clustering Way edited.pptxDixon Jones
 
BrightonSEO April 2023 Similar AI: Automation recipes for SEO success
BrightonSEO April 2023 Similar AI: Automation recipes for SEO successBrightonSEO April 2023 Similar AI: Automation recipes for SEO success
BrightonSEO April 2023 Similar AI: Automation recipes for SEO successDylan Fuler
 
Machine Learning use cases for Technical SEO Automation Brighton SEO Patrick ...
Machine Learning use cases for Technical SEO Automation Brighton SEO Patrick ...Machine Learning use cases for Technical SEO Automation Brighton SEO Patrick ...
Machine Learning use cases for Technical SEO Automation Brighton SEO Patrick ...Ahrefs
 
GretaMunari - The redemption of content automation
GretaMunari - The redemption of content automationGretaMunari - The redemption of content automation
GretaMunari - The redemption of content automationGretaMunari1
 
BrightonSEO - Apr 2022 - No excuses for doing UX
BrightonSEO - Apr 2022 - No excuses for doing UXBrightonSEO - Apr 2022 - No excuses for doing UX
BrightonSEO - Apr 2022 - No excuses for doing UXOban International
 
BrightonSEO - Master Crawl Budget Optimization for Enterprise Websites
BrightonSEO - Master Crawl Budget Optimization for Enterprise WebsitesBrightonSEO - Master Crawl Budget Optimization for Enterprise Websites
BrightonSEO - Master Crawl Budget Optimization for Enterprise WebsitesManick Bhan
 
Data Driven Approach to Scale SEO at BrightonSEO 2023
Data Driven Approach to Scale SEO at BrightonSEO 2023Data Driven Approach to Scale SEO at BrightonSEO 2023
Data Driven Approach to Scale SEO at BrightonSEO 2023Nitin Manchanda
 
Canonicalization for SEO BrightonSEO April 2023 Patrick Stox
Canonicalization for SEO BrightonSEO April 2023 Patrick StoxCanonicalization for SEO BrightonSEO April 2023 Patrick Stox
Canonicalization for SEO BrightonSEO April 2023 Patrick StoxAhrefs
 
How to overcome the fear of taking time off.pptx
How to overcome the fear of taking time off.pptxHow to overcome the fear of taking time off.pptx
How to overcome the fear of taking time off.pptxQuibble
 
[BrightonSEO 2022] Unlocking the Hidden Potential of Product Listing Pages
[BrightonSEO 2022] Unlocking the Hidden Potential of Product Listing Pages[BrightonSEO 2022] Unlocking the Hidden Potential of Product Listing Pages
[BrightonSEO 2022] Unlocking the Hidden Potential of Product Listing PagesAreej AbuAli
 
BrightonSEO: How to generate 8 million SEO test ideas - Will Critchlow
BrightonSEO: How to generate 8 million SEO test ideas - Will CritchlowBrightonSEO: How to generate 8 million SEO test ideas - Will Critchlow
BrightonSEO: How to generate 8 million SEO test ideas - Will CritchlowWill Critchlow
 
Fully Automated Link Building - Brighton SEO.pdf
Fully Automated Link Building - Brighton SEO.pdfFully Automated Link Building - Brighton SEO.pdf
Fully Automated Link Building - Brighton SEO.pdfSam Oh
 
How to Combat SERP Volatility - Adriana Stein - BrightonSEO Slides 2023pdf
How to Combat SERP Volatility - Adriana Stein - BrightonSEO Slides 2023pdfHow to Combat SERP Volatility - Adriana Stein - BrightonSEO Slides 2023pdf
How to Combat SERP Volatility - Adriana Stein - BrightonSEO Slides 2023pdfAS Marketing
 
How to rethink the traditional SEO workspace to promote team wellbeing and pr...
How to rethink the traditional SEO workspace to promote team wellbeing and pr...How to rethink the traditional SEO workspace to promote team wellbeing and pr...
How to rethink the traditional SEO workspace to promote team wellbeing and pr...Varn
 
[BrightonSEO October 2022] On-page SEO: from intention to conversion
[BrightonSEO October 2022] On-page SEO: from intention to conversion[BrightonSEO October 2022] On-page SEO: from intention to conversion
[BrightonSEO October 2022] On-page SEO: from intention to conversionFelipe Bazon
 
Core Web Vitals Audit - Sophie Gibson - PDF - BrightonSEO.pdf
Core Web Vitals Audit - Sophie Gibson - PDF - BrightonSEO.pdfCore Web Vitals Audit - Sophie Gibson - PDF - BrightonSEO.pdf
Core Web Vitals Audit - Sophie Gibson - PDF - BrightonSEO.pdfSophie Gibson
 
A Simple method to Create Content using NLP
A Simple method to Create Content using NLP A Simple method to Create Content using NLP
A Simple method to Create Content using NLP Sante J. Achille
 

What's hot (20)

Why your tech optimisations are still sat in the backlog
Why your tech optimisations are still sat in the backlogWhy your tech optimisations are still sat in the backlog
Why your tech optimisations are still sat in the backlog
 
Improving Crawling and Indexing using Real-Time Log File Insights
Improving Crawling and Indexing using Real-Time Log File InsightsImproving Crawling and Indexing using Real-Time Log File Insights
Improving Crawling and Indexing using Real-Time Log File Insights
 
Content Design & its Role in SEO and Accessibility [BrightonSEO Spring 2023]
Content Design & its Role in SEO and Accessibility [BrightonSEO Spring 2023]Content Design & its Role in SEO and Accessibility [BrightonSEO Spring 2023]
Content Design & its Role in SEO and Accessibility [BrightonSEO Spring 2023]
 
Internal Linking - The Topic Clustering Way edited.pptx
Internal Linking - The Topic Clustering Way edited.pptxInternal Linking - The Topic Clustering Way edited.pptx
Internal Linking - The Topic Clustering Way edited.pptx
 
BrightonSEO April 2023 Similar AI: Automation recipes for SEO success
BrightonSEO April 2023 Similar AI: Automation recipes for SEO successBrightonSEO April 2023 Similar AI: Automation recipes for SEO success
BrightonSEO April 2023 Similar AI: Automation recipes for SEO success
 
Machine Learning use cases for Technical SEO Automation Brighton SEO Patrick ...
Machine Learning use cases for Technical SEO Automation Brighton SEO Patrick ...Machine Learning use cases for Technical SEO Automation Brighton SEO Patrick ...
Machine Learning use cases for Technical SEO Automation Brighton SEO Patrick ...
 
GretaMunari - The redemption of content automation
GretaMunari - The redemption of content automationGretaMunari - The redemption of content automation
GretaMunari - The redemption of content automation
 
BrightonSEO - Apr 2022 - No excuses for doing UX
BrightonSEO - Apr 2022 - No excuses for doing UXBrightonSEO - Apr 2022 - No excuses for doing UX
BrightonSEO - Apr 2022 - No excuses for doing UX
 
BrightonSEO - Master Crawl Budget Optimization for Enterprise Websites
BrightonSEO - Master Crawl Budget Optimization for Enterprise WebsitesBrightonSEO - Master Crawl Budget Optimization for Enterprise Websites
BrightonSEO - Master Crawl Budget Optimization for Enterprise Websites
 
Data Driven Approach to Scale SEO at BrightonSEO 2023
Data Driven Approach to Scale SEO at BrightonSEO 2023Data Driven Approach to Scale SEO at BrightonSEO 2023
Data Driven Approach to Scale SEO at BrightonSEO 2023
 
Canonicalization for SEO BrightonSEO April 2023 Patrick Stox
Canonicalization for SEO BrightonSEO April 2023 Patrick StoxCanonicalization for SEO BrightonSEO April 2023 Patrick Stox
Canonicalization for SEO BrightonSEO April 2023 Patrick Stox
 
How to overcome the fear of taking time off.pptx
How to overcome the fear of taking time off.pptxHow to overcome the fear of taking time off.pptx
How to overcome the fear of taking time off.pptx
 
[BrightonSEO 2022] Unlocking the Hidden Potential of Product Listing Pages
[BrightonSEO 2022] Unlocking the Hidden Potential of Product Listing Pages[BrightonSEO 2022] Unlocking the Hidden Potential of Product Listing Pages
[BrightonSEO 2022] Unlocking the Hidden Potential of Product Listing Pages
 
BrightonSEO: How to generate 8 million SEO test ideas - Will Critchlow
BrightonSEO: How to generate 8 million SEO test ideas - Will CritchlowBrightonSEO: How to generate 8 million SEO test ideas - Will Critchlow
BrightonSEO: How to generate 8 million SEO test ideas - Will Critchlow
 
Fully Automated Link Building - Brighton SEO.pdf
Fully Automated Link Building - Brighton SEO.pdfFully Automated Link Building - Brighton SEO.pdf
Fully Automated Link Building - Brighton SEO.pdf
 
How to Combat SERP Volatility - Adriana Stein - BrightonSEO Slides 2023pdf
How to Combat SERP Volatility - Adriana Stein - BrightonSEO Slides 2023pdfHow to Combat SERP Volatility - Adriana Stein - BrightonSEO Slides 2023pdf
How to Combat SERP Volatility - Adriana Stein - BrightonSEO Slides 2023pdf
 
How to rethink the traditional SEO workspace to promote team wellbeing and pr...
How to rethink the traditional SEO workspace to promote team wellbeing and pr...How to rethink the traditional SEO workspace to promote team wellbeing and pr...
How to rethink the traditional SEO workspace to promote team wellbeing and pr...
 
[BrightonSEO October 2022] On-page SEO: from intention to conversion
[BrightonSEO October 2022] On-page SEO: from intention to conversion[BrightonSEO October 2022] On-page SEO: from intention to conversion
[BrightonSEO October 2022] On-page SEO: from intention to conversion
 
Core Web Vitals Audit - Sophie Gibson - PDF - BrightonSEO.pdf
Core Web Vitals Audit - Sophie Gibson - PDF - BrightonSEO.pdfCore Web Vitals Audit - Sophie Gibson - PDF - BrightonSEO.pdf
Core Web Vitals Audit - Sophie Gibson - PDF - BrightonSEO.pdf
 
A Simple method to Create Content using NLP
A Simple method to Create Content using NLP A Simple method to Create Content using NLP
A Simple method to Create Content using NLP
 

Similar to Crawl Budget: Everything you Need to Know

Programmatic SEO: How to Dominate SEO Like TripAdvisor, Yelp and Zillow
Programmatic SEO: How to Dominate SEO Like TripAdvisor, Yelp and ZillowProgrammatic SEO: How to Dominate SEO Like TripAdvisor, Yelp and Zillow
Programmatic SEO: How to Dominate SEO Like TripAdvisor, Yelp and ZillowBernard Huang
 
SEO - What matters and What to do about it
SEO - What matters and What to do about itSEO - What matters and What to do about it
SEO - What matters and What to do about itAbdullah Khan
 
Crawl Budget Optimisation at #dmwf2018
Crawl Budget Optimisation at #dmwf2018Crawl Budget Optimisation at #dmwf2018
Crawl Budget Optimisation at #dmwf2018Nitin Manchanda
 
How to preform an In-House SEO Audit
How to preform an In-House SEO AuditHow to preform an In-House SEO Audit
How to preform an In-House SEO AuditElliott Davidson
 
How to earn money from digital marketing wma study material
How to earn money from digital marketing   wma study materialHow to earn money from digital marketing   wma study material
How to earn money from digital marketing wma study materialjeevithakumar6
 
Search Engine Optimization Checklist You Need To Know
Search Engine Optimization Checklist You Need To KnowSearch Engine Optimization Checklist You Need To Know
Search Engine Optimization Checklist You Need To Knowi-engage
 
Things I've learnt as an SEO Analyst.pptx
Things I've learnt as an SEO Analyst.pptxThings I've learnt as an SEO Analyst.pptx
Things I've learnt as an SEO Analyst.pptxPratik Priyadarshi
 
Things I've learnt as an SEO Analyst.pdf
Things I've learnt as an SEO Analyst.pdfThings I've learnt as an SEO Analyst.pdf
Things I've learnt as an SEO Analyst.pdfPratik Priyadarshi
 
Role of SEO to website migration - How to keep your organic search performan...
Role of SEO to website migration  - How to keep your organic search performan...Role of SEO to website migration  - How to keep your organic search performan...
Role of SEO to website migration - How to keep your organic search performan...Anna Rhea Teves
 
Google new webmaster guidelines october 2012
Google new webmaster guidelines october 2012Google new webmaster guidelines october 2012
Google new webmaster guidelines october 2012Mark Simon
 
SEO for Web Developers Enhancing Website Ranking and Visibility in Canada.pptx
SEO for Web Developers Enhancing Website Ranking and Visibility in Canada.pptxSEO for Web Developers Enhancing Website Ranking and Visibility in Canada.pptx
SEO for Web Developers Enhancing Website Ranking and Visibility in Canada.pptxmatrix bricks infotech
 
SEO 101: How to Get Started Winning Google Search Traffic
SEO 101: How to Get Started Winning Google Search TrafficSEO 101: How to Get Started Winning Google Search Traffic
SEO 101: How to Get Started Winning Google Search TrafficBernard Huang
 
SEO Presentation-Cybernician
SEO Presentation-CybernicianSEO Presentation-Cybernician
SEO Presentation-Cybernicianfaiyaz Khan
 
Keeping Things Lean & Mean: Crawl Optimisation - Search Marketing Summit AU
Keeping Things Lean & Mean: Crawl Optimisation - Search Marketing Summit AUKeeping Things Lean & Mean: Crawl Optimisation - Search Marketing Summit AU
Keeping Things Lean & Mean: Crawl Optimisation - Search Marketing Summit AUJason Mun
 
THE COMPLETE SEO COURSE.pptx
THE COMPLETE SEO COURSE.pptxTHE COMPLETE SEO COURSE.pptx
THE COMPLETE SEO COURSE.pptxEcommerceGuru
 

Similar to Crawl Budget: Everything you Need to Know (20)

Programmatic SEO: How to Dominate SEO Like TripAdvisor, Yelp and Zillow
Programmatic SEO: How to Dominate SEO Like TripAdvisor, Yelp and ZillowProgrammatic SEO: How to Dominate SEO Like TripAdvisor, Yelp and Zillow
Programmatic SEO: How to Dominate SEO Like TripAdvisor, Yelp and Zillow
 
SEO - What matters and What to do about it
SEO - What matters and What to do about itSEO - What matters and What to do about it
SEO - What matters and What to do about it
 
Crawl Budget Optimisation at #dmwf2018
Crawl Budget Optimisation at #dmwf2018Crawl Budget Optimisation at #dmwf2018
Crawl Budget Optimisation at #dmwf2018
 
How to preform an In-House SEO Audit
How to preform an In-House SEO AuditHow to preform an In-House SEO Audit
How to preform an In-House SEO Audit
 
How to earn money from digital marketing wma study material
How to earn money from digital marketing   wma study materialHow to earn money from digital marketing   wma study material
How to earn money from digital marketing wma study material
 
Internet Marketing
Internet MarketingInternet Marketing
Internet Marketing
 
Search Engine Optimization Checklist You Need To Know
Search Engine Optimization Checklist You Need To KnowSearch Engine Optimization Checklist You Need To Know
Search Engine Optimization Checklist You Need To Know
 
Things I've learnt as an SEO Analyst.pptx
Things I've learnt as an SEO Analyst.pptxThings I've learnt as an SEO Analyst.pptx
Things I've learnt as an SEO Analyst.pptx
 
Things I've learnt as an SEO Analyst.pdf
Things I've learnt as an SEO Analyst.pdfThings I've learnt as an SEO Analyst.pdf
Things I've learnt as an SEO Analyst.pdf
 
Foxtail Website Audit
Foxtail Website AuditFoxtail Website Audit
Foxtail Website Audit
 
Role of SEO to website migration - How to keep your organic search performan...
Role of SEO to website migration  - How to keep your organic search performan...Role of SEO to website migration  - How to keep your organic search performan...
Role of SEO to website migration - How to keep your organic search performan...
 
Google new webmaster guidelines october 2012
Google new webmaster guidelines october 2012Google new webmaster guidelines october 2012
Google new webmaster guidelines october 2012
 
SEO for Web Developers Enhancing Website Ranking and Visibility in Canada.pptx
SEO for Web Developers Enhancing Website Ranking and Visibility in Canada.pptxSEO for Web Developers Enhancing Website Ranking and Visibility in Canada.pptx
SEO for Web Developers Enhancing Website Ranking and Visibility in Canada.pptx
 
SEO 101: How to Get Started Winning Google Search Traffic
SEO 101: How to Get Started Winning Google Search TrafficSEO 101: How to Get Started Winning Google Search Traffic
SEO 101: How to Get Started Winning Google Search Traffic
 
SEO Presentation-Cybernician
SEO Presentation-CybernicianSEO Presentation-Cybernician
SEO Presentation-Cybernician
 
Keeping Things Lean & Mean: Crawl Optimisation - Search Marketing Summit AU
Keeping Things Lean & Mean: Crawl Optimisation - Search Marketing Summit AUKeeping Things Lean & Mean: Crawl Optimisation - Search Marketing Summit AU
Keeping Things Lean & Mean: Crawl Optimisation - Search Marketing Summit AU
 
Site Analysis
Site AnalysisSite Analysis
Site Analysis
 
THE COMPLETE SEO COURSE.pptx
THE COMPLETE SEO COURSE.pptxTHE COMPLETE SEO COURSE.pptx
THE COMPLETE SEO COURSE.pptx
 
SEO & Its Importance.pptx
SEO & Its Importance.pptxSEO & Its Importance.pptx
SEO & Its Importance.pptx
 
SEO Agency In Shimla.pptx
SEO Agency In Shimla.pptxSEO Agency In Shimla.pptx
SEO Agency In Shimla.pptx
 

Recently uploaded

Understanding UAE Labour Law: Key Points for Employers and Employees
Understanding UAE Labour Law: Key Points for Employers and EmployeesUnderstanding UAE Labour Law: Key Points for Employers and Employees
Understanding UAE Labour Law: Key Points for Employers and EmployeesDragon Dream Bar
 
Cracking the Change Management Code Main New.pptx
Cracking the Change Management Code Main New.pptxCracking the Change Management Code Main New.pptx
Cracking the Change Management Code Main New.pptxWorkforce Group
 
Meaningful Technology for Humans: How Strategy Helps to Deliver Real Value fo...
Meaningful Technology for Humans: How Strategy Helps to Deliver Real Value fo...Meaningful Technology for Humans: How Strategy Helps to Deliver Real Value fo...
Meaningful Technology for Humans: How Strategy Helps to Deliver Real Value fo...Björn Rohles
 
Raising Seed Capital by Steve Schlafman at RRE Ventures
Raising Seed Capital by Steve Schlafman at RRE VenturesRaising Seed Capital by Steve Schlafman at RRE Ventures
Raising Seed Capital by Steve Schlafman at RRE VenturesAlejandro Cremades
 
The Inspiring Personality To Watch In 2024.pdf
The Inspiring Personality To Watch In 2024.pdfThe Inspiring Personality To Watch In 2024.pdf
The Inspiring Personality To Watch In 2024.pdfinsightssuccess2
 
MichaelStarkes_UncutGemsProjectSummary.pdf
MichaelStarkes_UncutGemsProjectSummary.pdfMichaelStarkes_UncutGemsProjectSummary.pdf
MichaelStarkes_UncutGemsProjectSummary.pdfmstarkes24
 
Copyright: What Creators and Users of Art Need to Know
Copyright: What Creators and Users of Art Need to KnowCopyright: What Creators and Users of Art Need to Know
Copyright: What Creators and Users of Art Need to KnowMiriam Robeson
 
Toyota Kata Coaching for Agile Teams & Transformations
Toyota Kata Coaching for Agile Teams & TransformationsToyota Kata Coaching for Agile Teams & Transformations
Toyota Kata Coaching for Agile Teams & TransformationsStefan Wolpers
 
بروفايل شركة ميار الخليج للاستشارات الهندسية.pdf
بروفايل شركة ميار الخليج للاستشارات الهندسية.pdfبروفايل شركة ميار الخليج للاستشارات الهندسية.pdf
بروفايل شركة ميار الخليج للاستشارات الهندسية.pdfomnme1
 
Equinox Gold Corporate Deck May 24th 2024
Equinox Gold Corporate Deck May 24th 2024Equinox Gold Corporate Deck May 24th 2024
Equinox Gold Corporate Deck May 24th 2024Equinox Gold Corp.
 
Creative Ideas for Interactive Team Presentations
Creative Ideas for Interactive Team PresentationsCreative Ideas for Interactive Team Presentations
Creative Ideas for Interactive Team PresentationsSlidesAI
 
How to Maintain Healthy Life style.pptx
How to Maintain  Healthy Life style.pptxHow to Maintain  Healthy Life style.pptx
How to Maintain Healthy Life style.pptxrdishurana
 
NewBase 24 May 2024 Energy News issue - 1727 by Khaled Al Awadi_compresse...
NewBase   24 May  2024  Energy News issue - 1727 by Khaled Al Awadi_compresse...NewBase   24 May  2024  Energy News issue - 1727 by Khaled Al Awadi_compresse...
NewBase 24 May 2024 Energy News issue - 1727 by Khaled Al Awadi_compresse...Khaled Al Awadi
 
Salesforce Loyalty Management A Comprehensive.pdf
Salesforce Loyalty Management A Comprehensive.pdfSalesforce Loyalty Management A Comprehensive.pdf
Salesforce Loyalty Management A Comprehensive.pdfqrsolutionsindia
 
Falcon Invoice Discounting Setup for Small Businesses
Falcon Invoice Discounting Setup for Small BusinessesFalcon Invoice Discounting Setup for Small Businesses
Falcon Invoice Discounting Setup for Small BusinessesFalcon investment
 
HAL Financial Performance Analysis and Future Prospects
HAL Financial Performance Analysis and Future ProspectsHAL Financial Performance Analysis and Future Prospects
HAL Financial Performance Analysis and Future ProspectsRajesh Gupta
 
Revolutionizing Industries: The Power of Carbon Components
Revolutionizing Industries: The Power of Carbon ComponentsRevolutionizing Industries: The Power of Carbon Components
Revolutionizing Industries: The Power of Carbon ComponentsConnova AG
 
Using Generative AI for Content Marketing
Using Generative AI for Content MarketingUsing Generative AI for Content Marketing
Using Generative AI for Content MarketingChuck Aikens
 
12 Conversion Rate Optimization Strategies for Ecommerce Websites.pdf
12 Conversion Rate Optimization Strategies for Ecommerce Websites.pdf12 Conversion Rate Optimization Strategies for Ecommerce Websites.pdf
12 Conversion Rate Optimization Strategies for Ecommerce Websites.pdfSOFTTECHHUB
 
TriStar Gold Corporate Presentation May 2024
TriStar Gold Corporate Presentation May 2024TriStar Gold Corporate Presentation May 2024
TriStar Gold Corporate Presentation May 2024Adnet Communications
 

Recently uploaded (20)

Understanding UAE Labour Law: Key Points for Employers and Employees
Understanding UAE Labour Law: Key Points for Employers and EmployeesUnderstanding UAE Labour Law: Key Points for Employers and Employees
Understanding UAE Labour Law: Key Points for Employers and Employees
 
Cracking the Change Management Code Main New.pptx
Cracking the Change Management Code Main New.pptxCracking the Change Management Code Main New.pptx
Cracking the Change Management Code Main New.pptx
 
Meaningful Technology for Humans: How Strategy Helps to Deliver Real Value fo...
Meaningful Technology for Humans: How Strategy Helps to Deliver Real Value fo...Meaningful Technology for Humans: How Strategy Helps to Deliver Real Value fo...
Meaningful Technology for Humans: How Strategy Helps to Deliver Real Value fo...
 
Raising Seed Capital by Steve Schlafman at RRE Ventures
Raising Seed Capital by Steve Schlafman at RRE VenturesRaising Seed Capital by Steve Schlafman at RRE Ventures
Raising Seed Capital by Steve Schlafman at RRE Ventures
 
The Inspiring Personality To Watch In 2024.pdf
The Inspiring Personality To Watch In 2024.pdfThe Inspiring Personality To Watch In 2024.pdf
The Inspiring Personality To Watch In 2024.pdf
 
MichaelStarkes_UncutGemsProjectSummary.pdf
MichaelStarkes_UncutGemsProjectSummary.pdfMichaelStarkes_UncutGemsProjectSummary.pdf
MichaelStarkes_UncutGemsProjectSummary.pdf
 
Copyright: What Creators and Users of Art Need to Know
Copyright: What Creators and Users of Art Need to KnowCopyright: What Creators and Users of Art Need to Know
Copyright: What Creators and Users of Art Need to Know
 
Toyota Kata Coaching for Agile Teams & Transformations
Toyota Kata Coaching for Agile Teams & TransformationsToyota Kata Coaching for Agile Teams & Transformations
Toyota Kata Coaching for Agile Teams & Transformations
 
بروفايل شركة ميار الخليج للاستشارات الهندسية.pdf
بروفايل شركة ميار الخليج للاستشارات الهندسية.pdfبروفايل شركة ميار الخليج للاستشارات الهندسية.pdf
بروفايل شركة ميار الخليج للاستشارات الهندسية.pdf
 
Equinox Gold Corporate Deck May 24th 2024
Equinox Gold Corporate Deck May 24th 2024Equinox Gold Corporate Deck May 24th 2024
Equinox Gold Corporate Deck May 24th 2024
 
Creative Ideas for Interactive Team Presentations
Creative Ideas for Interactive Team PresentationsCreative Ideas for Interactive Team Presentations
Creative Ideas for Interactive Team Presentations
 
How to Maintain Healthy Life style.pptx
How to Maintain  Healthy Life style.pptxHow to Maintain  Healthy Life style.pptx
How to Maintain Healthy Life style.pptx
 
NewBase 24 May 2024 Energy News issue - 1727 by Khaled Al Awadi_compresse...
NewBase   24 May  2024  Energy News issue - 1727 by Khaled Al Awadi_compresse...NewBase   24 May  2024  Energy News issue - 1727 by Khaled Al Awadi_compresse...
NewBase 24 May 2024 Energy News issue - 1727 by Khaled Al Awadi_compresse...
 
Salesforce Loyalty Management A Comprehensive.pdf
Salesforce Loyalty Management A Comprehensive.pdfSalesforce Loyalty Management A Comprehensive.pdf
Salesforce Loyalty Management A Comprehensive.pdf
 
Falcon Invoice Discounting Setup for Small Businesses
Falcon Invoice Discounting Setup for Small BusinessesFalcon Invoice Discounting Setup for Small Businesses
Falcon Invoice Discounting Setup for Small Businesses
 
HAL Financial Performance Analysis and Future Prospects
HAL Financial Performance Analysis and Future ProspectsHAL Financial Performance Analysis and Future Prospects
HAL Financial Performance Analysis and Future Prospects
 
Revolutionizing Industries: The Power of Carbon Components
Revolutionizing Industries: The Power of Carbon ComponentsRevolutionizing Industries: The Power of Carbon Components
Revolutionizing Industries: The Power of Carbon Components
 
Using Generative AI for Content Marketing
Using Generative AI for Content MarketingUsing Generative AI for Content Marketing
Using Generative AI for Content Marketing
 
12 Conversion Rate Optimization Strategies for Ecommerce Websites.pdf
12 Conversion Rate Optimization Strategies for Ecommerce Websites.pdf12 Conversion Rate Optimization Strategies for Ecommerce Websites.pdf
12 Conversion Rate Optimization Strategies for Ecommerce Websites.pdf
 
TriStar Gold Corporate Presentation May 2024
TriStar Gold Corporate Presentation May 2024TriStar Gold Corporate Presentation May 2024
TriStar Gold Corporate Presentation May 2024
 

Crawl Budget: Everything you Need to Know

  • 1. Crawl Budget: Everything you Need to Know https://www.slideshare.net/SallyR7 @salryam Sally Raymer HEYCAR /in/sally-r-seo/
  • 2. What is crawl budget and why does it matter?
  • 3.
  • 4. The bottom line ● Updated guidelines recognise medium and problematic sites ● Increasing the crawl rate to better fit your site ● Reducing crawl bloat
  • 5. How Google Interacts with our Sites
  • 6. Page is created Site requests page is crawled Google Crawls the page Indexing Does the page contain authoritative, high quality content that matches user intent? Page is indexed Crawled but not indexed Discovered not Indexed
  • 7. Crawl Demand Crawl Capacity ● Eligible indexable URLs ● How often content is created/ updated ● How much traffic you get to folders/ pages on site ● How quickly content loads ● How many errors are on site JS Rendering
  • 9. What is my crawl rate?
  • 10. Create Actionable data crawl requests/ 90 days = daily crawl budget indexable + errors + canonicals = no. of requested pages no. of requested pages / daily crawl budget = site ratio Website Daily crawl budget No. of requested crawls Site ratio No. Indexed Pages % Indexed pages vs requests example.com 20,000 4,826,000 x241 476,000 9.7%
  • 12. Uncover the coverage report ● Discovered not indexed ● Crawled not indexed ● Server errors ● Large number of canonical tags ● Large number redirects ● Indexed -blocked by robots.txt
  • 15. How often is Google crawling?
  • 16. What’s being crawled in comparison to organic traffic
  • 17. What response codes are being crawled? Define historic trends and volume
  • 18. What paginated pages & parameters are being crawled
  • 19. What file types is Google crawling? Images Video JS
  • 20. What is not being crawled?
  • 21. What commonalities do crawled pages share?
  • 23. Trying to Index All Pages
  • 24. Use Internal links to prioritise ● Capitalise on link equity ● Review link hierarchy regularly ● Strengthen content hubs ● Be mindful of where links are pointing
  • 25. Dynamically populate sitemaps ● Qualify all 200 canonicals ● Break your sitemaps up ● Don’t forget images and videos ● Minimise manual management
  • 26. Use robots.txt ● Prevent Google from crawling specific taxonomies ● Highlight top directories ● Link to sitemaps ● Remember: Blocked URLS can be indexed
  • 28. Three Questions to ask yourself 1.What are our KPIs? 2.How do we define user intent? 3.What is our web structure? Credit: https://www.screamingfrog.co.uk/site-architecture-crawl-visualisations/
  • 29. Low (low traffic) quality content If it doesn’t work, can we cut it? ● Are they high intent pages? ● Do they have quality backlinks? If the answer is no, what is the value in holding on to pages that aren’t working?
  • 30. Duplication & Cannibalisation ● Canonicals ➡️ Do we need both sets of content ● Bad taxonomy ➡️ Clean up the site structure ● Internal competition ➡️ Use GSC & keyword mapping
  • 31. Faceted navigation & parameters ● Do not deindex all parameters as a rule ● Are top level parameters valuable? ● Are static alternatives available? ● Is link equity and information architecture protected?
  • 32. Redirects & 404s ● Redirects used for OOS products ● Historical buildup of redirects or 404’s ● Embedded links will continue to be crawled ● Optimise the user journey not just site health
  • 33. Picking an Approach to Deindexation
  • 34. Noindex directive - if the page serves a purpose in the user journey, the noindex meta tag is probably the best method but does not take effect immediately.
  • 35. Blocking via robots.txt - this is a great way of keeping down crawl requests but it doesn’t stop indexation.
  • 36. 410 - Removing URLs that do not play a part in the user journey or provide value is a great way of dealing with crawl bloat.
  • 38. Ensure Content is Accessible If you have a large JS web app, ensuring that Google does not have to allocate more server resources than essential to render important content. Credit: https://nextjs.org/learn/basics/data-fetching/two-forms
  • 39. Updating & Creating New Content Regularly Creating fresh content that meets Google E-E-A-T quality rater guidelines will help increase crawl demand for your site.
  • 40. Improving Page Speed and Performance ● Invest in servers ● Improve page performance metrics ● Limit errors - redirect chains or loops
  • 41. Crawl Budget: A Happy Ending Identifying crawl rate Optimising the crawl budget Identifying crawl issues 1. 2. 3.
  • 42. THANK YOU Slideshare.Net/SallyRaymer2 @salraym /sally-r-seo/ Some great resources: Google Documentation: Managing Crawl Budget Verifying Google Bot for log file analysis Screaming Frog’s guide to log file analysis How to identify keyword cannibalisation SEJ internal link optimisation checklist for enterprise sites Mckinsey Consumer Decision Journey Mckinsey customer touchpoints