SlideShare a Scribd company logo
1 of 31
Download to read offline
Nitin Manchanda
Global Head of SEO at trivago
Wise use of Crawl Budget for
Better Rankings
Who is Nitin?
A technical SEO
Works at trivago
Started as Software Engineer
Host at SEMrush SEO Webinars
@_nitman
nitman
Crawl Budget?
Quick Summary
“For most sites, crawl budget is not something
to worry about. For really large sites, it
becomes something to consider looking at.
Prioritizing what to crawl, when, and how
much resource the server hosting the site can
allocate to crawling is more important for
bigger sites, or those that auto-generate
pages based on URL parameters”
- Gary Illyes, Google
Read Full Article: https://webmasters.googleblog.com/2017/01/what-crawl-budget-means-for-googlebot.html
Quick Summary
Crawl rate limit is
designed to help Google
not crawl your pages too
much and too fast where it
hurts your server
Quick Summary
Crawl demand is how
much Google wants to
crawl your pages. This is
based on how popular
your pages are and how
stale the content is in the
Google index
Quick Summary
Crawl budget is “taking
crawl rate and crawl
demand together”
Google defines crawl
budget as “the number of
URLs Googlebot can and
wants to crawl”
Too Technical?
In Easy Terms
Crawl budget is the
number of times a search
engine spider hits your
website during a given
period of time
For Example
If Googlebot typically hits
my site about 1,000 times
a month, I can say that 1K
is my monthly crawl
budget for Google
Please Note
There is no universal limit
as to the number and
frequency of these crawls;
we'll get to factors that
form your crawl budget in
a moment
Why should I care about
it?
Isn’t Google smart enough
to justify?
It is, for sure!
But..
You want bots to
discover all the
important pages on
your site
And..
You want bots to find
new content as quickly
as possible
And..
You don’t want to
waste this crawl budget
on unimportant pages
What’s the crawl budget
for my site?
There’s no way
to find that but
to get a feel of
it, you can..
1. Check average number
of pages crawled per
day in Google Search
Console, Bing
Webmaster Tools etc.
2. Check access logs to
understand more on
search patterns
All good! But what can be
done to use it wisely?
Pages are
crawlable
Check if all the important pages
are crawlable
- Not blocked from robots.txt
- Don’t have noindex tags or
X-Robots-Tag: noindex in
the header for these pages
Also, ensure the exact opposite
for unimportant pages.
Please note, if you use noindex meta
tag or X-Robots-Tag, you should not
disallow the page in robots.txt
The page must be crawled before the
tag will be seen and obeyed.
No redirect
chains
URL redirect = Waste of
crawl budget
Long redirect chains =
Spiders may drop off
before they reach your
destination page, which
means that page won’t be
indexed
No broken
links
If a page is broken, that’s
probably not important for
you. So, it doesn’t make
sense to waste crawl
budget on such pages.
Of course, fix it if it’s
important!
Handle URL
parameters
Spiders treat dynamic URLs as
separate pages even though they
lead to the same page (same
content). To handle that, you can
- Mark them as representative
parameters in search console
- Use canonical tags
For all the parameters which
doesn’t change the page content
like tracking parameters.
Clean and
up-to-date
sitemaps
Up-to-date and clean
sitemaps can help spiders
find your important URLs
quickly
Keep only index/important
URLs in your sitemaps
Site structure
and internal
linking
A well organised internal
linking structure would not
only make your customers
happy, but spiders would
also be able to understand
the importance of pages at
different levels
Site structure
and internal
linking
Make sure all important
pages are accessible
easily through site
navigation flow
Use RSS Feed
RSS feeds is a nice way to
boost your readership and
engagement, they’re also
among the most visited sites by
Googlebot.
When your website receives an
update like new products, blog
posts, website updates etc.,
submit it to Google’s Feed
Burner so that you’re sure it’s
properly indexed
Careful with
rich media files
Most of the spiders are still
struggling to crawl rich media
content like JavaScript, Flash
etc.
Google is much better than
others but, spending more
resources to crawl rich media,
hence using more crawl budget
for the same
Also read:
https://www.elephate.com/blog/javascript-vs-crawl-budget-ready-player-one/
Thank you!
@_nitman nitman

More Related Content

What's hot

Beyond Keywords: How To Create Content That Matches Your Buyer’s Journey
Beyond Keywords: How To Create Content That Matches Your Buyer’s JourneyBeyond Keywords: How To Create Content That Matches Your Buyer’s Journey
Beyond Keywords: How To Create Content That Matches Your Buyer’s JourneyMatthew Kay
 
How To Monitor Your Page Speed At Scale With PageSpeed API
How To Monitor Your Page Speed At Scale With PageSpeed APIHow To Monitor Your Page Speed At Scale With PageSpeed API
How To Monitor Your Page Speed At Scale With PageSpeed APIozanketenci
 
Monetising your Web Applications with Braintree
Monetising your Web Applications with BraintreeMonetising your Web Applications with Braintree
Monetising your Web Applications with BraintreeJoe Nash
 
Mapping Your Digital Customer Experience
Mapping Your Digital Customer ExperienceMapping Your Digital Customer Experience
Mapping Your Digital Customer ExperienceIan Keir
 
Site Search - patterns and analysis
Site Search - patterns and analysisSite Search - patterns and analysis
Site Search - patterns and analysisGerry White
 
How to run campaigns and client accounts using Scrum
How to run campaigns and client accounts using ScrumHow to run campaigns and client accounts using Scrum
How to run campaigns and client accounts using ScrumBethan Vincent
 
Entity HQ: Harnessing High Impact Profile Pages for SEO
Entity HQ: Harnessing High Impact Profile Pages for SEOEntity HQ: Harnessing High Impact Profile Pages for SEO
Entity HQ: Harnessing High Impact Profile Pages for SEOCrystal J Carter
 
SearchLeeds 2019 - Sam Marsden - Deepcrawl - Overcoming technical seo challen...
SearchLeeds 2019 - Sam Marsden - Deepcrawl - Overcoming technical seo challen...SearchLeeds 2019 - Sam Marsden - Deepcrawl - Overcoming technical seo challen...
SearchLeeds 2019 - Sam Marsden - Deepcrawl - Overcoming technical seo challen...SearchLeeds
 
Get your guest-posting submission published. Useful tips and automation of th...
Get your guest-posting submission published. Useful tips and automation of th...Get your guest-posting submission published. Useful tips and automation of th...
Get your guest-posting submission published. Useful tips and automation of th...Igor Gorbenko
 
WHAT IS RICH SNIPPET AND HOW IT WORKS?
WHAT IS RICH SNIPPET AND HOW IT WORKS?WHAT IS RICH SNIPPET AND HOW IT WORKS?
WHAT IS RICH SNIPPET AND HOW IT WORKS?Sandipti sahoo
 
Marketing Magic Mojo SEO 2016 PowerPoint
Marketing Magic Mojo SEO 2016 PowerPointMarketing Magic Mojo SEO 2016 PowerPoint
Marketing Magic Mojo SEO 2016 PowerPointMarketing Magic Mojo
 
Managing a Successful Website Migration
Managing a Successful Website MigrationManaging a Successful Website Migration
Managing a Successful Website MigrationSagittarius
 
MeasureFest - Weighted Sort - Google's not-so-hidden Algorithm that could Tra...
MeasureFest - Weighted Sort - Google's not-so-hidden Algorithm that could Tra...MeasureFest - Weighted Sort - Google's not-so-hidden Algorithm that could Tra...
MeasureFest - Weighted Sort - Google's not-so-hidden Algorithm that could Tra...Matthew Kay
 
Identifying Panda or Penguin on your websites - Digital Olympus - 12.03.2018
Identifying Panda or Penguin on your websites - Digital Olympus - 12.03.2018Identifying Panda or Penguin on your websites - Digital Olympus - 12.03.2018
Identifying Panda or Penguin on your websites - Digital Olympus - 12.03.2018Murat Yatağan
 
Creating Killer Dashboards That Inform & Persuade - SMX Advanced 2019 - Sam M...
Creating Killer Dashboards That Inform & Persuade - SMX Advanced 2019 - Sam M...Creating Killer Dashboards That Inform & Persuade - SMX Advanced 2019 - Sam M...
Creating Killer Dashboards That Inform & Persuade - SMX Advanced 2019 - Sam M...DeepCrawl
 
Data Studio for SEOs - Pint-sized Marketing Meetup 2019
Data Studio for SEOs - Pint-sized Marketing Meetup 2019Data Studio for SEOs - Pint-sized Marketing Meetup 2019
Data Studio for SEOs - Pint-sized Marketing Meetup 2019DeepCrawl
 

What's hot (20)

Seo Roadmap Presentation
Seo Roadmap PresentationSeo Roadmap Presentation
Seo Roadmap Presentation
 
Beyond Keywords: How To Create Content That Matches Your Buyer’s Journey
Beyond Keywords: How To Create Content That Matches Your Buyer’s JourneyBeyond Keywords: How To Create Content That Matches Your Buyer’s Journey
Beyond Keywords: How To Create Content That Matches Your Buyer’s Journey
 
How To Monitor Your Page Speed At Scale With PageSpeed API
How To Monitor Your Page Speed At Scale With PageSpeed APIHow To Monitor Your Page Speed At Scale With PageSpeed API
How To Monitor Your Page Speed At Scale With PageSpeed API
 
Monetising your Web Applications with Braintree
Monetising your Web Applications with BraintreeMonetising your Web Applications with Braintree
Monetising your Web Applications with Braintree
 
Seo Tips
Seo TipsSeo Tips
Seo Tips
 
Inbound Marketing - SEO
Inbound Marketing - SEOInbound Marketing - SEO
Inbound Marketing - SEO
 
Mapping Your Digital Customer Experience
Mapping Your Digital Customer ExperienceMapping Your Digital Customer Experience
Mapping Your Digital Customer Experience
 
Site Search - patterns and analysis
Site Search - patterns and analysisSite Search - patterns and analysis
Site Search - patterns and analysis
 
How to run campaigns and client accounts using Scrum
How to run campaigns and client accounts using ScrumHow to run campaigns and client accounts using Scrum
How to run campaigns and client accounts using Scrum
 
Entity HQ: Harnessing High Impact Profile Pages for SEO
Entity HQ: Harnessing High Impact Profile Pages for SEOEntity HQ: Harnessing High Impact Profile Pages for SEO
Entity HQ: Harnessing High Impact Profile Pages for SEO
 
SEO
SEO SEO
SEO
 
SearchLeeds 2019 - Sam Marsden - Deepcrawl - Overcoming technical seo challen...
SearchLeeds 2019 - Sam Marsden - Deepcrawl - Overcoming technical seo challen...SearchLeeds 2019 - Sam Marsden - Deepcrawl - Overcoming technical seo challen...
SearchLeeds 2019 - Sam Marsden - Deepcrawl - Overcoming technical seo challen...
 
Get your guest-posting submission published. Useful tips and automation of th...
Get your guest-posting submission published. Useful tips and automation of th...Get your guest-posting submission published. Useful tips and automation of th...
Get your guest-posting submission published. Useful tips and automation of th...
 
WHAT IS RICH SNIPPET AND HOW IT WORKS?
WHAT IS RICH SNIPPET AND HOW IT WORKS?WHAT IS RICH SNIPPET AND HOW IT WORKS?
WHAT IS RICH SNIPPET AND HOW IT WORKS?
 
Marketing Magic Mojo SEO 2016 PowerPoint
Marketing Magic Mojo SEO 2016 PowerPointMarketing Magic Mojo SEO 2016 PowerPoint
Marketing Magic Mojo SEO 2016 PowerPoint
 
Managing a Successful Website Migration
Managing a Successful Website MigrationManaging a Successful Website Migration
Managing a Successful Website Migration
 
MeasureFest - Weighted Sort - Google's not-so-hidden Algorithm that could Tra...
MeasureFest - Weighted Sort - Google's not-so-hidden Algorithm that could Tra...MeasureFest - Weighted Sort - Google's not-so-hidden Algorithm that could Tra...
MeasureFest - Weighted Sort - Google's not-so-hidden Algorithm that could Tra...
 
Identifying Panda or Penguin on your websites - Digital Olympus - 12.03.2018
Identifying Panda or Penguin on your websites - Digital Olympus - 12.03.2018Identifying Panda or Penguin on your websites - Digital Olympus - 12.03.2018
Identifying Panda or Penguin on your websites - Digital Olympus - 12.03.2018
 
Creating Killer Dashboards That Inform & Persuade - SMX Advanced 2019 - Sam M...
Creating Killer Dashboards That Inform & Persuade - SMX Advanced 2019 - Sam M...Creating Killer Dashboards That Inform & Persuade - SMX Advanced 2019 - Sam M...
Creating Killer Dashboards That Inform & Persuade - SMX Advanced 2019 - Sam M...
 
Data Studio for SEOs - Pint-sized Marketing Meetup 2019
Data Studio for SEOs - Pint-sized Marketing Meetup 2019Data Studio for SEOs - Pint-sized Marketing Meetup 2019
Data Studio for SEOs - Pint-sized Marketing Meetup 2019
 

Similar to Crawl Budget Optimisation at #dmwf2018

Crawl optimization - ( How to optimize to increase crawl budget)
Crawl optimization - ( How to optimize to increase crawl budget)Crawl optimization - ( How to optimize to increase crawl budget)
Crawl optimization - ( How to optimize to increase crawl budget)SyedFaraz41
 
How to perform a technical SEO audit and ramp up your content strategy in 10 ...
How to perform a technical SEO audit and ramp up your content strategy in 10 ...How to perform a technical SEO audit and ramp up your content strategy in 10 ...
How to perform a technical SEO audit and ramp up your content strategy in 10 ...Waqar Ahmad
 
Programmatic SEO: How to Dominate SEO Like TripAdvisor, Yelp and Zillow
Programmatic SEO: How to Dominate SEO Like TripAdvisor, Yelp and ZillowProgrammatic SEO: How to Dominate SEO Like TripAdvisor, Yelp and Zillow
Programmatic SEO: How to Dominate SEO Like TripAdvisor, Yelp and ZillowBernard Huang
 
Crawl Budget: Everything you Need to Know
Crawl Budget: Everything you Need to KnowCrawl Budget: Everything you Need to Know
Crawl Budget: Everything you Need to KnowSallyR7
 
Zero Budget Marketing - AdGlobal360
Zero Budget Marketing - AdGlobal360Zero Budget Marketing - AdGlobal360
Zero Budget Marketing - AdGlobal360AdGlobal360
 
Advanced latest Seo 2019
Advanced latest Seo 2019Advanced latest Seo 2019
Advanced latest Seo 2019RomeshKotian
 
NASSCOM - Power of Social Media & SEO for Lead Generation
NASSCOM - Power of Social Media & SEO for Lead GenerationNASSCOM - Power of Social Media & SEO for Lead Generation
NASSCOM - Power of Social Media & SEO for Lead GenerationNavneet Kaushal
 
Keeping Things Lean & Mean: Crawl Optimisation - Search Marketing Summit AU
Keeping Things Lean & Mean: Crawl Optimisation - Search Marketing Summit AUKeeping Things Lean & Mean: Crawl Optimisation - Search Marketing Summit AU
Keeping Things Lean & Mean: Crawl Optimisation - Search Marketing Summit AUJason Mun
 
Website Audit [On Page and Off Page] by Carl Benedic Pantaleon
Website Audit [On Page and Off Page] by Carl Benedic PantaleonWebsite Audit [On Page and Off Page] by Carl Benedic Pantaleon
Website Audit [On Page and Off Page] by Carl Benedic PantaleonJacque Doring
 
The Beginner's Guide to Googlebot Optimization
The Beginner's Guide to Googlebot OptimizationThe Beginner's Guide to Googlebot Optimization
The Beginner's Guide to Googlebot OptimizationCMI_Compas
 
SEO 101: How to Get Started Winning Google Search Traffic
SEO 101: How to Get Started Winning Google Search TrafficSEO 101: How to Get Started Winning Google Search Traffic
SEO 101: How to Get Started Winning Google Search TrafficBernard Huang
 
Modern SEO Techniques 2014
Modern SEO Techniques 2014Modern SEO Techniques 2014
Modern SEO Techniques 2014Milad Safarzadeh
 
The step by step guide to SEO Website Audit
The step by step guide to SEO Website Audit The step by step guide to SEO Website Audit
The step by step guide to SEO Website Audit amandacerry
 
Ecommerce seo checklist 15 questions to ask yourself to improve site ranking
Ecommerce seo checklist 15 questions to ask yourself to improve site rankingEcommerce seo checklist 15 questions to ask yourself to improve site ranking
Ecommerce seo checklist 15 questions to ask yourself to improve site rankingPros Global Inc
 
Free SEO Beginner's Guide From New Web Dimension
 Free SEO Beginner's Guide From New Web Dimension Free SEO Beginner's Guide From New Web Dimension
Free SEO Beginner's Guide From New Web DimensionNew Web Dimension
 
SEO services in chennai | Vlsa international services
SEO services in chennai | Vlsa international servicesSEO services in chennai | Vlsa international services
SEO services in chennai | Vlsa international servicesVelmurugan J
 

Similar to Crawl Budget Optimisation at #dmwf2018 (20)

Crawl optimization - ( How to optimize to increase crawl budget)
Crawl optimization - ( How to optimize to increase crawl budget)Crawl optimization - ( How to optimize to increase crawl budget)
Crawl optimization - ( How to optimize to increase crawl budget)
 
How to perform a technical SEO audit and ramp up your content strategy in 10 ...
How to perform a technical SEO audit and ramp up your content strategy in 10 ...How to perform a technical SEO audit and ramp up your content strategy in 10 ...
How to perform a technical SEO audit and ramp up your content strategy in 10 ...
 
Programmatic SEO: How to Dominate SEO Like TripAdvisor, Yelp and Zillow
Programmatic SEO: How to Dominate SEO Like TripAdvisor, Yelp and ZillowProgrammatic SEO: How to Dominate SEO Like TripAdvisor, Yelp and Zillow
Programmatic SEO: How to Dominate SEO Like TripAdvisor, Yelp and Zillow
 
Crawl Budget: Everything you Need to Know
Crawl Budget: Everything you Need to KnowCrawl Budget: Everything you Need to Know
Crawl Budget: Everything you Need to Know
 
Foxtail Website Audit
Foxtail Website AuditFoxtail Website Audit
Foxtail Website Audit
 
Zero Budget Marketing - AdGlobal360
Zero Budget Marketing - AdGlobal360Zero Budget Marketing - AdGlobal360
Zero Budget Marketing - AdGlobal360
 
On site optimization
On site optimizationOn site optimization
On site optimization
 
Advanced latest Seo 2019
Advanced latest Seo 2019Advanced latest Seo 2019
Advanced latest Seo 2019
 
NASSCOM - Power of Social Media & SEO for Lead Generation
NASSCOM - Power of Social Media & SEO for Lead GenerationNASSCOM - Power of Social Media & SEO for Lead Generation
NASSCOM - Power of Social Media & SEO for Lead Generation
 
Keeping Things Lean & Mean: Crawl Optimisation - Search Marketing Summit AU
Keeping Things Lean & Mean: Crawl Optimisation - Search Marketing Summit AUKeeping Things Lean & Mean: Crawl Optimisation - Search Marketing Summit AU
Keeping Things Lean & Mean: Crawl Optimisation - Search Marketing Summit AU
 
Website Audit [On Page and Off Page] by Carl Benedic Pantaleon
Website Audit [On Page and Off Page] by Carl Benedic PantaleonWebsite Audit [On Page and Off Page] by Carl Benedic Pantaleon
Website Audit [On Page and Off Page] by Carl Benedic Pantaleon
 
The Beginner's Guide to Googlebot Optimization
The Beginner's Guide to Googlebot OptimizationThe Beginner's Guide to Googlebot Optimization
The Beginner's Guide to Googlebot Optimization
 
Seo basics
Seo basicsSeo basics
Seo basics
 
SEO 101: How to Get Started Winning Google Search Traffic
SEO 101: How to Get Started Winning Google Search TrafficSEO 101: How to Get Started Winning Google Search Traffic
SEO 101: How to Get Started Winning Google Search Traffic
 
Modern SEO Techniques 2014
Modern SEO Techniques 2014Modern SEO Techniques 2014
Modern SEO Techniques 2014
 
Seo on page ppt
Seo on page pptSeo on page ppt
Seo on page ppt
 
The step by step guide to SEO Website Audit
The step by step guide to SEO Website Audit The step by step guide to SEO Website Audit
The step by step guide to SEO Website Audit
 
Ecommerce seo checklist 15 questions to ask yourself to improve site ranking
Ecommerce seo checklist 15 questions to ask yourself to improve site rankingEcommerce seo checklist 15 questions to ask yourself to improve site ranking
Ecommerce seo checklist 15 questions to ask yourself to improve site ranking
 
Free SEO Beginner's Guide From New Web Dimension
 Free SEO Beginner's Guide From New Web Dimension Free SEO Beginner's Guide From New Web Dimension
Free SEO Beginner's Guide From New Web Dimension
 
SEO services in chennai | Vlsa international services
SEO services in chennai | Vlsa international servicesSEO services in chennai | Vlsa international services
SEO services in chennai | Vlsa international services
 

More from Nitin Manchanda

SEO Forecasting by Nitin Manchanda at Berlin SEO & Content Club
SEO Forecasting by Nitin Manchanda at Berlin SEO & Content ClubSEO Forecasting by Nitin Manchanda at Berlin SEO & Content Club
SEO Forecasting by Nitin Manchanda at Berlin SEO & Content ClubNitin Manchanda
 
SEO Migrations for International Web Setups
SEO Migrations for International Web SetupsSEO Migrations for International Web Setups
SEO Migrations for International Web SetupsNitin Manchanda
 
SEO Prompt Engineering - A Duda Webinar
SEO Prompt Engineering - A Duda WebinarSEO Prompt Engineering - A Duda Webinar
SEO Prompt Engineering - A Duda WebinarNitin Manchanda
 
Data Driven Approach to Scale SEO at BrightonSEO 2023
Data Driven Approach to Scale SEO at BrightonSEO 2023Data Driven Approach to Scale SEO at BrightonSEO 2023
Data Driven Approach to Scale SEO at BrightonSEO 2023Nitin Manchanda
 
SEO at Scale - BrightonSEO April 2022
SEO at Scale - BrightonSEO April 2022SEO at Scale - BrightonSEO April 2022
SEO at Scale - BrightonSEO April 2022Nitin Manchanda
 
JavaScript and SEO relationship explained with tips to improve
JavaScript and SEO relationship explained with tips to improveJavaScript and SEO relationship explained with tips to improve
JavaScript and SEO relationship explained with tips to improveNitin Manchanda
 
International SEO - Expand Your Business & Grow Organically
International SEO - Expand Your Business & Grow OrganicallyInternational SEO - Expand Your Business & Grow Organically
International SEO - Expand Your Business & Grow OrganicallyNitin Manchanda
 
SEO Myths & Tips To Organise Organic Visibility
SEO Myths & Tips To Organise Organic VisibilitySEO Myths & Tips To Organise Organic Visibility
SEO Myths & Tips To Organise Organic VisibilityNitin Manchanda
 

More from Nitin Manchanda (8)

SEO Forecasting by Nitin Manchanda at Berlin SEO & Content Club
SEO Forecasting by Nitin Manchanda at Berlin SEO & Content ClubSEO Forecasting by Nitin Manchanda at Berlin SEO & Content Club
SEO Forecasting by Nitin Manchanda at Berlin SEO & Content Club
 
SEO Migrations for International Web Setups
SEO Migrations for International Web SetupsSEO Migrations for International Web Setups
SEO Migrations for International Web Setups
 
SEO Prompt Engineering - A Duda Webinar
SEO Prompt Engineering - A Duda WebinarSEO Prompt Engineering - A Duda Webinar
SEO Prompt Engineering - A Duda Webinar
 
Data Driven Approach to Scale SEO at BrightonSEO 2023
Data Driven Approach to Scale SEO at BrightonSEO 2023Data Driven Approach to Scale SEO at BrightonSEO 2023
Data Driven Approach to Scale SEO at BrightonSEO 2023
 
SEO at Scale - BrightonSEO April 2022
SEO at Scale - BrightonSEO April 2022SEO at Scale - BrightonSEO April 2022
SEO at Scale - BrightonSEO April 2022
 
JavaScript and SEO relationship explained with tips to improve
JavaScript and SEO relationship explained with tips to improveJavaScript and SEO relationship explained with tips to improve
JavaScript and SEO relationship explained with tips to improve
 
International SEO - Expand Your Business & Grow Organically
International SEO - Expand Your Business & Grow OrganicallyInternational SEO - Expand Your Business & Grow Organically
International SEO - Expand Your Business & Grow Organically
 
SEO Myths & Tips To Organise Organic Visibility
SEO Myths & Tips To Organise Organic VisibilitySEO Myths & Tips To Organise Organic Visibility
SEO Myths & Tips To Organise Organic Visibility
 

Recently uploaded

5 TH SEM BBA DIGITAL MARKETING NOTES.docx
5 TH SEM BBA DIGITAL MARKETING NOTES.docx5 TH SEM BBA DIGITAL MARKETING NOTES.docx
5 TH SEM BBA DIGITAL MARKETING NOTES.docxmskarthik1435
 
Master the art of Social Selling to increase sales by fostering relationships...
Master the art of Social Selling to increase sales by fostering relationships...Master the art of Social Selling to increase sales by fostering relationships...
Master the art of Social Selling to increase sales by fostering relationships...VereigenMedia1
 
HAGAN_Katalog_Saison23-24_Overview_Preview
HAGAN_Katalog_Saison23-24_Overview_PreviewHAGAN_Katalog_Saison23-24_Overview_Preview
HAGAN_Katalog_Saison23-24_Overview_PreviewEsteller
 
Catálogo HYD 2024 gama completa Hydrapak
Catálogo HYD 2024 gama completa HydrapakCatálogo HYD 2024 gama completa Hydrapak
Catálogo HYD 2024 gama completa HydrapakEsteller
 
AMAZON Copywriting Portfolio by Cielo Evangelista
AMAZON Copywriting Portfolio by Cielo EvangelistaAMAZON Copywriting Portfolio by Cielo Evangelista
AMAZON Copywriting Portfolio by Cielo Evangelistacrevangelista
 
Llanai Buyer Persona & Segmentation Strategy
Llanai Buyer Persona & Segmentation StrategyLlanai Buyer Persona & Segmentation Strategy
Llanai Buyer Persona & Segmentation StrategyMarianna Nakou
 
2024 WTF - what's working in mobile user acquisition
2024 WTF - what's working in mobile user acquisition2024 WTF - what's working in mobile user acquisition
2024 WTF - what's working in mobile user acquisitionJohn Koetsier
 
Paul Russell Confidential Resume for Fahlo.pdf
Paul Russell Confidential Resume for Fahlo.pdfPaul Russell Confidential Resume for Fahlo.pdf
Paul Russell Confidential Resume for Fahlo.pdfpaul8402
 
Make Your Message Go Viral with Nugget Global's Press Release Distribution Se...
Make Your Message Go Viral with Nugget Global's Press Release Distribution Se...Make Your Message Go Viral with Nugget Global's Press Release Distribution Se...
Make Your Message Go Viral with Nugget Global's Press Release Distribution Se...Nugget Global
 
Lesotho-Botswana Water Project Brand Manual developed with new logo
Lesotho-Botswana Water Project Brand Manual developed with  new logoLesotho-Botswana Water Project Brand Manual developed with  new logo
Lesotho-Botswana Water Project Brand Manual developed with new logonelaohaimbodi
 
Digital Marketing complete introduction.
Digital Marketing complete introduction.Digital Marketing complete introduction.
Digital Marketing complete introduction.Kashish Bindra
 
History of JWT by The Knowledge Center.pdf
History of JWT by The Knowledge Center.pdfHistory of JWT by The Knowledge Center.pdf
History of JWT by The Knowledge Center.pdfwilliam charnock
 
Francesco d’Angela, Service Designer di @HintoGroup- “Oltre la Frontiera Crea...
Francesco d’Angela, Service Designer di @HintoGroup- “Oltre la Frontiera Crea...Francesco d’Angela, Service Designer di @HintoGroup- “Oltre la Frontiera Crea...
Francesco d’Angela, Service Designer di @HintoGroup- “Oltre la Frontiera Crea...Associazione Digital Days
 
The Ultimate Guide to Financial Advertising Strategies.pdf
The Ultimate Guide to Financial Advertising Strategies.pdfThe Ultimate Guide to Financial Advertising Strategies.pdf
The Ultimate Guide to Financial Advertising Strategies.pdfFinance Advertising Network
 
Fritschi Collection 2022/23 EN gama completa
Fritschi Collection 2022/23 EN gama completaFritschi Collection 2022/23 EN gama completa
Fritschi Collection 2022/23 EN gama completaEsteller
 
20 Top Social Media Tips for Peer Specialists
20 Top Social Media Tips for Peer Specialists20 Top Social Media Tips for Peer Specialists
20 Top Social Media Tips for Peer Specialistsmlicam615
 
Professional Sales Representative by Sahil Srivastava.pptx
Professional Sales Representative by Sahil Srivastava.pptxProfessional Sales Representative by Sahil Srivastava.pptx
Professional Sales Representative by Sahil Srivastava.pptxSahil Srivastava
 
Catálogo Sea To Summit 2024 gama compelta
Catálogo Sea To Summit 2024 gama compeltaCatálogo Sea To Summit 2024 gama compelta
Catálogo Sea To Summit 2024 gama compeltaEsteller
 
TAM Sports-IPL 17 Advertising Report- M01 - M15.xlsx - IPL 17 FCT (Commercial...
TAM Sports-IPL 17 Advertising Report- M01 - M15.xlsx - IPL 17 FCT (Commercial...TAM Sports-IPL 17 Advertising Report- M01 - M15.xlsx - IPL 17 FCT (Commercial...
TAM Sports-IPL 17 Advertising Report- M01 - M15.xlsx - IPL 17 FCT (Commercial...Social Samosa
 
Social Media Content Lecture for Advanced Digital & Social Media Strategy at ...
Social Media Content Lecture for Advanced Digital & Social Media Strategy at ...Social Media Content Lecture for Advanced Digital & Social Media Strategy at ...
Social Media Content Lecture for Advanced Digital & Social Media Strategy at ...Valters Lauzums
 

Recently uploaded (20)

5 TH SEM BBA DIGITAL MARKETING NOTES.docx
5 TH SEM BBA DIGITAL MARKETING NOTES.docx5 TH SEM BBA DIGITAL MARKETING NOTES.docx
5 TH SEM BBA DIGITAL MARKETING NOTES.docx
 
Master the art of Social Selling to increase sales by fostering relationships...
Master the art of Social Selling to increase sales by fostering relationships...Master the art of Social Selling to increase sales by fostering relationships...
Master the art of Social Selling to increase sales by fostering relationships...
 
HAGAN_Katalog_Saison23-24_Overview_Preview
HAGAN_Katalog_Saison23-24_Overview_PreviewHAGAN_Katalog_Saison23-24_Overview_Preview
HAGAN_Katalog_Saison23-24_Overview_Preview
 
Catálogo HYD 2024 gama completa Hydrapak
Catálogo HYD 2024 gama completa HydrapakCatálogo HYD 2024 gama completa Hydrapak
Catálogo HYD 2024 gama completa Hydrapak
 
AMAZON Copywriting Portfolio by Cielo Evangelista
AMAZON Copywriting Portfolio by Cielo EvangelistaAMAZON Copywriting Portfolio by Cielo Evangelista
AMAZON Copywriting Portfolio by Cielo Evangelista
 
Llanai Buyer Persona & Segmentation Strategy
Llanai Buyer Persona & Segmentation StrategyLlanai Buyer Persona & Segmentation Strategy
Llanai Buyer Persona & Segmentation Strategy
 
2024 WTF - what's working in mobile user acquisition
2024 WTF - what's working in mobile user acquisition2024 WTF - what's working in mobile user acquisition
2024 WTF - what's working in mobile user acquisition
 
Paul Russell Confidential Resume for Fahlo.pdf
Paul Russell Confidential Resume for Fahlo.pdfPaul Russell Confidential Resume for Fahlo.pdf
Paul Russell Confidential Resume for Fahlo.pdf
 
Make Your Message Go Viral with Nugget Global's Press Release Distribution Se...
Make Your Message Go Viral with Nugget Global's Press Release Distribution Se...Make Your Message Go Viral with Nugget Global's Press Release Distribution Se...
Make Your Message Go Viral with Nugget Global's Press Release Distribution Se...
 
Lesotho-Botswana Water Project Brand Manual developed with new logo
Lesotho-Botswana Water Project Brand Manual developed with  new logoLesotho-Botswana Water Project Brand Manual developed with  new logo
Lesotho-Botswana Water Project Brand Manual developed with new logo
 
Digital Marketing complete introduction.
Digital Marketing complete introduction.Digital Marketing complete introduction.
Digital Marketing complete introduction.
 
History of JWT by The Knowledge Center.pdf
History of JWT by The Knowledge Center.pdfHistory of JWT by The Knowledge Center.pdf
History of JWT by The Knowledge Center.pdf
 
Francesco d’Angela, Service Designer di @HintoGroup- “Oltre la Frontiera Crea...
Francesco d’Angela, Service Designer di @HintoGroup- “Oltre la Frontiera Crea...Francesco d’Angela, Service Designer di @HintoGroup- “Oltre la Frontiera Crea...
Francesco d’Angela, Service Designer di @HintoGroup- “Oltre la Frontiera Crea...
 
The Ultimate Guide to Financial Advertising Strategies.pdf
The Ultimate Guide to Financial Advertising Strategies.pdfThe Ultimate Guide to Financial Advertising Strategies.pdf
The Ultimate Guide to Financial Advertising Strategies.pdf
 
Fritschi Collection 2022/23 EN gama completa
Fritschi Collection 2022/23 EN gama completaFritschi Collection 2022/23 EN gama completa
Fritschi Collection 2022/23 EN gama completa
 
20 Top Social Media Tips for Peer Specialists
20 Top Social Media Tips for Peer Specialists20 Top Social Media Tips for Peer Specialists
20 Top Social Media Tips for Peer Specialists
 
Professional Sales Representative by Sahil Srivastava.pptx
Professional Sales Representative by Sahil Srivastava.pptxProfessional Sales Representative by Sahil Srivastava.pptx
Professional Sales Representative by Sahil Srivastava.pptx
 
Catálogo Sea To Summit 2024 gama compelta
Catálogo Sea To Summit 2024 gama compeltaCatálogo Sea To Summit 2024 gama compelta
Catálogo Sea To Summit 2024 gama compelta
 
TAM Sports-IPL 17 Advertising Report- M01 - M15.xlsx - IPL 17 FCT (Commercial...
TAM Sports-IPL 17 Advertising Report- M01 - M15.xlsx - IPL 17 FCT (Commercial...TAM Sports-IPL 17 Advertising Report- M01 - M15.xlsx - IPL 17 FCT (Commercial...
TAM Sports-IPL 17 Advertising Report- M01 - M15.xlsx - IPL 17 FCT (Commercial...
 
Social Media Content Lecture for Advanced Digital & Social Media Strategy at ...
Social Media Content Lecture for Advanced Digital & Social Media Strategy at ...Social Media Content Lecture for Advanced Digital & Social Media Strategy at ...
Social Media Content Lecture for Advanced Digital & Social Media Strategy at ...
 

Crawl Budget Optimisation at #dmwf2018

  • 1. Nitin Manchanda Global Head of SEO at trivago Wise use of Crawl Budget for Better Rankings
  • 2. Who is Nitin? A technical SEO Works at trivago Started as Software Engineer Host at SEMrush SEO Webinars @_nitman nitman
  • 4. Quick Summary “For most sites, crawl budget is not something to worry about. For really large sites, it becomes something to consider looking at. Prioritizing what to crawl, when, and how much resource the server hosting the site can allocate to crawling is more important for bigger sites, or those that auto-generate pages based on URL parameters” - Gary Illyes, Google Read Full Article: https://webmasters.googleblog.com/2017/01/what-crawl-budget-means-for-googlebot.html
  • 5. Quick Summary Crawl rate limit is designed to help Google not crawl your pages too much and too fast where it hurts your server
  • 6. Quick Summary Crawl demand is how much Google wants to crawl your pages. This is based on how popular your pages are and how stale the content is in the Google index
  • 7. Quick Summary Crawl budget is “taking crawl rate and crawl demand together” Google defines crawl budget as “the number of URLs Googlebot can and wants to crawl”
  • 9. In Easy Terms Crawl budget is the number of times a search engine spider hits your website during a given period of time
  • 10. For Example If Googlebot typically hits my site about 1,000 times a month, I can say that 1K is my monthly crawl budget for Google
  • 11. Please Note There is no universal limit as to the number and frequency of these crawls; we'll get to factors that form your crawl budget in a moment
  • 12. Why should I care about it?
  • 13. Isn’t Google smart enough to justify?
  • 14. It is, for sure!
  • 15. But.. You want bots to discover all the important pages on your site
  • 16. And.. You want bots to find new content as quickly as possible
  • 17. And.. You don’t want to waste this crawl budget on unimportant pages
  • 18. What’s the crawl budget for my site?
  • 19. There’s no way to find that but to get a feel of it, you can.. 1. Check average number of pages crawled per day in Google Search Console, Bing Webmaster Tools etc. 2. Check access logs to understand more on search patterns
  • 20. All good! But what can be done to use it wisely?
  • 21. Pages are crawlable Check if all the important pages are crawlable - Not blocked from robots.txt - Don’t have noindex tags or X-Robots-Tag: noindex in the header for these pages Also, ensure the exact opposite for unimportant pages.
  • 22. Please note, if you use noindex meta tag or X-Robots-Tag, you should not disallow the page in robots.txt The page must be crawled before the tag will be seen and obeyed.
  • 23. No redirect chains URL redirect = Waste of crawl budget Long redirect chains = Spiders may drop off before they reach your destination page, which means that page won’t be indexed
  • 24. No broken links If a page is broken, that’s probably not important for you. So, it doesn’t make sense to waste crawl budget on such pages. Of course, fix it if it’s important!
  • 25. Handle URL parameters Spiders treat dynamic URLs as separate pages even though they lead to the same page (same content). To handle that, you can - Mark them as representative parameters in search console - Use canonical tags For all the parameters which doesn’t change the page content like tracking parameters.
  • 26. Clean and up-to-date sitemaps Up-to-date and clean sitemaps can help spiders find your important URLs quickly Keep only index/important URLs in your sitemaps
  • 27. Site structure and internal linking A well organised internal linking structure would not only make your customers happy, but spiders would also be able to understand the importance of pages at different levels
  • 28. Site structure and internal linking Make sure all important pages are accessible easily through site navigation flow
  • 29. Use RSS Feed RSS feeds is a nice way to boost your readership and engagement, they’re also among the most visited sites by Googlebot. When your website receives an update like new products, blog posts, website updates etc., submit it to Google’s Feed Burner so that you’re sure it’s properly indexed
  • 30. Careful with rich media files Most of the spiders are still struggling to crawl rich media content like JavaScript, Flash etc. Google is much better than others but, spending more resources to crawl rich media, hence using more crawl budget for the same Also read: https://www.elephate.com/blog/javascript-vs-crawl-budget-ready-player-one/