Combatting Crawl Bloat & Pruning Your Content Effectively

Charlie Whitworth
Charlie WhitworthHead of SEO at Big Brand Ideas | Whitworth SEO
Charlie Whitworth
SEO Director
@WhitworthSEO
http://www.slideshare.net/whitworthseo
Banc
what will we cover?
bancmedia.com
Crawl Bloat & Pruning
@WhitworthSEO
beat the bloat
Crawl Bloat
Making search engines work too hard
and not taking full advantage of your
crawl budget
@WhitworthSEO bancmedia.com
crawl definitions
=
Crawl Budget
bancmedia.com@WhitworthSEO
Crawl Rate & Crawl Demand
crawl rate (limit) : maximum fetching rate
Search Console Limit
s
bancmedia.com@WhitworthSEO
Website/Server Health
crawl demand
Popularity (Traffic) Staleness
@WhitworthSEO bancmedia.com
How much does Google want to see your URLs?
Crawl budget – determined by rate & demand
What Google can & wants to crawl
@WhitworthSEO bancmedia.com
• Empty pages
• No valued added
• Dupe
• Soft 404
• SPAM
• Hacked content
crawl budget
“Prioritizing what to crawl, when, and how much resource the
server hosting the site can allocate to crawling is more important
for bigger sites, or those that auto-generate pages based on URL
parameters”
Gary Ilyes
bancmedia.com@WhitworthSEO
how to identify crawl bloat
bancmedia.com@WhitworthSEO
how to identify crawl bloat
bancmedia.com@WhitworthSEO
why is this a problem?
bancmedia.com@WhitworthSEO
Don’t work them too hard
Search engine bots hate crawling rubbish
why is this a problem?
Engagement metrics are key
especially now we’re “mobile first”
@WhitworthSEO bancmedia.com
examples of rubbish: search result URLs
/search/
bancmedia.com@WhitworthSEO
/sessionkey=
/search-results?q=
examples of rubbish: paginated URLs
bancmedia.com
rel=next/prevnoindex
@WhitworthSEO
examples of rubbish: faceted navigations
json-ld
disallow parameters
bancmedia.com@WhitworthSEO
noindex
who’s seen this?
cut the crap
Get Rid Of It
@WhitworthSEO bancmedia.com
plan
noindex
<meta name=“robots" content=“noindex">
Won’t deal with crawl bloat in the short term
Still an effective tactic
bancmedia.com@WhitworthSEO
url parameters
@WhitworthSEO
Powerful tool when used properly
No need for developer support
bancmedia.com
xml sitemaps
Redirected URLs
Noindexed URLs
Poor quality pages
@WhitworthSEO bancmedia.com
robots.txt
bancmedia.com@WhitworthSEO
Exercise Caution!
Powerful crawl bloat tool when used effectively
robots.txt
Blocking Crawl Paths – Stems Authority Flow
bancmedia.com@WhitworthSEO
Google needs to see tags to honour them!
google’s hint
Google gives you all you need to
monitor your results
All these features are your friend
s
@WhitworthSEO bancmedia.com
crawl stats
Great way to gauge your efforts
@WhitworthSEO bancmedia.com
bancmedia.com@WhitworthSEO
Pruning
the prune
pruning – the fun bit
SEO Housekeepin
g
Now you have your crawl under control,
show off your best bits
bancmedia.com@WhitworthSEO
pruning – why?
• Spinning
• Excessive blogging
• Sparse articles
@WhitworthSEO bancmedia.com
• Over optimisation
• Saturation
stage one – content audit/content is king
Content auditing still essential nonetheless
bancmedia.com@WhitworthSEO
SEO’s most annoying cliché
stage two: the prune
410 – Gone
@WhitworthSEO bancmedia.com
For pages you want gone forever
stage two: the prune
bancmedia.com@WhitworthSEO
Won’t help crawl bloat short term
Will disappear eventually*
stage two: the prune
Crawl Bloat
Authority
Indexation
@WhitworthSEO bancmedia.com
stage two: the prune
NoIndex
Will be crawled less over time ensures a high quality SER
P
For duplicate content, cannibalisation and API style pages
@WhitworthSEO bancmedia.com
stage two: canonical
For duplicate content, cannibalisation and API style pages
@WhitworthSEO bancmedia.com
canonical
stage three: content repurposing/enhancement
@WhitworthSEO bancmedia.com
Evergreen
Seasonal
Long form
Research
search quality guidelines
@WhitworthSEO bancmedia.com
Search Quality Evaluator Guidelines
stage four: the finishing touch
Link Profile Analysis
Links pointing at excluded
pages
Usual disavow request
@WhitworthSEO bancmedia.com
Don’t ruin all your hard work with a crap link profile
crawl bloat and pruning checklist
Does your site have crawl bloat?
Choose the appropriate fixes for your site
Prune your remaining content
Continually improve and enhance
@WhitworthSEO bancmedia.com
final tips: thanks for coming
@WhitworthSEO bancmedia.com
Thanks for Listening!
2 Tips
1 of 39

Recommended

Easier and faster tagging with Kermit by
Easier and faster tagging with KermitEasier and faster tagging with Kermit
Easier and faster tagging with KermitAlban Gérôme
2.6K views68 slides
11 Advanced Uses of Screaming Frog Nov 2019 DMSS by
11 Advanced Uses of Screaming Frog Nov 2019 DMSS11 Advanced Uses of Screaming Frog Nov 2019 DMSS
11 Advanced Uses of Screaming Frog Nov 2019 DMSSOliver Brett
3.4K views290 slides
How to Improve Your Website's Indexation - Sean Butcher Brighton SEO Present... by
How to Improve Your Website's Indexation  - Sean Butcher Brighton SEO Present...How to Improve Your Website's Indexation  - Sean Butcher Brighton SEO Present...
How to Improve Your Website's Indexation - Sean Butcher Brighton SEO Present...Sean Butcher
3.8K views167 slides
SEO Myths Busted II by
SEO Myths Busted IISEO Myths Busted II
SEO Myths Busted IIFinishJoomla
1.4K views39 slides
Runnings SEO on shoestring budget by Yauhen Khutarniuk by
Runnings SEO on shoestring budget by Yauhen KhutarniukRunnings SEO on shoestring budget by Yauhen Khutarniuk
Runnings SEO on shoestring budget by Yauhen KhutarniukLink-Assistant.Com
14K views38 slides
The State of the Web: Pagination and Infinite Scroll by
The State of the Web: Pagination and Infinite ScrollThe State of the Web: Pagination and Infinite Scroll
The State of the Web: Pagination and Infinite ScrollAdam Gent
11.7K views65 slides

More Related Content

What's hot

#CMC2019: Advanced SEO: Competitive intelligence, Web Scraping, and More. by
#CMC2019: Advanced SEO: Competitive intelligence, Web Scraping, and More. #CMC2019: Advanced SEO: Competitive intelligence, Web Scraping, and More.
#CMC2019: Advanced SEO: Competitive intelligence, Web Scraping, and More. Mel Sciorra
873 views45 slides
Single Page Apps - Gerry White @ BrightonSEO by
Single Page Apps - Gerry White @ BrightonSEOSingle Page Apps - Gerry White @ BrightonSEO
Single Page Apps - Gerry White @ BrightonSEOGerry White
466 views87 slides
Building an SEO Exponential Growth model by closing your content gaps by
Building an SEO Exponential Growth model by closing your content gapsBuilding an SEO Exponential Growth model by closing your content gaps
Building an SEO Exponential Growth model by closing your content gapsRazvan Gavrilas
29K views63 slides
Conflicting Website Signals & Confused Search Engines | Raleigh SEO Conferenc... by
Conflicting Website Signals & Confused Search Engines | Raleigh SEO Conferenc...Conflicting Website Signals & Confused Search Engines | Raleigh SEO Conferenc...
Conflicting Website Signals & Confused Search Engines | Raleigh SEO Conferenc...DeepCrawl
829 views82 slides
How to Survive & Thrive after Mobile First Indexing - Rachel Costello, Techni... by
How to Survive & Thrive after Mobile First Indexing - Rachel Costello, Techni...How to Survive & Thrive after Mobile First Indexing - Rachel Costello, Techni...
How to Survive & Thrive after Mobile First Indexing - Rachel Costello, Techni...DeepCrawl
610 views34 slides
HOW TO INCREASE YOUR TRAFFIC 5X WITH THIS ONE SEO METHOD by
HOW TO INCREASE YOUR TRAFFIC 5X WITH THIS ONE SEO METHODHOW TO INCREASE YOUR TRAFFIC 5X WITH THIS ONE SEO METHOD
HOW TO INCREASE YOUR TRAFFIC 5X WITH THIS ONE SEO METHODChristoph C. Cemper
15.1K views38 slides

What's hot(19)

#CMC2019: Advanced SEO: Competitive intelligence, Web Scraping, and More. by Mel Sciorra
#CMC2019: Advanced SEO: Competitive intelligence, Web Scraping, and More. #CMC2019: Advanced SEO: Competitive intelligence, Web Scraping, and More.
#CMC2019: Advanced SEO: Competitive intelligence, Web Scraping, and More.
Mel Sciorra873 views
Single Page Apps - Gerry White @ BrightonSEO by Gerry White
Single Page Apps - Gerry White @ BrightonSEOSingle Page Apps - Gerry White @ BrightonSEO
Single Page Apps - Gerry White @ BrightonSEO
Gerry White466 views
Building an SEO Exponential Growth model by closing your content gaps by Razvan Gavrilas
Building an SEO Exponential Growth model by closing your content gapsBuilding an SEO Exponential Growth model by closing your content gaps
Building an SEO Exponential Growth model by closing your content gaps
Razvan Gavrilas29K views
Conflicting Website Signals & Confused Search Engines | Raleigh SEO Conferenc... by DeepCrawl
Conflicting Website Signals & Confused Search Engines | Raleigh SEO Conferenc...Conflicting Website Signals & Confused Search Engines | Raleigh SEO Conferenc...
Conflicting Website Signals & Confused Search Engines | Raleigh SEO Conferenc...
DeepCrawl829 views
How to Survive & Thrive after Mobile First Indexing - Rachel Costello, Techni... by DeepCrawl
How to Survive & Thrive after Mobile First Indexing - Rachel Costello, Techni...How to Survive & Thrive after Mobile First Indexing - Rachel Costello, Techni...
How to Survive & Thrive after Mobile First Indexing - Rachel Costello, Techni...
DeepCrawl610 views
HOW TO INCREASE YOUR TRAFFIC 5X WITH THIS ONE SEO METHOD by Christoph C. Cemper
HOW TO INCREASE YOUR TRAFFIC 5X WITH THIS ONE SEO METHODHOW TO INCREASE YOUR TRAFFIC 5X WITH THIS ONE SEO METHOD
HOW TO INCREASE YOUR TRAFFIC 5X WITH THIS ONE SEO METHOD
Christoph C. Cemper15.1K views
WordPress SEO Mistakes that Kill - BigDigital 2017 - Peter Mead by Peter Mead
WordPress SEO Mistakes that Kill - BigDigital 2017 - Peter MeadWordPress SEO Mistakes that Kill - BigDigital 2017 - Peter Mead
WordPress SEO Mistakes that Kill - BigDigital 2017 - Peter Mead
Peter Mead3.4K views
URL Funnel Optimisation: How to get budget for SEO - Michal Magdziarz, CEO, D... by DeepCrawl
URL Funnel Optimisation: How to get budget for SEO - Michal Magdziarz, CEO, D...URL Funnel Optimisation: How to get budget for SEO - Michal Magdziarz, CEO, D...
URL Funnel Optimisation: How to get budget for SEO - Michal Magdziarz, CEO, D...
DeepCrawl787 views
Conflicting Website Signals & Confused Search Engines - Rachel Costello, Tech... by Rachel Costello
Conflicting Website Signals & Confused Search Engines - Rachel Costello, Tech...Conflicting Website Signals & Confused Search Engines - Rachel Costello, Tech...
Conflicting Website Signals & Confused Search Engines - Rachel Costello, Tech...
Rachel Costello8.5K views
Joomla 1.6 Core SEO and Best Practices by FinishJoomla
Joomla 1.6 Core SEO and Best PracticesJoomla 1.6 Core SEO and Best Practices
Joomla 1.6 Core SEO and Best Practices
FinishJoomla2.3K views
How to Kick Ass at Internal Linking by Martin Hayman
How to Kick Ass at Internal Linking How to Kick Ass at Internal Linking
How to Kick Ass at Internal Linking
Martin Hayman1.2K views
Mauro Cattaneo - Why hreflang is crucial to international SEO success - Brigh... by Mauro Cattaneo
Mauro Cattaneo - Why hreflang is crucial to international SEO success - Brigh...Mauro Cattaneo - Why hreflang is crucial to international SEO success - Brigh...
Mauro Cattaneo - Why hreflang is crucial to international SEO success - Brigh...
Mauro Cattaneo8.8K views
Analytics Horror Stories - BrightonSEO 2016 by Anna Lewis
Analytics Horror Stories - BrightonSEO 2016Analytics Horror Stories - BrightonSEO 2016
Analytics Horror Stories - BrightonSEO 2016
Anna Lewis6.9K views
SearchLove Boston 2018 - Tom Anthony - Hacking Google: what you can learn fro... by Distilled
SearchLove Boston 2018 - Tom Anthony - Hacking Google: what you can learn fro...SearchLove Boston 2018 - Tom Anthony - Hacking Google: what you can learn fro...
SearchLove Boston 2018 - Tom Anthony - Hacking Google: what you can learn fro...
Distilled2.4K views
Keeping Things Lean & Mean: Crawl Optimisation - Search Marketing Summit AU by Jason Mun
Keeping Things Lean & Mean: Crawl Optimisation - Search Marketing Summit AUKeeping Things Lean & Mean: Crawl Optimisation - Search Marketing Summit AU
Keeping Things Lean & Mean: Crawl Optimisation - Search Marketing Summit AU
Jason Mun3.1K views
Kostas Voudouris - BrightonSEO - Perfromance-based optimisation using Google ... by kvonweb
Kostas Voudouris - BrightonSEO - Perfromance-based optimisation using Google ...Kostas Voudouris - BrightonSEO - Perfromance-based optimisation using Google ...
Kostas Voudouris - BrightonSEO - Perfromance-based optimisation using Google ...
kvonweb18.8K views
Mark Osborne - Brighton SEO April 2019 The Seedy Underbelly of Keyword Resear... by Mark Osborne
Mark Osborne - Brighton SEO April 2019 The Seedy Underbelly of Keyword Resear...Mark Osborne - Brighton SEO April 2019 The Seedy Underbelly of Keyword Resear...
Mark Osborne - Brighton SEO April 2019 The Seedy Underbelly of Keyword Resear...
Mark Osborne9.6K views

Similar to Combatting Crawl Bloat & Pruning Your Content Effectively

Technical SEO: Crawl Space Management - SEOZone Istanbul 2014 by
Technical SEO: Crawl Space Management - SEOZone Istanbul 2014Technical SEO: Crawl Space Management - SEOZone Istanbul 2014
Technical SEO: Crawl Space Management - SEOZone Istanbul 2014Bastian Grimm
10.1K views56 slides
SEO for Beginners Feb 2020 - Bristol Media by
SEO for Beginners Feb 2020  - Bristol MediaSEO for Beginners Feb 2020  - Bristol Media
SEO for Beginners Feb 2020 - Bristol MediaJon Payne
114 views103 slides
How to create seo report by seoruchi.com by
How to create  seo report by seoruchi.comHow to create  seo report by seoruchi.com
How to create seo report by seoruchi.comphp2ranjan
313 views11 slides
Website Audit [On Page and Off Page] by Carl Benedic Pantaleon by
Website Audit [On Page and Off Page] by Carl Benedic PantaleonWebsite Audit [On Page and Off Page] by Carl Benedic Pantaleon
Website Audit [On Page and Off Page] by Carl Benedic PantaleonJacque Doring
536 views117 slides
10 steps to effective seo & great ranking's by
10 steps to effective seo & great ranking's10 steps to effective seo & great ranking's
10 steps to effective seo & great ranking'sT Byh
1.8K views102 slides
The New Age of Web & SEO by
The New Age of Web & SEOThe New Age of Web & SEO
The New Age of Web & SEOVin Lim
552 views44 slides

Similar to Combatting Crawl Bloat & Pruning Your Content Effectively(20)

Technical SEO: Crawl Space Management - SEOZone Istanbul 2014 by Bastian Grimm
Technical SEO: Crawl Space Management - SEOZone Istanbul 2014Technical SEO: Crawl Space Management - SEOZone Istanbul 2014
Technical SEO: Crawl Space Management - SEOZone Istanbul 2014
Bastian Grimm10.1K views
SEO for Beginners Feb 2020 - Bristol Media by Jon Payne
SEO for Beginners Feb 2020  - Bristol MediaSEO for Beginners Feb 2020  - Bristol Media
SEO for Beginners Feb 2020 - Bristol Media
Jon Payne114 views
How to create seo report by seoruchi.com by php2ranjan
How to create  seo report by seoruchi.comHow to create  seo report by seoruchi.com
How to create seo report by seoruchi.com
php2ranjan313 views
Website Audit [On Page and Off Page] by Carl Benedic Pantaleon by Jacque Doring
Website Audit [On Page and Off Page] by Carl Benedic PantaleonWebsite Audit [On Page and Off Page] by Carl Benedic Pantaleon
Website Audit [On Page and Off Page] by Carl Benedic Pantaleon
Jacque Doring536 views
10 steps to effective seo & great ranking's by T Byh
10 steps to effective seo & great ranking's10 steps to effective seo & great ranking's
10 steps to effective seo & great ranking's
T Byh1.8K views
The New Age of Web & SEO by Vin Lim
The New Age of Web & SEOThe New Age of Web & SEO
The New Age of Web & SEO
Vin Lim552 views
Technical SEO: How to Perform an SEO Audit (Step by Step Guide) by Ryan Stewart
Technical SEO: How to Perform an SEO Audit (Step by Step Guide)Technical SEO: How to Perform an SEO Audit (Step by Step Guide)
Technical SEO: How to Perform an SEO Audit (Step by Step Guide)
Ryan Stewart12.6K views
Seo Onpage Optimization Training by SEOBANK
Seo Onpage Optimization TrainingSeo Onpage Optimization Training
Seo Onpage Optimization Training
SEOBANK465 views
SEO Surgery APAC #2 by Nik Ranger by Anton Shulke
SEO Surgery APAC #2 by Nik RangerSEO Surgery APAC #2 by Nik Ranger
SEO Surgery APAC #2 by Nik Ranger
Anton Shulke857 views
SEO Training Course Online, Learn SEO, SEO for Beginners, Complete SEO Tutorial by Deep Mehta
SEO Training Course Online, Learn SEO, SEO for Beginners, Complete SEO TutorialSEO Training Course Online, Learn SEO, SEO for Beginners, Complete SEO Tutorial
SEO Training Course Online, Learn SEO, SEO for Beginners, Complete SEO Tutorial
Deep Mehta3.5K views
Crawl Budget Optimization - Technical SEO Meetup 1 by From The Future
Crawl Budget Optimization - Technical SEO Meetup 1Crawl Budget Optimization - Technical SEO Meetup 1
Crawl Budget Optimization - Technical SEO Meetup 1
From The Future170 views
Why You Should Invest in Technical SEO by Ruth Burr Reedy by Glen Dimaandal
Why You Should Invest in Technical SEO by Ruth Burr ReedyWhy You Should Invest in Technical SEO by Ruth Burr Reedy
Why You Should Invest in Technical SEO by Ruth Burr Reedy
Glen Dimaandal1.7K views
Demand quest SEO training Session 1 May 2017 by Nate Plaunt
Demand quest SEO training Session 1 May 2017Demand quest SEO training Session 1 May 2017
Demand quest SEO training Session 1 May 2017
Nate Plaunt372 views
Demand quest seo training by Nate Plaunt
Demand quest seo trainingDemand quest seo training
Demand quest seo training
Nate Plaunt361 views
How To Optimize Your Site's Crawl Budget - Technical SEO Philly by Nick Eubanks
How To Optimize Your Site's Crawl Budget - Technical SEO PhillyHow To Optimize Your Site's Crawl Budget - Technical SEO Philly
How To Optimize Your Site's Crawl Budget - Technical SEO Philly
Nick Eubanks520 views
Crawl Budget Optimisation at #dmwf2018 by Nitin Manchanda
Crawl Budget Optimisation at #dmwf2018Crawl Budget Optimisation at #dmwf2018
Crawl Budget Optimisation at #dmwf2018
Nitin Manchanda333 views
SearchLove Boston 2018 - Emily Grossman - The Marketer’s Guide to Performance... by Distilled
SearchLove Boston 2018 - Emily Grossman - The Marketer’s Guide to Performance...SearchLove Boston 2018 - Emily Grossman - The Marketer’s Guide to Performance...
SearchLove Boston 2018 - Emily Grossman - The Marketer’s Guide to Performance...
Distilled2.6K views

Recently uploaded

First 30 days of Your CRO Program by
First 30 days of Your CRO ProgramFirst 30 days of Your CRO Program
First 30 days of Your CRO ProgramVWO
47 views57 slides
Aurora Vietnam_Credential_2023_Eng_Update2811 by
Aurora Vietnam_Credential_2023_Eng_Update2811Aurora Vietnam_Credential_2023_Eng_Update2811
Aurora Vietnam_Credential_2023_Eng_Update2811Aurora Vietnam IMC Agency
24 views131 slides

Recently uploaded(20)

First 30 days of Your CRO Program by VWO
First 30 days of Your CRO ProgramFirst 30 days of Your CRO Program
First 30 days of Your CRO Program
VWO47 views
Bridging the Gap: How SEO and CRO Work Together to Maximize User Satisfaction... by Rio Ichikawa
Bridging the Gap: How SEO and CRO Work Together to Maximize User Satisfaction...Bridging the Gap: How SEO and CRO Work Together to Maximize User Satisfaction...
Bridging the Gap: How SEO and CRO Work Together to Maximize User Satisfaction...
Rio Ichikawa117 views
Turning Business Challenges into Testable Ideas - 29 Nov '23.pdf by VWO
Turning Business Challenges into Testable Ideas - 29 Nov '23.pdfTurning Business Challenges into Testable Ideas - 29 Nov '23.pdf
Turning Business Challenges into Testable Ideas - 29 Nov '23.pdf
VWO34 views

Combatting Crawl Bloat & Pruning Your Content Effectively