Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Charlie Whitworth
SEO Director
@WhitworthSEO
http://www.slideshare.net/whitworthseo
Banc
what will we cover?
bancmedia.com
Crawl Bloat & Pruning
@WhitworthSEO
beat the bloat
Crawl Bloat
Making search engines work too hard
and not taking full advantage of your
crawl budget
@Whitwor...
crawl definitions
=
Crawl Budget
bancmedia.com@WhitworthSEO
Crawl Rate & Crawl Demand
crawl rate (limit) : maximum fetching rate
Search Console Limit
s
bancmedia.com@WhitworthSEO
Website/Server Health
crawl demand
Popularity (Traffic) Staleness
@WhitworthSEO bancmedia.com
How much does Google want to see your URLs?
Crawl budget – determined by rate & demand
What Google can & wants to crawl
@WhitworthSEO bancmedia.com
• Empty pages
• No...
crawl budget
“Prioritizing what to crawl, when, and how much resource the
server hosting the site can allocate to crawling...
how to identify crawl bloat
bancmedia.com@WhitworthSEO
how to identify crawl bloat
bancmedia.com@WhitworthSEO
why is this a problem?
bancmedia.com@WhitworthSEO
Don’t work them too hard
Search engine bots hate crawling rubbish
why is this a problem?
Engagement metrics are key
especially now we’re “mobile first”
@WhitworthSEO bancmedia.com
examples of rubbish: search result URLs
/search/
bancmedia.com@WhitworthSEO
/sessionkey=
/search-results?q=
examples of rubbish: paginated URLs
bancmedia.com
rel=next/prevnoindex
@WhitworthSEO
examples of rubbish: faceted navigations
json-ld
disallow parameters
bancmedia.com@WhitworthSEO
noindex
who’s seen this?
cut the crap
Get Rid Of It
@WhitworthSEO bancmedia.com
plan
noindex
<meta name=“robots" content=“noindex">
Won’t deal with crawl bloat in the short term
Still an effective tactic
ban...
url parameters
@WhitworthSEO
Powerful tool when used properly
No need for developer support
bancmedia.com
xml sitemaps
Redirected URLs
Noindexed URLs
Poor quality pages
@WhitworthSEO bancmedia.com
robots.txt
bancmedia.com@WhitworthSEO
Exercise Caution!
Powerful crawl bloat tool when used effectively
robots.txt
Blocking Crawl Paths – Stems Authority Flow
bancmedia.com@WhitworthSEO
Google needs to see tags to honour them!
google’s hint
Google gives you all you need to
monitor your results
All these features are your friend
s
@WhitworthSEO ban...
crawl stats
Great way to gauge your efforts
@WhitworthSEO bancmedia.com
bancmedia.com@WhitworthSEO
Pruning
the prune
pruning – the fun bit
SEO Housekeepin
g
Now you have your crawl under control,
show off your best bits
bancmedia.com@Whitw...
pruning – why?
• Spinning
• Excessive blogging
• Sparse articles
@WhitworthSEO bancmedia.com
• Over optimisation
• Saturat...
stage one – content audit/content is king
Content auditing still essential nonetheless
bancmedia.com@WhitworthSEO
SEO’s mo...
stage two: the prune
410 – Gone
@WhitworthSEO bancmedia.com
For pages you want gone forever
stage two: the prune
bancmedia.com@WhitworthSEO
Won’t help crawl bloat short term
Will disappear eventually*
stage two: the prune
Crawl Bloat
Authority
Indexation
@WhitworthSEO bancmedia.com
stage two: the prune
NoIndex
Will be crawled less over time ensures a high quality SER
P
For duplicate content, cannibalis...
stage two: canonical
For duplicate content, cannibalisation and API style pages
@WhitworthSEO bancmedia.com
canonical
stage three: content repurposing/enhancement
@WhitworthSEO bancmedia.com
Evergreen
Seasonal
Long form
Research
search quality guidelines
@WhitworthSEO bancmedia.com
Search Quality Evaluator Guidelines
stage four: the finishing touch
Link Profile Analysis
Links pointing at excluded
pages
Usual disavow request
@WhitworthSEO...
crawl bloat and pruning checklist
Does your site have crawl bloat?
Choose the appropriate fixes for your site
Prune your r...
final tips: thanks for coming
@WhitworthSEO bancmedia.com
Thanks for Listening!
2 Tips
Upcoming SlideShare
Loading in …5
×

of

Combatting Crawl Bloat & Pruning Your Content Effectively Slide 1 Combatting Crawl Bloat & Pruning Your Content Effectively Slide 2 Combatting Crawl Bloat & Pruning Your Content Effectively Slide 3 Combatting Crawl Bloat & Pruning Your Content Effectively Slide 4 Combatting Crawl Bloat & Pruning Your Content Effectively Slide 5 Combatting Crawl Bloat & Pruning Your Content Effectively Slide 6 Combatting Crawl Bloat & Pruning Your Content Effectively Slide 7 Combatting Crawl Bloat & Pruning Your Content Effectively Slide 8 Combatting Crawl Bloat & Pruning Your Content Effectively Slide 9 Combatting Crawl Bloat & Pruning Your Content Effectively Slide 10 Combatting Crawl Bloat & Pruning Your Content Effectively Slide 11 Combatting Crawl Bloat & Pruning Your Content Effectively Slide 12 Combatting Crawl Bloat & Pruning Your Content Effectively Slide 13 Combatting Crawl Bloat & Pruning Your Content Effectively Slide 14 Combatting Crawl Bloat & Pruning Your Content Effectively Slide 15 Combatting Crawl Bloat & Pruning Your Content Effectively Slide 16 Combatting Crawl Bloat & Pruning Your Content Effectively Slide 17 Combatting Crawl Bloat & Pruning Your Content Effectively Slide 18 Combatting Crawl Bloat & Pruning Your Content Effectively Slide 19 Combatting Crawl Bloat & Pruning Your Content Effectively Slide 20 Combatting Crawl Bloat & Pruning Your Content Effectively Slide 21 Combatting Crawl Bloat & Pruning Your Content Effectively Slide 22 Combatting Crawl Bloat & Pruning Your Content Effectively Slide 23 Combatting Crawl Bloat & Pruning Your Content Effectively Slide 24 Combatting Crawl Bloat & Pruning Your Content Effectively Slide 25 Combatting Crawl Bloat & Pruning Your Content Effectively Slide 26 Combatting Crawl Bloat & Pruning Your Content Effectively Slide 27 Combatting Crawl Bloat & Pruning Your Content Effectively Slide 28 Combatting Crawl Bloat & Pruning Your Content Effectively Slide 29 Combatting Crawl Bloat & Pruning Your Content Effectively Slide 30 Combatting Crawl Bloat & Pruning Your Content Effectively Slide 31 Combatting Crawl Bloat & Pruning Your Content Effectively Slide 32 Combatting Crawl Bloat & Pruning Your Content Effectively Slide 33 Combatting Crawl Bloat & Pruning Your Content Effectively Slide 34 Combatting Crawl Bloat & Pruning Your Content Effectively Slide 35 Combatting Crawl Bloat & Pruning Your Content Effectively Slide 36 Combatting Crawl Bloat & Pruning Your Content Effectively Slide 37 Combatting Crawl Bloat & Pruning Your Content Effectively Slide 38 Combatting Crawl Bloat & Pruning Your Content Effectively Slide 39
Upcoming SlideShare
What to Upload to SlideShare
Next
Download to read offline and view in fullscreen.

4 Likes

Share

Download to read offline

Combatting Crawl Bloat & Pruning Your Content Effectively

Download to read offline

My #BrightonSEO talk on the best way to refine your crawl bloat and prune your remaining content effectively. Follow me at @WhitworthSEO for more tech seo nonsense.

Related Books

Free with a 30 day trial from Scribd

See all

Combatting Crawl Bloat & Pruning Your Content Effectively

  1. 1. Charlie Whitworth SEO Director @WhitworthSEO http://www.slideshare.net/whitworthseo Banc
  2. 2. what will we cover? bancmedia.com Crawl Bloat & Pruning @WhitworthSEO
  3. 3. beat the bloat Crawl Bloat Making search engines work too hard and not taking full advantage of your crawl budget @WhitworthSEO bancmedia.com
  4. 4. crawl definitions = Crawl Budget bancmedia.com@WhitworthSEO Crawl Rate & Crawl Demand
  5. 5. crawl rate (limit) : maximum fetching rate Search Console Limit s bancmedia.com@WhitworthSEO Website/Server Health
  6. 6. crawl demand Popularity (Traffic) Staleness @WhitworthSEO bancmedia.com How much does Google want to see your URLs?
  7. 7. Crawl budget – determined by rate & demand What Google can & wants to crawl @WhitworthSEO bancmedia.com • Empty pages • No valued added • Dupe • Soft 404 • SPAM • Hacked content
  8. 8. crawl budget “Prioritizing what to crawl, when, and how much resource the server hosting the site can allocate to crawling is more important for bigger sites, or those that auto-generate pages based on URL parameters” Gary Ilyes bancmedia.com@WhitworthSEO
  9. 9. how to identify crawl bloat bancmedia.com@WhitworthSEO
  10. 10. how to identify crawl bloat bancmedia.com@WhitworthSEO
  11. 11. why is this a problem? bancmedia.com@WhitworthSEO Don’t work them too hard Search engine bots hate crawling rubbish
  12. 12. why is this a problem? Engagement metrics are key especially now we’re “mobile first” @WhitworthSEO bancmedia.com
  13. 13. examples of rubbish: search result URLs /search/ bancmedia.com@WhitworthSEO /sessionkey= /search-results?q=
  14. 14. examples of rubbish: paginated URLs bancmedia.com rel=next/prevnoindex @WhitworthSEO
  15. 15. examples of rubbish: faceted navigations json-ld disallow parameters bancmedia.com@WhitworthSEO noindex
  16. 16. who’s seen this?
  17. 17. cut the crap Get Rid Of It @WhitworthSEO bancmedia.com
  18. 18. plan
  19. 19. noindex <meta name=“robots" content=“noindex"> Won’t deal with crawl bloat in the short term Still an effective tactic bancmedia.com@WhitworthSEO
  20. 20. url parameters @WhitworthSEO Powerful tool when used properly No need for developer support bancmedia.com
  21. 21. xml sitemaps Redirected URLs Noindexed URLs Poor quality pages @WhitworthSEO bancmedia.com
  22. 22. robots.txt bancmedia.com@WhitworthSEO Exercise Caution! Powerful crawl bloat tool when used effectively
  23. 23. robots.txt Blocking Crawl Paths – Stems Authority Flow bancmedia.com@WhitworthSEO Google needs to see tags to honour them!
  24. 24. google’s hint Google gives you all you need to monitor your results All these features are your friend s @WhitworthSEO bancmedia.com
  25. 25. crawl stats Great way to gauge your efforts @WhitworthSEO bancmedia.com
  26. 26. bancmedia.com@WhitworthSEO Pruning the prune
  27. 27. pruning – the fun bit SEO Housekeepin g Now you have your crawl under control, show off your best bits bancmedia.com@WhitworthSEO
  28. 28. pruning – why? • Spinning • Excessive blogging • Sparse articles @WhitworthSEO bancmedia.com • Over optimisation • Saturation
  29. 29. stage one – content audit/content is king Content auditing still essential nonetheless bancmedia.com@WhitworthSEO SEO’s most annoying cliché
  30. 30. stage two: the prune 410 – Gone @WhitworthSEO bancmedia.com For pages you want gone forever
  31. 31. stage two: the prune bancmedia.com@WhitworthSEO Won’t help crawl bloat short term Will disappear eventually*
  32. 32. stage two: the prune Crawl Bloat Authority Indexation @WhitworthSEO bancmedia.com
  33. 33. stage two: the prune NoIndex Will be crawled less over time ensures a high quality SER P For duplicate content, cannibalisation and API style pages @WhitworthSEO bancmedia.com
  34. 34. stage two: canonical For duplicate content, cannibalisation and API style pages @WhitworthSEO bancmedia.com canonical
  35. 35. stage three: content repurposing/enhancement @WhitworthSEO bancmedia.com Evergreen Seasonal Long form Research
  36. 36. search quality guidelines @WhitworthSEO bancmedia.com Search Quality Evaluator Guidelines
  37. 37. stage four: the finishing touch Link Profile Analysis Links pointing at excluded pages Usual disavow request @WhitworthSEO bancmedia.com Don’t ruin all your hard work with a crap link profile
  38. 38. crawl bloat and pruning checklist Does your site have crawl bloat? Choose the appropriate fixes for your site Prune your remaining content Continually improve and enhance @WhitworthSEO bancmedia.com
  39. 39. final tips: thanks for coming @WhitworthSEO bancmedia.com Thanks for Listening! 2 Tips
  • ChrisDrury6

    Apr. 29, 2018
  • vik21000

    Jun. 13, 2017
  • carralon

    Apr. 10, 2017
  • AnitaValentinova

    Apr. 10, 2017

My #BrightonSEO talk on the best way to refine your crawl bloat and prune your remaining content effectively. Follow me at @WhitworthSEO for more tech seo nonsense.

Views

Total views

3,851

On Slideshare

0

From embeds

0

Number of embeds

463

Actions

Downloads

26

Shares

0

Comments

0

Likes

4

×