Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

How to Find & Fix Crawl Optimisation Issues - #BrightonSEO

13,873 views

Published on

Slides from my talk at BrightonSEO in April 2016 about finding and fixing crawl optimisation issues on your website for improved SEO performance.

Published in: Internet

How to Find & Fix Crawl Optimisation Issues - #BrightonSEO

  1. 1. @badams Crawl Optimisation Barry Adams
  2. 2. @badams Barry Adams • Doing SEO since 1998 • Founder of Polemic Digital • Senior Editor at State of Digital
  3. 3. @badams Crawl Optimisation
  4. 4. @badams What is Crawl Optimisation? Ensuring search engine spiders waste as little time as possible crawling the right URLs on your site.
  5. 5. @badams Why is Crawl Optimisation important? If you waste crawl budget, the right pages are unlikely to be crawled & indexed.
  6. 6. @badams Crawl Budget
  7. 7. @badams Google’s Crawl Sources • Site crawl • XML Sitemaps • Inbound links • DNS records • Domain registrations • Browsing data
  8. 8. @badams Identifying Crawl Waste
  9. 9. @badams Identifying Crawl Waste
  10. 10. @badams Identifying Crawl Waste
  11. 11. @badams Crawl Waste • Bogus URLs in XML Sitemap
  12. 12. @badams Optimise XML Sitemaps • Ensure your sitemap contains final URLs only • Minimise 301-redirects or other non-200 status codes • Use multiple sitemaps to identify crawl waste in GSC
  13. 13. @badams Crawl Waste • Paginated Listings • Faceted Navigation http://website.com/jewellery/?page=2&cat=5&color =silver&style=glass&collection=autumnsort=a&…
  14. 14. @badams Optimise Paginated Listings • List more items on a single page • Implement rel=prev/next pagination meta tags • Block sorting parameters in robots.txt  Disallow: /*?sort=*
  15. 15. @badams Optimise Faceted Navigation • Decide which facets have SEO value  Build static pages for these • All other facets:  robots.txt disallow  ‘rel=nofollow’ on facet links
  16. 16. @badams Crawl Waste • Internal Site Search Results
  17. 17. @badams Block Internal Site Search Pages • Block in robots.txt User-agent: * Disallow: /SearchResults.aspx Disallow: /*query=* Disallow: /*s=*
  18. 18. @badams Crawl Waste • Internal redirects
  19. 19. @badams Minimise Internal Redirects • Find redirects with Screaming Frog • Internal links should all be 200 OK • Flat site structure
  20. 20. @badams Crawl Waste • Canonicalised Pages
  21. 21. @badams Use Canonicals Wisely • “rel=canonical” is primarily for index issues It is not a fix for crawl waste Search engines need to see the canonical tag before they can act on it Ergo, pages need to be crawled before rel=canonical has any effect Ditto with meta noindex tags
  22. 22. @badams DON’T use Canonicals for… • Faceted navigation • Pagination & sorting • Site Search pages
  23. 23. @badams OK to use Canonicals for… • Separate mobile URLs • Session-specific URL parameters • Content syndication • Unavoidable content duplication
  24. 24. @badams Crawl Waste • Slow loading pages
  25. 25. @badams Optimise Load Speed • Time to First Byte • Lightweight pages • Caching • Compression
  26. 26. @badams Crawl Optimisation Summarised • Don’t let search engines do the hard work • Tools at your disposal;  DeepCrawl  Google Search Console  Screaming Frog SEO Crawler  WebPageTest.org • Solutions; – XML Sitemaps – robots.txt – rel=nofollow – rel=prev / rel=next – Load speed
  27. 27. @badams The End Goal
  28. 28. @badams Thank You barry@polemicdigital.com www.polemicdigital.com @polemicdigital @badams

×