Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
#OnCrawlBreakfast
(and make Technical SEO great again)
@FrancoisGoube, CEO @Oncrawl
How to optimize your crawl budget?
Who am I?
Francois Goube
Founder @OnCRAWL
15 years SEO experience, Serial
Entrepreneur. French Majestic
Ambassador
Semanti...
Make your inner SEO super hero grow up
Super-powers = Knowledge + Tools
What Google says about « Crawl Budget »
If new pages tend to be crawled the same day they're
published, crawl budget is no...
Your Website What Google really knows!
This is what a log file looks like
What your log files look like in OnCrawl
• All bots data
• Status codes
• Crawl frequency
• List of URL fetched by bots
• ...
• 100% of your GSC Properties show exploration
statistics
• With log analysis you can detect errors in Bots
behaviour
• Ba...
Google Crawl Budget
“Taking crawl rate and crawl demand together we define crawl budget
as the number of URLs Googlebot ca...
Understanding Google’s crawl bud
Understanding Google’s crawl bud
Understanding Google’s crawl bud
Google patents about Crawl rate & Demand
• US 8666964 B1 : Managing items in crawl schedule
• US 8707312 B1 : Document reu...
Page Importance
« Page Importance » is not the PageRank
• Where is the page in my architecture? – Depth influences Crawl r...
What are the Factors influencing
Google’s Crawl Budget?
All websites
Are not born equal
Which Ranking Factor affects Crawl Ra
Which Ranking Factor affects Crawl Ra
Which Ranking Factor affects Crawl Ra
Which Ranking Factor affects Crawl Ra
Which Ranking Factor affects Crawl Ra
Crawler
3 000 000
Google
7 000 000
All pages available from
your linking structure
All pages known by
Google
Take care of ...
Orphan pages
Are pages that are not
linked from your internal
linking structure,
but that Google knows
Take care of your O...
Orphan pages generic cases
• Ecommerce:
• Out of stocks products
• No longer available products
• Revamping of menus, pagi...
How to deal with your Orphan pages?
Is it Normal?
Redirect
301
Noindex via
Robots.txt
Yes
No
Do they receive
Organic Traff...
What ROI can I espect?
Less pages crawled
(Unuseful
& Inactive)
More useful
pages &
active pages
Better indexation
Better ...
Thank you !
@OnCrawl – Booth 29
Try to win a 1-year Pro Subscription!
How to optimize Google's crawl budget? - BrightonSEO 2017
Upcoming SlideShare
Loading in …5
×

How to optimize Google's crawl budget? - BrightonSEO 2017

2,644 views

Published on

Francois Goube, CEO and Founder of Oncrawl shares his thoughts about how to optimize Google's Crawl Budget.
Insights about what is Crawl Budget
What factors may influence Crawl Budget?
How to deal with orphan pages?

Published in: Business
  • Hello! High Quality And Affordable Essays For You. Starting at $4.99 per page - Check our website! https://vk.cc/82gJD2
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

How to optimize Google's crawl budget? - BrightonSEO 2017

  1. 1. #OnCrawlBreakfast (and make Technical SEO great again) @FrancoisGoube, CEO @Oncrawl How to optimize your crawl budget?
  2. 2. Who am I? Francois Goube Founder @OnCRAWL 15 years SEO experience, Serial Entrepreneur. French Majestic Ambassador Semantic Nerd Data addict & SEO maniac
  3. 3. Make your inner SEO super hero grow up
  4. 4. Super-powers = Knowledge + Tools
  5. 5. What Google says about « Crawl Budget » If new pages tend to be crawled the same day they're published, crawl budget is not something webmasters need to focus on. […] if a site has fewer than a few thousand URLs, most of the time it will be crawled efficiently. […] we don't have a single term that would describe everything that "crawl budget" stands for externally. https://webmasters.googleblog.com/2017/01/what-crawl-budget-means-for-googlebot.html Understanding Google’s crawl bud
  6. 6. Your Website What Google really knows!
  7. 7. This is what a log file looks like
  8. 8. What your log files look like in OnCrawl • All bots data • Status codes • Crawl frequency • List of URL fetched by bots • All referring traffic data • Active pages • Freshrank • … You know what Google did!
  9. 9. • 100% of your GSC Properties show exploration statistics • With log analysis you can detect errors in Bots behaviour • Bad internal linking structure, Pagination, Facetting, Orphan pages or spider trap can affect Google’s ability to explore your website properly. For all our customers, an optimisation of Crawl Budget leads to better rankings Every webmaster should keep an eye on his Crawl Budget Understanding Google’s crawl bud
  10. 10. Google Crawl Budget “Taking crawl rate and crawl demand together we define crawl budget as the number of URLs Googlebot can and wants to crawl.”  So everyday Google crawls a determined number of pages  As an SEO you need to help Google crawl your Money Pages Understanding Google’s crawl bud
  11. 11. Understanding Google’s crawl bud
  12. 12. Understanding Google’s crawl bud
  13. 13. Understanding Google’s crawl bud
  14. 14. Google patents about Crawl rate & Demand • US 8666964 B1 : Managing items in crawl schedule • US 8707312 B1 : Document reuse in a search engine crawler • US 8037054 B2 : Web crawler scheduler that utilizes sitemaps from websites • US 7305610 B1 : Distributed crawling of hyperlinked documents • US 8407204 B2 : Minimizing visibility of stale content in web searching including revisine web crawl intervals of documents • US 8386459 B1 : Scheduling a recrawl • US 8042112 B1 : Scheduler for search engine crawler Crawl Scheduling is the big thing! Understanding Google’s crawl bud
  15. 15. Page Importance « Page Importance » is not the PageRank • Where is the page in my architecture? – Depth influences Crawl ratio • Page Rank: TF/CF of a page - Majestic • Internal Page Rank – InRank OnCrawl • Type of document: PDF, HTML, TXT • Inclusion in sitemap.xml • Quality of anchors • Quality of content: Nb of Words, Near duplicates • … Combine all these pieces of Data with your logs! Understanding Google’s crawl bud
  16. 16. What are the Factors influencing Google’s Crawl Budget?
  17. 17. All websites Are not born equal
  18. 18. Which Ranking Factor affects Crawl Ra
  19. 19. Which Ranking Factor affects Crawl Ra
  20. 20. Which Ranking Factor affects Crawl Ra
  21. 21. Which Ranking Factor affects Crawl Ra
  22. 22. Which Ranking Factor affects Crawl Ra
  23. 23. Crawler 3 000 000 Google 7 000 000 All pages available from your linking structure All pages known by Google Take care of your Orphan Pag
  24. 24. Orphan pages Are pages that are not linked from your internal linking structure, but that Google knows Take care of your Orphan Pag
  25. 25. Orphan pages generic cases • Ecommerce: • Out of stocks products • No longer available products • Revamping of menus, pagination, facetting,… • Media • Bad internal linking structure • Archives only available through Sitemap.xml Main problems: • those pages don’t receive any linkjuice –> chances are they can’t rank! • You are wasting Google Crawl budget Take care of your Orphan Pag
  26. 26. How to deal with your Orphan pages? Is it Normal? Redirect 301 Noindex via Robots.txt Yes No Do they receive Organic Traffic? Yes Is the page Valuable for my current Business? No No Can’t answer questions? Yes Ask an expert! Add link from structure Take care of your Orphan Pag
  27. 27. What ROI can I espect? Less pages crawled (Unuseful & Inactive) More useful pages & active pages Better indexation Better Internal Popularity Boost Your Organic traffic Grow your Organic Traffic with less Pages
  28. 28. Thank you ! @OnCrawl – Booth 29 Try to win a 1-year Pro Subscription!

×