How to-crawl-your-way-into-ranking-with-indexable-content
•
1 like•474 views
To get content indexed and ranked, search engines prioritize crawling important and frequently changing pages based on links and queries. Crawling is constrained by computational resources, so search engines use techniques like importance scoring, change detection, and URL patterns to efficiently discover new and updated content while being polite to websites. Maintaining consistent internal linking, sitemaps, and canonicalization helps search engines confidently discover and represent website content.
Recommended
More Related Content
Similar to How to-crawl-your-way-into-ranking-with-indexable-content
Similar to How to-crawl-your-way-into-ranking-with-indexable-content (20)
More from Anton Shulke
More from Anton Shulke (20)
Recently uploaded
Recently uploaded (20)
How to-crawl-your-way-into-ranking-with-indexable-content
- 3. HOW TO CRAWL YOUR WAY INTO RANKING WITH INDEXABLE CONTENT
- 5. BUT HERE’S A STATEMENT OF THE OBVIOUS
- 7. BUT NOT GETTING INDEXED DOES GUARANTEE NOT RANKING
- 9. BUT… THERE’S A LOT TO CRAWL OUT THERE
- 10. AND SEARCH ENGINE RESOURCES ARE NOT FINITE
- 11. There are considerations FINANCIAL EXPENSE ENVIRONMENTAL EXPENSE COMPUTATIONAL EXPENSE
- 12. EFFICIENCY VERSUS EFFECTIVENESS CONSIDERATIONS IMPACT THE WHOLE SEARCH PIPELINE
- 14. Multi Stage Retrieval & Ranking Fetch initial set of x documents above x quality / relevance threshold Refine for fine tuning a top-k set of results from initial fetch
- 16. WHY STOCK YOUR SHELVES WITH THINGS NO-ONE WANTS?
- 19. NOT ONLY IS CONTENT CHANGING BUT QUERY INTENT & DEMAND IS TOO
- 21. SO IS THAT URL WORTH IT?
- 23. CRAWL SCHEDULING
- 26. TIERS
- 27. Crawl Scheduling / Process The Frontier (Discovery) URL extraction Render queueing Indexing Refresh
- 29. NOT EVERYONE CAN GET THROUGH A DOOR AT THE SAME TIME
- 30. PRIORITIES
- 33. Priorities Not all content is equally important Not all content changes frequently Not all change is substantively important Resources are not finite (although extensive) The amount of content available continues to grow
- 34. Two primary ‘notions’ Demand Safe crawling
- 35. SEARCH ENGINES ‘PROBABLY’ STOCK THEIR SHELVES WITH CONTENT PEOPLE WANT
- 36. CRAWLING POLITENESS – NEVER DAMAGE THE HOST
- 37. CRAWL BUDGET
- 38. IT'S A MADE- UP PHRASE NOT USED BY ENGINEERS
- 39. Zipfian Distributions A few URLs will be very, very popular Be Most URLs will be rarely visited Be A few URLs will be linked to hugely Be Most URLs will have few or no links Have
- 40. IMPORTANCE & POWER LAW DISTRIBUTION
- 42. GOING WIDE RATHER THAN DEEP
- 43. DEEPER PAGES MAY BE LESS IMPORTANT
- 44. Why bother crawling down a path? Is it safe to do so (for the site)? Is there a good ‘yield’ of informational value to be gained?
- 45. REDUNDANCY IN RESULTS ARE A POOR EXPERIENCE
- 47. DUST DETECTION
- 50. CRAWLING IS ALSO DRIVEN BY CHANGE FREQUENCY
- 51. ‘Probably’ Supervised Learning Predicting value from patterns / paths / site sections Predicting change frequency Predicting substantive change frequency
- 54. TO INDEX SEARCH ENGINES MUST BE ‘AWARE’
- 55. DISCOVERY
- 56. ‘THE FRONTIER’
- 57. MOSTLY BY FOLLOWING LINKS OR URL ‘PATTERNS’
- 59. Making content findable Well connected XML sitemaps Informative intent & query driven internal anchors Strong ‘relatedness’ clustering
- 60. INDICATIONS IN URL / FILE NAMES
- 61. FEEDS & APIS (E.G. BING API)
- 62. MAKING SURE CONTENT IS CRAWLABLE & RENDERABLE
- 65. Search Engines May Still Index Unrenderable Content E.g. pages with many links might still get indexed Pages blocked at robots might still get indexed
- 68. Improving Consistency Cleaning up legacy cruft Consistent internal linking Consistent canonical tags Inclusion in XML sitemaps Growing confidence in canonicals amongst cluster candidates
- 74. OVER TO YOU JASON & FABRICE