The amount of pages or URLs that a search engine bot, sometimes referred to as a crawler or spider, is willing to crawl on a website over a given time frame, usually within a day, is referred to as the crawl budget. Search engines allot a limited amount of this resource to every website, taking into account a number of variables such as the website's relevancy, authority, and past crawl patterns.