Crawl Budget
Crawl budget is the number of pages Google is willing and able to crawl on your site within a given timeframe. Google has a finite amount of crawling capacity to distribute across the entire web, and each site gets an allocation based on how valuable and accessible its pages are.
Before going further, it is worth being direct about who this matters for. Google's own Search Central documentation states: "If your site doesn't have a large number of pages that change rapidly, or if your pages seem to be crawled the same day that they are published, you don't need to read this guide." For most small and medium-sized sites with under 10,000 pages, crawl budget is rarely a constraint. Where it becomes genuinely critical is large ecommerce catalogues, news sites, and any site that generates URLs at scale - whether through filtered navigation, URL parameters, or dynamic content.
What determines crawl budget
Google's crawl budget is shaped by two factors working together.
Crawl capacity is how fast Googlebot can crawl your site without overloading your server. If your site returns slow responses or server errors, Googlebot reduces its crawl rate to avoid causing problems. A faster, more stable site gets crawled more efficiently - not because Google rewards speed with a larger budget, but because the server can handle more requests per session.
Crawl demand is how much Google wants to crawl your site. Pages that are more popular - those with more external links pointing to them and more search traffic - are crawled more frequently to keep Google's index fresh. New content on a well-established site tends to get crawled quickly. New content on a low-authority site with few links may wait days or weeks.
What wastes crawl budget
Google's crawling team has stated directly that "having many low-value-add URLs can negatively affect a site's crawling and indexing." The categories that consume budget without contributing value include:
- URL parameters and faceted navigation. Ecommerce filter combinations - colour, size, sort order - can generate thousands of unique URLs for what is essentially the same content. Without canonical tags or parameter handling, Googlebot crawls all of them. See the canonicalisation page for how to address this.
- Redirect chains. Each redirect in a chain costs a crawl request. Long chains - three or more redirects to reach a final URL - waste budget and can cause Googlebot to abandon the crawl. Resolve redirect chains to direct 301s wherever possible.
- Orphan pages. Pages with no internal links pointing to them are harder for Googlebot to find. Without a path to them, they get crawled infrequently or not at all. Adding internal links from relevant pages is the fix - covered in the internal linking page.
- Low-quality and thin pages. Empty category pages, auto-generated tag archives, paginated pages beyond the first few, and soft-404 pages consume budget that would be better spent on pages with actual ranking potential.
- Soft 404s and server errors. Pages that return a 200 status code but display "no results found" or similar messages are treated as crawlable pages. Returning a proper 404 or blocking them in robots.txt is more efficient.
The practical effect on indexing
A Sitebulb case study on a Magento ecommerce site found that after blocking low-value backend URLs in robots.txt, Google Search Console showed fewer "Discovered - Currently Not Indexed" statuses, and time to indexing for new products and collections reduced by approximately 25%. The budget that had been consumed on worthless URLs was redirected to content that mattered.
This is the practical consequence of poor crawl budget management: important pages - new products, updated articles, newly published content - wait in a queue while Googlebot works through low-value URLs that should never have been crawlable in the first place.
How to monitor crawl budget
Google Search Console has a Crawl Stats report under Settings. It shows how many pages Googlebot is requesting per day, the response codes it is receiving, and whether crawl activity is being limited by server capacity. Two warning signs to look for:
- A high proportion of "Discovered - currently not indexed" pages in the Indexing report, which indicates Google has found URLs but is not crawling or processing them.
- Consistent server errors or "host load exceeded" messages in the Crawl Stats report, which indicates your server is the bottleneck.
For most sites, keeping your sitemap clean, addressing canonical issues, fixing redirect chains, and ensuring important pages have strong internal links is sufficient. Crawl budget optimisation at a deeper level - log file analysis, parameter handling rules - is a task for larger sites where indexing delays are already causing measurable problems.
With crawl budget covered, the technical SEO chapter is complete. The next chapter covers link building - how external authority is earned and why it remains one of Google's strongest ranking signals.
Next: Link building overview