Crawling
Before Google can rank a page, it has to know it exists. Crawling is how it finds out. If your page is never crawled, it cannot be indexed, and if it is not indexed, it cannot rank - regardless of how good the content is.
What crawling is
Google runs automated programs called crawlers, or Googlebot, that move across the web by following links from page to page. When Googlebot visits a page, it downloads the content and passes it back to Google's systems for processing. According to Cloudflare's annual Year in Review report, Googlebot accounted for more than 25% of all verified bot traffic on the web - making it by far the most active crawler operating at scale. For a plain-English tour of what these crawlers are and how to make your site easy for them, see my guide to what a Google crawler is.
Google's index currently contains an estimated 400 billion documents. Keeping that index fresh requires Googlebot to crawl billions of pages every day, re-visiting known pages to detect changes and constantly discovering new ones.
How Google discovers new pages
There is no central registry of every page on the web. Google finds new pages through three main routes:
Links from already-known pages. When Googlebot crawls a page, it collects every link on that page and adds those URLs to its queue. This is the primary discovery mechanism. A page with no links pointing to it - called an orphan page - may never be found.
XML sitemaps. A sitemap is a file you submit to Google Search Console that lists the pages on your site. It does not guarantee crawling, but it signals to Google what exists and where. For large sites or newly published pages with few internal links, a sitemap is an important safety net. You can build a valid one in a minute with my free XML sitemap generator.
Direct URL submission. Using Search Console's URL Inspection tool, you can request that Google crawl a specific page. This is useful after publishing important new content or fixing a previously broken page.
What affects how often Google crawls your site
Google does not crawl all sites at the same frequency. Several factors influence how often Googlebot returns:
Page authority and popularity. Pages with more high-quality backlinks pointing to them tend to be crawled more frequently. Google treats links as a signal that content is worth reading, and that signal also affects crawl priority.
Update frequency. Sites and pages that change regularly are crawled more often. A news site may be crawled multiple times a day. A static landing page that never changes may only be re-crawled every few weeks.
Server health. Google's own Search Console documentation states that if your server error rate increases, Googlebot will automatically throttle back its requests to avoid overloading your server. Slow load times and frequent 5xx errors reduce crawl frequency - which can delay how quickly new or updated content appears in search results.
Site quality signals. Google allocates crawl resources based on its assessment of a site's overall quality. A well-structured site with clean URLs, working links, and strong content tends to earn more crawl attention than one with thin pages and technical problems.
What can block crawling
Three common issues stop Googlebot from reaching pages it should be able to find.
A robots.txt file at your site's root can instruct crawlers not to access certain pages or directories. Blocking the wrong pages - or accidentally blocking your entire site with a single misconfigured rule - is one of the most damaging technical SEO errors. The sitemaps and robots.txt page covers how to use this file safely.
Orphan pages - pages with no internal links pointing to them - may never be discovered unless they are in your sitemap. Even if Googlebot visits your site regularly, it can only follow links it finds. Pages that exist in isolation are invisible to it.
Server errors and slow response times reduce crawl efficiency and frequency. If Googlebot repeatedly encounters errors when trying to fetch a page, it will deprioritise that page and may stop trying altogether.
How to check your crawling status
Google Search Console's Crawl Stats report shows how often Googlebot has visited your site, how many pages it requested, and whether it encountered any serving problems. For most sites under a few thousand pages, this report is a useful health check rather than a daily concern. For larger sites, it becomes essential for diagnosing why important pages are not getting crawled often enough. For a raw view straight from the source, drop a slice of your server access log into my free log file analyser to see exactly which pages Googlebot is hitting.
The URL Inspection tool in Search Console tells you whether a specific page has been crawled, when it was last crawled, and whether Google was able to access and render it properly. It is the fastest way to diagnose why a page is not appearing in search results.
Once a page is crawled, it moves to the next stage. Crawling just means Google has visited the page - it does not mean Google has stored it or will show it in results. That depends on indexing.