How Search Engines Work
Before you optimise anything on your site, you need to understand what you are optimising for. Search engines do not read websites the way humans do. They follow a specific process - and if your site creates problems at any stage of that process, your pages will not rank regardless of how good your content is.
Google processes over 8.5 billion search queries every day, according to its own published data. Delivering relevant results at that scale requires a fully automated system that runs continuously in the background. Understanding how that system works is what separates SEO decisions based on evidence from those based on guesswork.
The three stages every page must pass through
Google's own documentation describes search as working in three stages. Not all pages make it through all three.
Crawling is how Google discovers pages. Automated programs called crawlers follow links across the web, downloading the content they find and passing it back to Google. If Google cannot reach your page - because it is blocked, has no links pointing to it, or is behind a login - the page will never appear in search results.
Indexing is how Google processes and stores what it finds. After crawling a page, Google analyses its content, determines what it is about, and decides whether to add it to the index. The index is the database Google searches when someone types a query. A page that is crawled but not indexed cannot rank.
Ranking is how Google orders results when someone searches. Once a page is in the index, Google evaluates hundreds of signals to decide where it should appear for any given query. This is where most of SEO lives - relevance, authority, page experience, and search intent all play a role.
Why this matters for your SEO
Most SEO problems trace back to one of these three stages. A page that is not ranking might not be crawled at all. A page that is crawled might not be indexed. A page that is indexed might be ranking for the wrong queries because its content does not match what searchers actually want.
Knowing which stage is failing tells you what to fix. Without this foundation, you end up optimising the wrong thing - improving content on a page Google cannot even find, or chasing rankings for keywords that do not match your page's intent.
This is why the How Search Engines Work chapter comes before keyword research, on-page SEO, and everything else. The fixes only make sense once you understand the process they feed into.
What this chapter covers
The four pages in this chapter each go deep on one part of the process:
- Crawling - how Googlebot discovers pages, what blocks it, and how to make sure your site is fully crawlable
- Indexing - what happens after a page is crawled, why some pages are excluded from the index, and how to check your indexing status
- Ranking - the core signals Google uses to order results, including relevance, authority, and page experience
- How Google uses AI in search - how machine learning systems like BERT, RankBrain, and AI Overviews are changing the way results are generated and what it means for your content
Read them in order if you are new to SEO. Each one builds directly on the previous.