XML Sitemaps and Robots.txt
XML sitemaps and robots.txt solve opposite problems. A sitemap tells Google which pages exist and should be crawled. Robots.txt tells Google which areas to stay out of. Used correctly together, they give you direct control over how Googlebot moves through your site. Used incorrectly, they can block pages you need indexed or waste crawl resources on pages that add no value.
XML sitemaps
An XML sitemap is a file - hosted at your domain root, typically at /sitemap.xml - that lists the URLs you want Google to know about. Search engines are generally good at discovering pages through internal links, but a sitemap provides a direct, reliable signal that does not depend on your internal linking being perfect.
A sitemap is especially valuable when your site is new and has few external links pointing to it, when you have pages that are difficult to reach through internal navigation, when you have a large catalogue of URLs, or when you want to confirm that recently published or updated content is being found promptly.
According to Google's own Search Central documentation, each sitemap file has two hard limits: 50MB uncompressed and 50,000 URLs. For sites exceeding either limit, the correct approach is to split content into multiple sitemap files and reference them all from a sitemap index file - a single file that lists the locations of your individual sitemaps.
What to include: canonical, indexable URLs only - your core service or product pages, published blog posts and articles, and any other pages you actively want to rank.
What to exclude: pages blocked by robots.txt, pages with noindex tags, redirect URLs, duplicate or thin content pages, and any admin or staging URLs. Including non-indexable URLs in your sitemap sends conflicting signals to Google and reduces its confidence in the sitemap as a reliable guide. As the Women in Tech SEO analysis notes, when a sitemap contains many invalid or blocked URLs, Google may reduce the crawl attention it gives the sitemap - or stop trusting it as a useful source altogether.
How to submit your sitemap
Google Search Central recommends submitting through two channels simultaneously. First, submit the sitemap URL directly in Google Search Console under Indexing > Sitemaps. This gives you direct confirmation of how many URLs Google has processed and flags any errors. Second, reference the sitemap inside your robots.txt file with a Sitemap directive. This ensures any crawler visiting your site can find the sitemap immediately, without relying on a manual submission.
Robots.txt
Robots.txt is a plain text file at yourdomain.com/robots.txt. It uses Disallow directives to tell crawlers which URLs or directories to skip. The most common legitimate uses are blocking staging environments, admin panels, duplicate filtered URLs on ecommerce sites, and other low-value sections that would otherwise consume crawl resources without contributing to rankings. If you are writing one from scratch, my free robots.txt generator builds a correct file for WordPress, Shopify or a blank site, with the AI crawlers handled too.
The most important thing to understand about robots.txt is what it does not do. Blocking a URL in robots.txt prevents Google from crawling it, but does not prevent Google from indexing it. If other pages link to a blocked URL, Google can still include it in its index based on those links alone - without ever reading the page content. If you need a page out of Google's index entirely, use a noindex meta tag on the page itself, not a robots.txt disallow directive.
This distinction causes one of the most damaging technical SEO mistakes in practice: adding a noindex tag to a page and also blocking it in robots.txt. Because Googlebot cannot crawl the page, it cannot read the noindex tag, and the page may remain indexed indefinitely.
The critical mistake to avoid
The single most damaging robots.txt error is accidentally blocking your entire site. A misconfigured Disallow: / directive under User-agent: * tells every crawler to avoid every page. This can be silently introduced during a site migration or CMS update and can cause rankings to drop significantly before it is noticed.
Check your robots.txt file in Google Search Console under Settings > robots.txt. The tool shows you the current live file and lets you test specific URLs against it to confirm whether they are blocked or accessible. Run this check after any significant site change - or, before you deploy, paste your file and a list of URLs into my free robots.txt tester, which applies Google's own longest-match rules and shows you exactly what is allowed or blocked.
The two files working together
Your sitemap and robots.txt should never conflict. If a URL appears in your sitemap, it should be accessible to crawlers - not blocked in robots.txt. The sitemap is your list of pages you want Google to find and index. Robots.txt is your list of areas you want Google to skip. There should be no overlap between them.
For most small to medium sites, the practical workflow is straightforward: generate a sitemap automatically via your CMS or SEO plugin - or build one by hand with my free XML sitemap generator - verify it contains only canonical indexable URLs with the sitemap validator, submit it in Search Console, reference it in robots.txt, and then monitor the Indexing report in Search Console for any coverage errors. If you want the full picture of how Google finds these pages in the first place, I break it down in what is a Google crawler.
Next: Canonicalisation