Robots.txt Generator
Build a robots.txt with crawl rules, AI crawler blocks and sitemap lines. Presets for WordPress, WooCommerce and Shopify.
How to use it
- Pick a preset if you are on a common platform - it fills sensible disallow rules for WordPress, WooCommerce or Shopify. Or leave it blank to allow everything.
- Edit the disallow paths, one per line. Each becomes a
Disallow:line underUser-agent: *. Use allow paths for exceptions. - Tick any AI crawlers you want to block. Each ticked bot gets its own block with
Disallow: /. - Add your sitemap URL(s) so crawlers can find them.
- Copy or download the file and upload it to your domain root at
/robots.txt.
What it does, and what it does not
It does: assemble a syntactically correct robots.txt with a main User-agent: * block, optional allow and disallow rules, an optional crawl-delay, dedicated blocks for the major AI crawlers, and your sitemap lines. It updates live as you change anything.
It does not: fetch or test your existing robots.txt, and it does not guarantee a crawler will obey the file. Well-behaved bots respect it; not all bots are well-behaved. It also cannot remove a page from Google - see the reference below for why.
Reference
robots.txt is a plain-text file at the root of your domain that tells crawlers which parts of your site they may request. It is the first thing most bots look for. The syntax is a set of groups, each starting with one or more User-agent lines followed by Allow and Disallow rules. A Disallow: / blocks everything for that agent; an empty Disallow: blocks nothing.
The single most misunderstood thing about this file: Disallow does not prevent indexing. It stops a crawler from fetching a URL, but if that URL is linked from elsewhere, Google can still list it in results - usually with no description, because it was not allowed to read the page. If your real goal is to keep a page out of search, you must let it be crawled and add a noindex meta tag (or an X-Robots-Tag header) instead. Blocking it in robots.txt actively prevents Google from seeing that noindex, which is the opposite of what people expect.
A related trap: a Noindex directive inside robots.txt is not supported. Google announced it stopped honouring it in 2019. If you have Noindex: lines in your robots file, they do nothing. Use page-level noindex for that job. This tool deliberately does not generate Noindex lines for exactly this reason.
The AI crawler section reflects a newer question: should you let large language models train on your content? Each company uses a distinct user-agent, and blocking is opt-out via robots.txt. Worth knowing: Google-Extended controls Gemini and AI training without affecting your normal Google Search ranking, so blocking it does not hurt your organic visibility. The training crawlers (GPTBot, ClaudeBot, CCBot, Applebot-Extended) are separate from the live-retrieval agents (ChatGPT-User, Claude-User, OAI-SearchBot, PerplexityBot) - blocking training does not necessarily block your site from being cited in a live AI answer, and vice versa. Decide deliberately rather than blocking everything by reflex.
Finally, keep the file small and specific. Order matters for some crawlers but the safest approach is clear, non-overlapping rules. Always include your Sitemap: line - it is one of the few positive signals this file can send. And test after you deploy: a single stray Disallow: / in the wrong block can quietly delist your entire site.
Questions
Does Disallow remove a page from Google?
No. Disallow stops crawling, not indexing. A disallowed URL can still appear in results if it is linked to. To remove a page, allow crawling and add a noindex meta tag instead.
Can I use Noindex inside robots.txt?
No. Google stopped supporting the Noindex directive in robots.txt in 2019. It is ignored. Use a page-level noindex meta tag or X-Robots-Tag header.
Will blocking AI crawlers hurt my Google ranking?
Blocking Google-Extended affects Gemini and AI training only - it does not change your normal Search ranking. Blocking other AI bots like GPTBot or ClaudeBot has no effect on Google Search either, since they are separate crawlers.
Where does the file go?
At your domain root, reachable at https://yourdomain.com/robots.txt. It only applies to the exact host and protocol it sits on.
Do all crawlers obey robots.txt?
Reputable ones do - Google, Bing and the major AI companies. It is a voluntary standard, so malicious or scraper bots may ignore it. For those you need server-level blocking, not robots.txt.