Robots.txt Generator

Robots.txt is a plain text file that tells search engine and AI crawlers which parts of your site they're allowed to request. This tool builds a valid one from simple Allow and Disallow rules, adds a reference to your sitemap automatically, and can generate explicit rules for AI crawlers like GPTBot and ClaudeBot alongside the default rule that applies to Google, Bing, and everything else.

It's built for whoever manages the website — no code required. Pick a starting preset, confirm your site URL, adjust the rules if needed, and download the file.

Getting this file right matters more than its size suggests. A misconfigured robots.txt is one of the most common ways a business accidentally blocks its entire site from search engines — often a leftover "block everything" rule from when the site was still in development. Explicitly welcoming crawlers, including AI ones, is basic technical hygiene for a site that wants to be found.

User-agent: *
Allow: /
Disallow: /admin

# AI crawlers — explicitly welcome

User-agent: GPTBot
Allow: /
Disallow: /admin

User-agent: ClaudeBot
Allow: /
Disallow: /admin

User-agent: Google-Extended
Allow: /
Disallow: /admin

User-agent: PerplexityBot
Allow: /
Disallow: /admin

Sitemap: https://yoursite.com/sitemap.xml

How to use this tool

1. Pick a starting point from the presets — "Allow everything" for a normal live site, "Allow everything except admin" if you have a login or admin area, or "Block everything" if this is a staging or dev site not ready for search engines.
2. Enter your site's URL in the "Site URL" field so the tool can build the correct sitemap reference.
3. Review the rules list — each row sets an Allow or Disallow instruction for a specific path. Edit the dropdown and path to match your site's actual folders, or click "Add rule" to add more.
4. Remove any rule you don't need with the trash icon next to it.
5. Decide whether to leave "Explicitly welcome AI crawlers" checked — leaving it on adds matching rules for GPTBot, ClaudeBot, Google-Extended, and PerplexityBot.
6. Copy the generated code or download it as robots.txt, then upload it to the root of your domain at yoursite.com/robots.txt.

One mistake worth double-checking

Many sites get built on a staging URL with a "block everything" robots.txt (or a similar noindex setup) applied on purpose, so search engines don't index the work-in-progress version. The problem shows up when the site goes live and nobody removes that rule — the result is a fully functioning site that Google simply can't crawl, sometimes unnoticed for weeks until traffic numbers look wrong. If you're publishing this for a live site, use the "Allow everything" or "Allow everything except admin" preset, and check the generated output doesn't contain a bare "Disallow: /" line under "User-agent: *".