Vazagency
Tools/Robots.txt Generator
Free tool

Robots.txt Generator

Build a valid robots.txt with the rules you actually need — including explicit AI crawler access — and download it in seconds.

Robots.txt is a plain text file that tells search engine and AI crawlers which parts of your site they're allowed to request. This tool builds a valid one from simple Allow and Disallow rules, adds a reference to your sitemap automatically, and can generate explicit rules for AI crawlers like GPTBot and ClaudeBot alongside the default rule that applies to Google, Bing, and everything else.

It's built for whoever manages the website — no code required. Pick a starting preset, confirm your site URL, adjust the rules if needed, and download the file.

Getting this file right matters more than its size suggests. A misconfigured robots.txt is one of the most common ways a business accidentally blocks its entire site from search engines — often a leftover "block everything" rule from when the site was still in development. Explicitly welcoming crawlers, including AI ones, is basic technical hygiene for a site that wants to be found.

Start from a preset

Rules

Generated robots.txt

User-agent: *
Allow: /
Disallow: /admin

# AI crawlers — explicitly welcome

User-agent: GPTBot
Allow: /
Disallow: /admin

User-agent: ClaudeBot
Allow: /
Disallow: /admin

User-agent: Google-Extended
Allow: /
Disallow: /admin

User-agent: PerplexityBot
Allow: /
Disallow: /admin

Sitemap: https://yoursite.com/sitemap.xml

How to use this tool

  1. Pick a starting point from the presets — "Allow everything" for a normal live site, "Allow everything except admin" if you have a login or admin area, or "Block everything" if this is a staging or dev site not ready for search engines.
  2. Enter your site's URL in the "Site URL" field so the tool can build the correct sitemap reference.
  3. Review the rules list — each row sets an Allow or Disallow instruction for a specific path. Edit the dropdown and path to match your site's actual folders, or click "Add rule" to add more.
  4. Remove any rule you don't need with the trash icon next to it.
  5. Decide whether to leave "Explicitly welcome AI crawlers" checked — leaving it on adds matching rules for GPTBot, ClaudeBot, Google-Extended, and PerplexityBot.
  6. Copy the generated code or download it as robots.txt, then upload it to the root of your domain at yoursite.com/robots.txt.

One mistake worth double-checking

Many sites get built on a staging URL with a "block everything" robots.txt (or a similar noindex setup) applied on purpose, so search engines don't index the work-in-progress version. The problem shows up when the site goes live and nobody removes that rule — the result is a fully functioning site that Google simply can't crawl, sometimes unnoticed for weeks until traffic numbers look wrong. If you're publishing this for a live site, use the "Allow everything" or "Allow everything except admin" preset, and check the generated output doesn't contain a bare "Disallow: /" line under "User-agent: *".

Frequently asked questions

Does robots.txt keep a page out of Google entirely?
Not reliably by itself. Disallow tells crawlers not to fetch the page's content, but if other pages link to it, Google can still index the URL with no description, based only on the link text. To reliably keep a page out of search results, use a noindex meta tag on the page itself (which requires the page to be crawlable) instead of, or alongside, robots.txt.
Where does robots.txt need to live?
At the root of your domain — yoursite.com/robots.txt — and nowhere else. A robots.txt file placed in a subfolder is ignored entirely.
Should I block AI crawlers like GPTBot?
That's a real choice, not a default. Blocking them means your content won't be used to train or be cited by that AI system's answers. Allowing them means your business can potentially get cited when someone asks an AI assistant a question your content answers — the same logic as SEO, applied to a newer channel. Most content-driven service businesses benefit more from being found than from opting out.
What happens if I have no robots.txt file at all?
Crawlers treat a missing robots.txt as "everything is allowed" — the same practical effect as a file containing just "User-agent: * / Allow: /". A robots.txt is only strictly necessary once you actually need to restrict something (like an admin area) or want to point crawlers to your sitemap.

Related guides

Website Structure for SEOStructured Data (Schema Markup), Explained

Want this handled for you instead?

We build the SEO foundation and handle the ongoing work — no long-term contract, no guaranteed-rankings sales pitch.

Get in touchSee our SEO Foundation