Robots.txt vs. Noindex vs. Canonical

Robots.txt, the noindex tag, and the canonical tag are three of the most commonly used tools in technical SEO, and three of the most commonly confused. They sound like they're all doing roughly the same job — "keep this page out of search results" — but they actually control three distinct things: crawling, indexing, and duplicate consolidation. Mixing them up doesn't just fail to fix the problem, it frequently creates a new one. This guide lays out exactly what each does, precisely, with no hand-waving.

Robots.txt: controls crawling, not indexing

A robots.txt file is a plain text file at the root of your domain (yoursite.com/robots.txt) that gives crawlers instructions about which parts of the site they're allowed to fetch. A rule like:

tells all crawlers not to fetch anything under /admin/. This is a crawling instruction only. It does not, by itself, prevent a URL from appearing in search results. If other pages (on your site or elsewhere) link to a disallowed URL, Google can still index that URL as a bare address — sometimes with a title pulled from the linking page's anchor text — while never reading its actual content, because it's forbidden from fetching it. This is exactly why disallowed pages sometimes still show up in search results with no description.

Robots.txt is also purely voluntary — it works because well-behaved crawlers choose to respect it, not because it technically blocks access. And critically, the file itself is public and readable by anyone, so it should never be treated as a way to keep something private or secret.

Noindex: controls indexing, but requires crawling to work

The noindex directive — usually a meta tag in the page's head (<meta name="robots" content="noindex" />) or an equivalent HTTP header (X-Robots-Tag) for non-HTML files — tells a search engine directly: don't include this page in search results. This is a real, direct indexing instruction, unlike robots.txt.

The catch is the mechanism itself: the noindex instruction lives inside the page's content (or its HTTP response headers), which means the crawler has to actually be able to fetch the page to see it. A page that's noindexed but also blocked from crawling in robots.txt puts the crawler in an impossible position — it's told not to fetch the page, so it never sees the instruction telling it not to index the page. Result: the noindex tag is silently ignored, and the page can still be indexed via other signals, like inbound links.

Canonical: a consolidation hint, not an exclusion

A canonical tag (<link rel="canonical" href="..." />) addresses a different problem entirely: multiple URLs serving the same or very similar content. It doesn't say "don't index this" — it says "if you're going to index one version of this content, index this specific URL as the primary one." Both the canonical and non-canonical URLs typically remain crawlable and can both still get visited by crawlers; the tag only affects which one search engines treat as the authoritative version for indexing and ranking purposes.

It's also explicitly a hint rather than a directive — search engines weigh it alongside other signals and can choose a different canonical if the declared one doesn't match their own assessment. See canonical tags, explained for the full mechanics.

Side by side

• Robots.txt disallow — controls whether a crawler may fetch a URL at all. Does not guarantee exclusion from search results. Publicly readable. No effect on pages already indexed before the rule was added.
• Noindex — controls whether a specific, crawlable page is included in the index. Requires the page to be crawlable to take effect. The correct tool for "keep this exact page out of search results."
• Canonical — controls which of several similar/duplicate URLs is treated as authoritative. Does not block crawling or guarantee exclusion of the alternates. The correct tool for "these are duplicates, here's the one that should win."

Which one do you actually need?

1. Keep crawlers out of an entire section (staging environment, internal search results, admin area) with no concern about pages already indexed: robots.txt disallow. Pair with actual authentication if the content is sensitive, since robots.txt is not access control.
2. Keep one specific, real page out of search results while still letting it be crawled and pass link value onward: noindex, and confirm it isn't also blocked by robots.txt.
3. Consolidate near-duplicate or parameterized versions of the same content into one indexable URL: canonical tag, pointing every variant at the preferred version.
4. Permanently retire a URL and merge its value into a different, real page: a 301 redirect, which none of the above three tools actually is.

For a walk-through of how to fix duplicate-content situations specifically using canonical tags (and when noindex or a redirect is the better fit instead), see how to fix duplicate content issues. And if the concern is specifically about AI crawlers rather than classic search indexing, see how to check if AI bots can crawl your site, since AI crawlers don't uniformly handle noindex the way Google does.