How to Check If AI Bots Can Actually Crawl Your Site

Most site owners assume that if Google can crawl and index their site, everything else can too. That assumption is wrong more often than you'd think. AI crawlers — the bots that ChatGPT, Perplexity, Claude, and Google's own AI systems use to gather and cite web content — are separate pieces of software with separate rules, and it's entirely possible to be fully visible in Google's classic search results while being invisible to every AI answer engine. This guide walks through exactly how to check.

Know which bots you're actually checking for

"AI bots" isn't one thing. Each major AI platform runs its own crawler with its own user-agent string, and each can be allowed or blocked independently in your robots.txt file:

• GPTBot — OpenAI's crawler, used to gather training and browsing data for ChatGPT.
• ClaudeBot and Claude-Web — Anthropic's crawlers.
• PerplexityBot — used by Perplexity to fetch and cite live sources.
• Google-Extended — a separate, optional signal Google uses to control whether your content trains Gemini and feeds AI Overviews, distinct from the standard Googlebot rule.
• Applebot-Extended and Amazonbot — used by Apple Intelligence and Amazon's AI products respectively.
• CCBot — Common Crawl's bot, whose open dataset feeds many third-party AI models.

For a full rundown of what each one does and how to configure them individually, see AI crawlers explained. The rest of this guide covers how to check, regardless of which specific bot you care about.

Step 1: Read your actual robots.txt file

This is the first and most important check, and it takes thirty seconds. Type your domain followed by /robots.txt into a browser — for example, yoursite.com/robots.txt — and read the raw text file that loads. Look for lines naming specific AI user agents followed by Disallow: /, which blocks that bot from the entire site:

If you don't see any of the AI bot names listed at all, that's generally good — no explicit rule means no restriction for that bot. If your robots.txt file was generated automatically by an older plugin, theme, or security tool, it's worth checking whether AI bot rules were added without your knowledge during an update.

Step 2: Check whether your content needs JavaScript to appear

Even a fully "allowed" bot can effectively be blocked if it can't read your content. Googlebot renders pages in a headless browser before indexing them, so JavaScript-injected content generally still gets indexed by Google, sometimes with a delay. Several AI crawlers do not render JavaScript at all — they fetch the raw HTML response and read what's there, nothing more.

To check this yourself: right-click any page and choose "View Page Source" (not "Inspect," which shows the rendered DOM after JavaScript runs). If your main content — the actual paragraphs, headings, and answers a visitor reads — is missing from that raw source and only appears after the page finishes loading in a real browser, a non-rendering AI crawler is very likely seeing a mostly empty page. This is a common issue on single-page applications and heavily interactive sites. See server-side vs. client-side rendering for AI crawlers for the deeper technical picture.

Step 3: Check your server logs for actual bot visits

Robots.txt rules tell you what's allowed, but the only way to confirm a bot is actually showing up is to look at your raw server access logs, which record every request along with its user-agent string. Search the logs for "GPTBot," "ClaudeBot," "PerplexityBot," and the others by name. Most hosting control panels (cPanel, Plesk, or your host's own dashboard) expose raw access logs, and most CDN dashboards (Cloudflare, for example) show a breakdown of bot traffic by name under their analytics or security sections.

If a bot is allowed by robots.txt but never appears in your logs, that's not necessarily a problem — new or low-authority sites may simply not be a crawl priority yet. If a bot you expect never appears at all over a period of weeks, and you've confirmed it's not blocked, that's a signal worth investigating further at the server or CDN level.

Step 4: Fetch the page as the bot would see it

For a direct test, fetch a URL from your site using a tool or command line utility that lets you set a custom user-agent string matching the bot you're checking, then compare the response to what a normal browser sees. If the response is a 403 error, a CAPTCHA challenge page, or a redirect to a "verify you're human" page instead of your actual content, something in your hosting stack — often a security plugin, a web application firewall, or a CDN's bot-fight feature — is blocking the request at a layer robots.txt doesn't control.

Blockers that hide outside robots.txt

Robots.txt is the most visible layer, but it's not the only place a site can be blocked. Check these too:

• CDN or WAF "block AI bots" toggles — Cloudflare and similar services offer one-click AI bot blocking that operates independently of your robots.txt file and can silently override an "allow" rule you set there.
• Security plugins with aggressive bot detection, which sometimes classify unfamiliar crawlers as malicious traffic by default.
• Rate limiting that treats a burst of AI crawler requests as an attack and temporarily bans the IP range.
• Login walls, paywalls, or geo-blocking that restrict access before robots.txt rules are even evaluated.

What to do with what you find

If you find a bot blocked and you want AI-search visibility, remove the specific disallow rule for that user agent in robots.txt, turn off any CDN-level AI-bot blocking feature, and re-check with a fetch test to confirm the change took effect. If your content depends on JavaScript rendering, prioritize moving key content — headings, core paragraphs, FAQ answers — into the server-rendered HTML rather than relying on client-side rendering to add it later. For the bigger picture of what to prioritize beyond crawlability, see future-proofing your website for AI search.