AI Crawler Checker

What is an AI crawler?

An AI crawler is a bot run by an AI company to fetch web pages — either to train a model, to build a search index for AI answers, or to fetch a single page when a user asks about it. The important thing to know is that these are three different jobs, and most AI companies run a separate crawler for each one.

OpenAI, for example, runs GPTBot for training, OAI-SearchBot for ChatGPT search, and ChatGPT-User for user-triggered fetches. Anthropic runs ClaudeBot for training and Claude-SearchBot for search. Perplexity runs PerplexityBot for answers and Perplexity-User for user-triggered fetches. Your robots.txt can treat each one differently.

Training crawlers vs search crawlers — the distinction that matters

Blocking a training crawler is a content-use decision: it tells the AI company not to use your pages to train future models. It costs you nothing in search traffic or AI citations. Blocking a search crawler is a visibility decision: it removes your pages from that AI engine’s answers, and with them the citations and referrals those answers bring.

The most common mistake is blocking both at once. Many sites added a few lines to robots.txt in 2023 to “block AI” and quietly blocked the search crawlers too — then wondered why they stopped appearing in ChatGPT and Perplexity answers. The fix is to split the two decisions, which is exactly what this checker helps you do.

Who uses an AI crawler checker?

Site owners who want to appear in AI answers without giving their content to model training. Publishers checking whether a robots.txt change — or a CDN rule — accidentally locked out the search crawlers. SEO teams auditing a client’s AI visibility, and developers who want to confirm that a new robots.txt does what the team intended before it goes live.

How robots.txt decides: the RFC 9309 rule

Robots.txt is a set of groups. Each group starts with one or more User-agent lines, followed by Allow and Disallow rules. A crawler follows only the group that names it; if no group names it, it falls back to the User-agent: * group. Within a group, the longest matching rule wins — so “Disallow: /” plus “Allow: /public/” lets the bot fetch /public/ but nothing else. On an equal-length tie, Allow wins. That is the rule in RFC 9309, the robots.txt standard, and this checker applies it exactly.

Two more facts matter. An empty “Disallow:” matches nothing, so it means “allowed”. And a missing robots.txt (a 4xx response) allows every crawler, while a server error (5xx) tells crawlers to stay away from the whole site.

The crawlers this checker covers

Sixteen documented bots, in three groups:

• Training — GPTBot, ClaudeBot, anthropic-ai, CCBot, Google-Extended, Applebot-Extended, Meta-ExternalAgent, Bytespider, cohere-ai. Blocking these keeps your content out of model training.
• Search / answers — OAI-SearchBot, Claude-SearchBot, PerplexityBot, DuckAssistBot, YouBot. Blocking these removes you from AI answers and citations.
• User-triggered — ChatGPT-User, Perplexity-User. These fetch a page when a person asks about it; OpenAI says robots.txt rules may not apply to ChatGPT-User because a person started the request.

How to check your robots.txt

Paste your robots.txt above and pick the path to test — the homepage by default, or a section such as /blog/. The checker reports every bot as allowed, blocked or not mentioned, and flags the cases that cost you visibility: a blocked search crawler, or a training crawler you meant to block but did not.

If the result is not what you intended, copy the “allow AI search, block AI training” snippet at the bottom of the tool, adjust it to your site, and paste it into your robots.txt. Then run the check again — OpenAI says changes take about 24 hours to reach its search systems.

Best practices — and common mistakes

Most AI-crawler problems come from a short list of avoidable errors:

• Do not block search crawlers by accident — a blanket “Disallow: /” for User-agent: * blocks every AI crawler that has no group of its own.
• Keep training and search decisions separate — block GPTBot if you want, but keep OAI-SearchBot allowed if you want ChatGPT citations.
• Remember Google-Extended is not a search control — it has no effect on Google Search, AI Overviews or AI Mode.
• Check your CDN and firewall too — robots.txt is only one layer, and a bot rule there can block crawlers your robots.txt allows.
• Re-check after every robots.txt change — and confirm in server logs that the bots you allow are actually visiting.