Skip to content
PromptMentions

Free tool · In your browser, optional live check · no sign-up

AI crawler and robots.txt checker

Paste a robots.txt file (or try a sample) to see which AI crawlers it allows or blocks, grouped by what each crawler is for, with the exact line that decided every result. The check runs in your browser; what you paste is not uploaded.

Samples: · ·

How the checker decides

It follows the robots.txt standard (RFC 9309) the same way PromptMentions's site audit does, with one shared implementation. For each crawler it finds the group whose User-agent line names that crawler's exact token; if none does, the * group applies. Within that group the longest matching Allow or Disallow path wins, and Allow wins a tie. * matches any characters and a trailing $ anchors the end of the path.

Apple documents one exception, which the checker applies: when a file has no Applebot group but has a Googlebot group, Applebot follows the Googlebot rules.

Example

User-agent: GPTBot
Disallow: /

User-agent: *
Disallow: /admin/

GPTBot is blocked everywhere by line 2. OAI-SearchBot has no group of its own, so the * group applies: it may read /pricing but not /admin/users.

Permission, access and visibility are different things

Permission is what robots.txt says. Well-behaved crawlers read it before fetching; it is a request, not an access control. Several operators say their user-triggered fetchers may not apply it.

Access is whether a request actually succeeds. Firewalls, CDN bot protection, rate limits, logins and pages that need JavaScript to show text can all stop a crawler that robots.txt allows. This tool cannot see those.

Visibility is whether an assistant mentions or cites a site in its answers. That depends on far more than crawling, and only measuring real answers over time shows it.

Search, assistant and training crawlers

Blocking a training crawler such as GPTBot or ClaudeBot is a policy choice and does not remove a site from that company's search features, which use separate crawlers (OAI-SearchBot, Claude-SearchBot). Google-Extended is not a crawler at all: it is a token that controls Gemini training and grounding, and Google says it does not affect Google Search, AI Overviews or AI Mode. Those draw on Googlebot's crawl, so they cannot be blocked separately without leaving Google Search.

The crawler list was checked against each operator's documentation on 2026-11-02. Operators add and rename tokens; follow the Docs link for any crawler that matters to you.

Limitations

  • It evaluates robots.txt rules only. It does not test firewalls, CDN settings, server responses or JavaScript rendering.
  • Crawlers not in the list are not evaluated. Some crawlers apply rules differently from the standard (for example, some ignore Crawl-delay).
  • Rules whose path does not start with "/" are ignored, as the site audit does; Google accepts some of these.
  • Live checks of arbitrary websites are not switched on yet: they wait for a secure pinned-address fetch gateway. Until then, paste the file.
  • Being allowed is not evidence that an assistant uses, quotes or cites a site.

Questions

Does blocking GPTBot remove my site from ChatGPT search?

Not according to OpenAI. GPTBot collects training data; ChatGPT search uses OAI-SearchBot, and pages a user asks about are fetched by ChatGPT-User. Each has its own robots.txt token.

Will blocking Google-Extended hide my site from AI Overviews?

No. Google says Google-Extended does not affect Google Search, AI Overviews or AI Mode. It controls use for Gemini training and grounding.

My robots.txt allows everything. Why does an AI crawler still fail to read my pages?

Robots.txt is only permission. A CDN bot rule, firewall, rate limit or a page that needs JavaScript to render its text can still block or hide content. Check your CDN's bot settings and server logs.

Do AI crawlers support wildcards and Allow?

The major operators say they follow robots.txt, and RFC 9309 defines *, $ and Allow. Whether every crawler implements every detail is not something this tool can verify.

Is anything I paste stored or sent anywhere?

No. The pasted file is checked by code running in your browser. The page counts anonymous events such as "a check was run", never the content.