AI Crawler Checker for robots.txt: GPTBot, ClaudeBot and More
See at a glance which AI crawlers may read your site according to robots.txt, separate model-training crawlers from search crawlers, and build rules to block the ones you do not want.
Free · no sign-up · runs in your browser · updated
Use AI Crawler Checker (robots.txt)
Open yourdomain.com/robots.txt in your browser, copy everything, and paste it here. It is read in this browser and not sent anywhere.
E.g. / or /blog/article
Paste your robots.txt above to see which AI crawlers are allowed or blocked. Without a robots.txt, all crawlers are assumed to be allowed.
Results per crawler
Status of each AI crawler according to robots.txt for the path /
On user requestOpens pages when a ChatGPT user or GPT Actions asks it to. OpenAI states that robots.txt rules may not apply because the action is started by a user.
On user requestOpens pages when a Perplexity user asks for an answer with sources. Perplexity states that this crawler generally ignores robots.txt because its requests come from users.
Model trainingControls whether already crawled content may be used to train Gemini and for grounding. A robots.txt token only, with no user agent of its own. It does not affect appearance or ranking in Google Search.
SearchCrawls for Spotlight, Siri and Safari; its data may also help train Apple models. If there is no Applebot rule but there is a Googlebot rule, Applebot follows the Googlebot rule.
Model trainingControls whether data crawled by Applebot may be used to train Apple’s foundation models. It does not crawl by itself; it is only a control token. It does not remove your site from Apple search results.
On user requestFetches specific links when a user asks for a Meta AI feature. Meta states that this crawler may bypass robots.txt rules because it fetches on user request.
On user requestFetches information live to answer a user request, e.g. in Alexa. Amazon states that this crawler may not follow all robots.txt directives.
Model trainingCollects the open web archive (Common Crawl) that is widely used to train models.
Allowed
No rules, so it is allowed
Yes
Each crawler name links to its operator’s official documentation. Crawlers that act on a user’s request often do not follow robots.txt.
Build blocking rules
Tick crawlers in the “Block” column, or use a quick choice, then add the result to your robots.txt.
No crawlers selected yet.
How to use AI Crawler Checker (robots.txt)
1
Open yourdomain.com/robots.txt in your browser, copy everything, and paste it into the robots.txt field. If you have no robots.txt, click “Use example” to try it.
2
Read the summary and table: each crawler is marked Allowed, “Allowed, except some paths” or Blocked, with the rule that decides it. Change “Path to test” to check a specific page.
3
Tick the crawlers you want to block (or use “Model training only”), copy the rules, then add them to the robots.txt on your site.
Training, search and user requests
AI operators generally use different crawlers for different purposes. GPTBot, ClaudeBot and CCBot collect content that may be used to train models. OAI-SearchBot, Claude-SearchBot and PerplexityBot build AI search results that link to your site. ChatGPT-User, Claude-User and Perplexity-User open pages when a user asks them to. Blocking a training crawler is not the same as blocking a search crawler, so a site can still be a source in AI answers.
Google-Extended and Applebot-Extended are different again: they are only robots.txt tokens that control the use of data for training, with no crawler of their own. Blocking them does not change your appearance in Google Search or Apple search.
How this tool reads robots.txt
Parsing follows RFC 9309: a group is chosen by crawler name without regard to upper or lower case, groups with the same name are merged, and if there is no specific group the User-agent: * group is used. The longest matching rule wins, and Allow wins when the lengths are equal. The * and $ characters are supported. Applebot follows Googlebot’s rules when it has none of its own, as stated in Apple’s documentation.
robots.txt is a request, not a security measure. Well-behaved crawlers follow it, but bad ones can ignore it, and some crawlers that act on a user’s request deliberately do not follow it, as their operators state. To protect real content, use authentication or restrictions on the server.
A list that keeps changing
The list of crawlers and what they do comes from each operator’s official documentation as of 1 November 2026. Operators can add new crawlers or change what they do at any time, and a change to robots.txt can take about 24 hours to take effect. Check the operator’s documentation (linked in the table) before you rely on the result.
Sources and date checked
The rules and data in this tool were last checked on . References we use:
Add a User-agent: GPTBot group followed by Disallow: /, and the same for ClaudeBot, in the robots.txt at the root of your domain. This tool builds the rules once you tick the crawlers you mean.
Does blocking AI crawlers remove my site from Google?
No, unless Googlebot is blocked too. Google-Extended only controls use for Gemini and does not affect Google Search. This tool warns you if Googlebot or all crawlers are blocked.
What is the difference between training and AI search crawlers?
Training crawlers collect content to train models; search crawlers build an index so your site can be shown and linked in AI answers. They are controlled separately in robots.txt.
Why does Perplexity-User still read pages I blocked?
Perplexity states that Perplexity-User generally ignores robots.txt because its requests come from users. The same applies to ChatGPT-User, meta-externalfetcher and Amzn-User, according to their operators.
Is my robots.txt sent to a server?
No. Its content is read in your browser; nothing is uploaded. This tool also cannot fetch robots.txt from a URL, so you need to copy it yourself.