GPTBot: What It Is, Should You Block It, and robots.txt Rules
AI companies run separate bots for training, for search and for fetching pages on a user's request. Block the wrong one and you vanish from ChatGPT answers; block the right one and nothing changes except who trains on your content.
AmIPicked · 11 min read · updated
GPTBot is OpenAI's web crawler for collecting public content that may be used to train its AI models. Blocking it in robots.txt opts your site out of OpenAI model training, and according to OpenAI's own documentation it does not remove you from ChatGPT search. That job belongs to a different bot, OAI-SearchBot.
That split is the most important thing to understand about AI crawlers in 2026. OpenAI, Anthropic and Perplexity each run separate bots for training, for search and for fetching pages on a user's request, and Google and Apple use control tokens instead of separate bots. A robots.txt rule copied from a 2023 article can block the wrong thing. This guide lists every major token, what blocking it does, and rules you can paste.
Check your current setup first: our free AI crawler checker reads your robots.txt and shows which AI bots can reach your site.
What is GPTBot?
OpenAI announced GPTBot in August 2023 as the crawler that gathers training data. OpenAI's current crawler documentation describes it as crawling content that may be used to train its generative AI foundation models, and says that disallowing GPTBot signals that your content should not be used for training.
GPTBot respects robots.txt. It does not decide whether ChatGPT can cite you in a search answer, and it has nothing to do with Google rankings.
GPTBot user agent string
In robots.txt you write only the token, GPTBot. In your server logs you will see the full user agent string. OpenAI lists the current one as:
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot
The version number has changed before and will change again, so never write rules that depend on it. OpenAI notes that requests for robots.txt itself may carry an extra robots.txt marker in the string.
Three kinds of AI crawlers: training, search and user-triggered
Training crawlers collect public pages for future model training. Blocking them keeps your content out of future training data. It does not remove you from AI search answers.
Search crawlers index pages so an assistant can cite them when it searches the web. Blocking them means the assistant cannot quote your pages in its answers, which makes it much harder for it to recommend you.
User-triggered fetchers visit a page because a person asked, for example by pasting your URL into a chat. Because a human started the request, OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores them.
AI crawler user agents and what blocking each one does
Status as of 17 November 2026, based on each vendor's own documentation.
| robots.txt token | Company | Type | What blocking it does |
|---|---|---|---|
| GPTBot | OpenAI | Training | Content is not used to train OpenAI foundation models |
| OAI-SearchBot | OpenAI | Search | Site is not shown in ChatGPT search answers, though it can still appear as a navigational link |
| ChatGPT-User | OpenAI | User-triggered | OpenAI says robots.txt rules may not apply; not used to decide search inclusion |
| OAI-AdsBot | OpenAI | Ads review | Only visits landing pages submitted as ChatGPT ads; not used for training |
| ClaudeBot | Anthropic | Training | Future content is excluded from Anthropic's training datasets |
| Claude-SearchBot | Anthropic | Search | May reduce your visibility and accuracy in Claude's search results |
| Claude-User | Anthropic | User-triggered | Claude cannot fetch your pages when a user asks, which may reduce visibility in user-directed search |
| PerplexityBot | Perplexity | Search | Site is not surfaced or linked in Perplexity answers; Perplexity says this bot is not used for training foundation models |
| Perplexity-User | Perplexity | User-triggered | Generally ignores robots.txt because a user requested the page |
| Google-Extended | Control token | Content is not used to train Gemini models or to ground answers in Gemini Apps and Vertex AI; no effect on Google Search inclusion or ranking | |
| Applebot-Extended | Apple | Control token | Content is not used to train Apple's foundation models; pages can still appear in Apple search features |
| Bingbot | Microsoft | Search (also feeds Copilot) | Removes you from Bing and from what Bing supplies to Copilot; there is no separate AI token |
Google-Extended is not a crawler
Google-Extended has no user agent of its own and never visits your site. Google crawls with its normal crawlers, and the token only tells Google how it may use what it collected. Google's documentation says Google-Extended does not affect inclusion in Google Search and is not a ranking signal.
AI Overviews and AI Mode are part of Google Search, so Google-Extended does not control them. They are governed by Googlebot access and the usual snippet controls such as nosnippet and max-snippet. Blocking Googlebot to stay out of AI Overviews would also remove you from regular search results.
Applebot-Extended works the same way
Apple states that Applebot-Extended does not crawl webpages. It only sets whether content already crawled by Applebot may train Apple's generative models. Applebot itself powers search features in Spotlight, Siri and Safari, so leave it allowed.
Bingbot and Copilot
Microsoft does not run a separate AI crawler for Copilot. Copilot answers draw on the Bing index, so blocking Bingbot removes you from both. Microsoft manages how Copilot may use a page through page-level meta tags documented in the Bing Webmaster Guidelines, not through a robots.txt token, and it has been revising those controls, so read the current guidelines before relying on an older tutorial.
ClaudeBot vs GPTBot
They are the same kind of bot from different companies. GPTBot collects training data for OpenAI; ClaudeBot collects training data for Anthropic. The mapping across the two companies is direct:
| Job | OpenAI | Anthropic |
|---|---|---|
| Training | GPTBot | ClaudeBot |
| Search index | OAI-SearchBot | Claude-SearchBot |
| Fetch on user request | ChatGPT-User | Claude-User |
Two differences are worth knowing. Anthropic supports the non-standard Crawl-delay directive, so you can slow ClaudeBot down instead of blocking it. And Anthropic says its bots, Claude-User included, honor robots.txt directives, while OpenAI warns that robots.txt may not apply to ChatGPT-User.
Does blocking GPTBot affect ChatGPT search?
No. OpenAI's documentation states that each setting is independent. A site can disallow GPTBot and allow OAI-SearchBot, and its pages can still appear and be cited in ChatGPT search answers.
The reverse is the expensive mistake. If you block OAI-SearchBot, OpenAI says your site will not be shown in ChatGPT search answers. Long "block all AI bots" lists copied from the web can include OAI-SearchBot, so check your file even if you never meant to block search.
Should you block GPTBot?
It depends on what you sell.
Allow everything if you are a local or service business that wants AI assistants to recommend you. Your service pages are not a valuable training asset, and letting the model learn about you will not hurt.
Block training, allow search if you publish original work you would rather not hand over for training, such as paid courses, research, journalism or a large content library. This is the most common middle ground and it keeps you visible in AI answers.
Block everything only if you are willing to disappear from AI search answers. That is a legitimate choice for some publishers, but it is rarely right for a business that wants customers to find it.
Whatever you choose, keep search bots and user-triggered fetchers allowed if you want to be recommended. Our guide to how to rank in ChatGPT explains why being citable matters so much.
How to block GPTBot in robots.txt
Your robots.txt file lives at yourdomain.com/robots.txt. To block GPTBot from the whole site:
User-agent: GPTBot
Disallow: /
To block it from one section only:
User-agent: GPTBot
Disallow: /members/
robots.txt template: allow AI search, block AI training
This is the ready-made version of the middle option. It opts out of training for OpenAI, Anthropic, Google and Apple, and leaves every search bot and user-triggered fetcher on the default rules, which allow them.
# Block AI training
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Applebot-Extended
Disallow: /
# Everyone else, including OAI-SearchBot, ChatGPT-User,
# Claude-SearchBot, Claude-User, PerplexityBot, Googlebot and Bingbot
User-agent: *
Disallow: /admin/
Sitemap: https://www.yourdomain.com/sitemap.xml
If you prefer to be explicit, you can add groups that allow the search bots:
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
Note that explicit groups change how those bots read the file, as the next section explains.
To slow Anthropic's training crawler instead of blocking it:
User-agent: ClaudeBot
Crawl-delay: 1
The robots.txt trap: named groups ignore the * rules
A crawler looks for the group that matches its own name. If it finds one, it follows only that group and ignores everything under User-agent: *.
Say your * group blocks /admin/ and /cart/, and you add User-agent: OAI-SearchBot with Allow: /. OAI-SearchBot now ignores the /admin/ and /cart/ rules. If a bot gets its own group, repeat any rules that should still apply to it.
The second classic mistake is a leftover User-agent: * with Disallow: / from when the site was being built. It blocks every bot without its own group, including OAI-SearchBot, PerplexityBot and Googlebot.
A new training opt-out signal is on the way
Several vendors are adopting a shared training preference based on an IETF AI preferences (aipref) draft. Anthropic's documentation says that from 30 October 2026 it recognizes this machine-readable "no AI training" signal and is rolling it out across its services, with robots.txt rules that name ClaudeBot taking priority. Google's crawler documentation says it intends to support the signal, written as user-agent: __usage_ai-training, in early 2027.
The draft may still change, so keep your named rules for GPTBot, ClaudeBot and the others. Treat the shared signal as an extra layer, not a replacement.
How to verify real GPTBot traffic
Any script can send a user agent string that says GPTBot. To know a request is genuine, check its IP address against the vendor's published list:
| Bot | Published IP list |
|---|---|
| GPTBot | openai.com/gptbot.json |
| OAI-SearchBot | openai.com/searchbot.json |
| ChatGPT-User | openai.com/chatgpt-user.json |
| OAI-AdsBot | openai.com/adsbot.json |
| ClaudeBot, Claude-SearchBot, Claude-User | claude.com/crawling/bots.json |
| PerplexityBot | perplexity.ai/perplexitybot.json |
| Perplexity-User | perplexity.ai/perplexity-user.json |
For Applebot, Apple recommends a reverse DNS lookup: the IP should resolve to a hostname under applebot.apple.com, and that hostname should resolve back to the same IP. Apple also publishes a CIDR list in JSON. Googlebot and Bingbot can be verified the same way through Google's and Microsoft's published methods.
Requests that claim to be GPTBot but come from other IPs are impersonators. Block those at your firewall rather than in robots.txt.
Anthropic advises against blocking its IP addresses as a way to opt out, because its bots then cannot read your robots.txt and the opt-out may not register. The same logic applies to every vendor: express preferences in robots.txt, and use IP rules only for fakes.
Cloudflare's AI bot blocking: check your setting
robots.txt is a request; a firewall is a lock. If Cloudflare or another CDN blocks a bot at the edge, it never reads your robots.txt, and you can have a perfect file while ChatGPT still cannot see your site.
Cloudflare groups AI crawlers into three categories: Search, Agent and Training. For each one you can choose Block, Block on pages with ads, or Allow. Cloudflare's documentation, last updated 2 November 2026, lists Allow as the default for all three categories on new domains. There is also a "Disallow: Training only" option that writes training opt-outs to robots.txt and, with Bot Preference Sync turned on, blocks training crawlers at the edge.
Defaults have changed before and older zones keep whatever was chosen when they were set up, so open the setting in your dashboard rather than assuming. Also check security plugins such as Wordfence, your host's firewall and any other bot protection; any of them can block unfamiliar user agents.
How to check which AI bots visit your site
- Read your robots.txt as a bot would. The AI crawler checker evaluates your file for each token in the table above.
- Search your server logs for GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot and PerplexityBot. No visits from search bots for several weeks while robots.txt allows them usually points to a firewall.
- Use your CDN dashboard. Cloudflare shows which AI crawlers request your pages and how many requests were blocked.
- Verify IPs against the lists above before drawing conclusions from logs.
What to do next
Allowing the right crawlers is the entry ticket: it makes you citable, but it does not make AI tools pick you. What decides that is what they find on your pages and what other sites say about you, which our guide to generative engine optimization covers in detail.
To see whether it is working, run the free AmIPicked test. It writes five questions your customers might ask, puts them to ChatGPT and Gemini with live web search, and shows how often you are mentioned and which sources they cite. It also checks your robots.txt, and your llms.txt file and structured data while it is at it.
Frequently asked questions
What is GPTBot?
+
GPTBot is OpenAI's web crawler for collecting public content that may be used to train its generative AI foundation models. Its robots.txt token is GPTBot. It is separate from OAI-SearchBot, which powers ChatGPT search, and from ChatGPT-User, which fetches pages when a user asks.
Does blocking GPTBot affect ChatGPT search?
+
No. OpenAI's crawler documentation says each bot setting is independent. Blocking GPTBot tells OpenAI not to use your content for training, while OAI-SearchBot decides whether your pages can appear in ChatGPT search answers. You can block one and allow the other.
Should I block GPTBot?
+
If you want ChatGPT to recommend your business, keep OAI-SearchBot allowed whatever you decide about GPTBot. Blocking GPTBot is a reasonable choice if you publish original work you do not want in training data. Allowing it costs you little, and it does not affect your Google rankings either way.
What is the GPTBot user agent string?
+
OpenAI lists it as: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot. The version number changes over time, so write robots.txt rules for the token GPTBot only. To confirm a request is genuine, match its IP against openai.com/gptbot.json.
What is the difference between ClaudeBot and GPTBot?
+
Both are training crawlers: GPTBot for OpenAI and ClaudeBot for Anthropic. Each company also runs a separate search bot (OAI-SearchBot and Claude-SearchBot) and a user-triggered fetcher (ChatGPT-User and Claude-User). Anthropic additionally supports the non-standard Crawl-delay directive for its bots.
How long does a robots.txt change take to apply?
+
OpenAI says about 24 hours for its search systems to adjust, and Perplexity says up to 24 hours. Other vendors do not publish a figure, so allow a few days before checking your logs.