Free tool

Can GPTBot and other AI crawlers see your site?

We read your robots.txt and show which AI crawlers are allowed: GPTBot and OAI-SearchBot from OpenAI, ClaudeBot and Claude-SearchBot from Anthropic, PerplexityBot, Google-Extended and more. If you block the search crawlers, ChatGPT, Claude and Perplexity cannot cite your pages.

Which AI crawlers should you allow?

AI crawlers fall into three groups. Search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) fetch pages so the assistant can cite them in answers. User-triggered fetchers (ChatGPT-User, Claude-User, Perplexity-User) open a page when someone asks about it in a chat. Training crawlers (GPTBot, ClaudeBot, CCBot, Google-Extended) collect data to train models.

A business that wants AI to recommend it should allow the first two groups. Blocking training crawlers is a separate decision and does not remove you from search-based answers. The full list of user agents, ready robots.txt snippets and how to verify real bot traffic are in our GPTBot and robots.txt guide.

The checker reads the rules that apply to your homepage. Server-level blocks, such as Cloudflare's AI bot blocking, do not show up in robots.txt, so check your firewall settings too. Next step: see whether ChatGPT actually recommends you.