One robots.txt line can make your brand invisible to ChatGPT, Perplexity and Google AI Overviews. Check your site in seconds — and get the exact fix.
GPTBot
OpenAI — training & ChatGPT browsing
OAI-SearchBot
OpenAI — ChatGPT search results
ChatGPT-User
OpenAI — user-triggered page fetches
ClaudeBot
Anthropic — Claude
Claude-Web
Anthropic — Claude browsing
PerplexityBot
Perplexity — answers & citations
Google-Extended
Google — Gemini & AI Overviews
Applebot-Extended
Apple Intelligence
Bytespider
ByteDance / TikTok AI
CCBot
Common Crawl — feeds many AI datasets
If GPTBot, ClaudeBot or PerplexityBot cannot read your pages, the assistants they power cannot cite you. Many sites blocked these crawlers years ago without realising it now costs them visibility in AI answers.
GPTBot, OAI-SearchBot and ChatGPT-User belong to OpenAI (ChatGPT). ClaudeBot and Claude-Web are Anthropic. PerplexityBot is Perplexity. Google-Extended controls Gemini and AI Overviews. Applebot-Extended is Apple Intelligence.
If you want to be cited in AI answers, yes. Some publishers deliberately block training crawlers to protect content — in that case allow the search-focused ones (OAI-SearchBot, PerplexityBot, Google-Extended) and block only the training ones.
It fetches your live robots.txt and applies the same precedence rules crawlers use: a crawler-specific group overrides the wildcard group, and an empty Disallow means allow-all.
SeoLens tracks whether the AI engines actually cite you, who they cite instead, and what to change so you get quoted.