We ran this check on India's 500 biggest .in websites. See who blocks ChatGPT, Claude and Perplexity — and who blocks Google.
Two kinds of AI crawler, and why the difference matters
Every big AI company runs more than one crawler. One collects pages to train its models. Another finds pages to show and link when someone asks the assistant a question. They have different names, and robots.txt treats each name separately.
That split is the whole point of this check. Blocking a training crawler is a reasonable choice — your text stays out of the next model. Blocking a search crawler is usually an accident, and it has a cost: the assistant can no longer find your pages, so it can't recommend or cite you.
| Company | Finds pages for answers | Collects pages for training |
|---|---|---|
| OpenAI | OAI-SearchBot, ChatGPT-User | GPTBot |
| Anthropic | Claude-SearchBot, Claude-User | ClaudeBot |
| Perplexity | PerplexityBot, Perplexity-User | — |
Googlebot (Search and AI Overviews) | Google-Extended (Gemini) | |
| Apple | Applebot (Siri, Spotlight) | Applebot-Extended |
How robots.txt decides
The rules are standard (RFC 9309), and this checker follows them the way the crawlers do:
- A crawler obeys one group. If a group names it —
User-agent: GPTBot— it follows that group and ignores the*group completely. Only if nothing names it does it fall back to*. - The longest matching rule wins.
Allow: /blog/beatsDisallow: /for pages under /blog/. When an Allow and a Disallow are the same length, Allow wins. *matches anything,$marks the end.Disallow: /*.pdf$blocks PDFs and nothing else.- A robots.txt that errors blocks everything. If the file returns a server error, crawlers assume they're not allowed in. A missing file (404) means the opposite: no rules at all.
Let assistants in, keep training out
If you want ChatGPT, Claude and Perplexity to find and cite your business but would rather your content didn't train their models, say exactly that:
User-agent: GPTBot User-agent: ClaudeBot User-agent: Google-Extended User-agent: Applebot-Extended User-agent: CCBot Disallow: / User-agent: * Allow: /
The search and answer crawlers aren't named in the first group, so they follow the * group and read everything.
The mistakes we see most
- A staging robots.txt shipped to production.
User-agent: *withDisallow: /keeps every crawler out — Google included. - Blocking "all AI" in one go. Lists copied from the internet often include
OAI-SearchBotandPerplexityBotalongside the training crawlers, which removes the business from AI answers along with the training data. - A firewall that disagrees with robots.txt. CDN and hosting settings can block AI crawlers before they read a single rule. If the check above says "Allowed" and an assistant still can't see you, look there.
Being readable is the first step. To be recommended, an assistant also needs to understand what you are — that's what structured data and an llms.txt are for. The full check looks at all of it at once.