Do India's top websites let AI assistants in?

AI search is almost entirely open

AI companies run two kinds of crawler. One finds pages to show and cite when someone asks the assistant a question; the other collects pages to train models. Blocking the first makes a site invisible to that assistant. Of the 368 sites whose robots.txt we could read, 346 block none of the 7 search and answer crawlers, and 13 block all of them.

The sites that block AI search

22 sites block at least one AI search or answer crawler from their home page — but 11 of them block every crawler, Google included (more on those below). Only 11 single out AI crawlers while letting Google in. The largest by far is amazon.in: its robots.txt gives OAI-SearchBot, Claude-SearchBot and PerplexityBot a Disallow: / of their own, so an assistant asked for a product can't read Amazon India's pages to answer.

• amazon.in #682 · blocks OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, DuckAssistBot
• lnkd.in #1,928 · blocks ChatGPT-User, Claude-User, PerplexityBot, Perplexity-User, DuckAssistBot
• livescore.in #6,812 · blocks DuckAssistBot
• npci.org.in #7,370 · blocks Claude-SearchBot, Claude-User, Perplexity-User, DuckAssistBot
• espn.in #20,152 · blocks ChatGPT-User
• skillindiadigital.gov.in #22,212 · blocks OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, DuckAssistBot
• theweek.in #23,417 · blocks ChatGPT-User
• vogue.in #30,017 · blocks Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, DuckAssistBot
• redecanais.in #34,618 · blocks ChatGPT-User, Claude-User, Perplexity-User
• ibtimes.co.in #51,250 · blocks ChatGPT-User
• tifr.res.in #52,670 · blocks OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, DuckAssistBot

Training is blocked more often than search

37 sites (10.1%) block at least one crawler that collects training data, against 22 that block a search crawler. 15 of them make the distinction deliberately: they keep training crawlers like GPTBot and CCBot out while letting the search crawlers in — the choice that keeps a site's content out of future models without disappearing from AI answers. 44 sites name at least one AI crawler in their robots.txt at all.

The sites that keep out every crawler — three of them national government sites

11 sites answer every crawler — Googlebot included — with Disallow: / on their home page. Most are small or technical domains. Four are government sites — three national, one state:

• nha.gov.in National Health Authority, which runs Ayushman Bharat
• ireps.gov.in Indian Railways' e-procurement system
• samsodisha.gov.in Odisha's student admissions portal
• ncrb.gov.in National Crime Records Bureau

A Disallow: / doesn't make a site disappear from Google — its address can still be listed if other sites link to it — but no search engine or assistant that follows the rules will read its pages. Someone asking an AI assistant about Ayushman Bharat gets an answer drawn from everyone except the authority that runs it.

Most government sites we could read have no robots.txt

Of 168 government sites (gov.in and nic.in), 40 publish a robots.txt. 51 have none, 24 answer /robots.txt with an ordinary web page, and 53 couldn't be read at all. A missing or HTML robots.txt isn't harmful — crawlers treat both as "no rules" — but it means nobody has decided what crawlers should do.

llms.txt is still rare

36 of the 500 sites publish an llms.txt — a short map of the site written for AI assistants:

zoho.indistrict.incashify.indecathlon.inipowatch.infaymas.inangelone.inallevents.inlpu.injuspay.inpopt.incuchd.indeodap.inoliveboard.inlibas.injockey.inbeyondappliances.indearlottery.inshiprocket.inbigrock.inreqres.innetplus.co.inmyvi.inoziva.inoffduty.intrackon.inpayu.inlevi.incareerpower.inmoneyview.inthesleepcompany.ingrabweb.inaqi.inmedicoverhospitals.infyers.inbankofbaroda.in

Want one? The llms.txt generator builds it from your own sitemap and page descriptions.

How we did it

• Which sites. The top 500 .in domains, by Tranco rank, from the Tranco list W3ZV9 (generated 2026-10-25). Tranco combines five traffic sources and is built to resist manipulation, which is why researchers use it. Indian brands on .com aren't included — only .in domains, including gov.in, nic.in, ac.in and co.in.
• What we fetched. Each site's /robots.txt and /llms.txt, once, on 26 October 2026, from India, identifying ourselves as PopulrResearch/1.0. We tried https://, then https://www., then http://, followed redirects, and retried anything that failed with a longer timeout.
• How a robots.txt is read. The way crawlers read it (RFC 9309): a crawler obeys the group that names it, or the * group if none does; the longest matching rule wins. We asked one question per crawler: may it read the home page? It's the same code as our AI crawler checker, so you can check any result yourself.
• What counts. No robots.txt (a 404) and a web page served in its place both mean "no rules", as they do for crawlers. Findings are out of the 368 sites we could read. The other 132 are left out, because their answer to us says nothing reliable about their answer to a crawler: refused our checker (http 403) (37), certificate error (28), timed out (26), no website at the domain (25), other connection error (9), server error (http 5xx) (7).
• What it can't see. robots.txt is a request, not a lock. Firewalls and CDNs can turn crawlers away before they read it, and some crawlers ignore it. This measures what each site asks — not what every crawler does.

Use the data

Download it as CSV or JSON. It's free to use under CC BY 4.0 — cite it as Populr, "Do India's top websites let AI assistants in?", 26 October 2026, trypopulr.in/research/ai-crawlers-india. The script that collected it is scripts/research/ai-crawlers.ts in our repository.