Create complete product launches, not just content.

Free · AI crawler presets · no sign-up

robots.txt generator, with AI crawlers built in

Pick who may read your site — search engines, AI search, AI training — add the paths that should stay private, and get a robots.txt that does exactly that, checked line by line.

AI search & answers
AI training
Search engines
robots.txt
# robots.txt — made with https://www.trypopulr.in/tools/robots-txt-generator

# Kept out entirely: AI training
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: Meta-ExternalAgent
User-agent: Bytespider
Disallow: /

# Everyone else
User-agent: *
Disallow: /admin
Disallow: /cart
Disallow: /checkout

Check: /admin is closed to Google, and the home page is open.

Save it as robots.txt at your site's root — public/robots.txt in most frameworks — then check it with the AI crawler checker.

What robots.txt does — and doesn't

robots.txt is a short text file at the root of your site that tells crawlers which parts they may read. Search engines and the major AI companies follow it. It's a request, not a lock: it keeps well-behaved crawlers out of your admin pages and away from content you don't want used, but it doesn't hide anything from a person, and it doesn't stop a crawler that ignores it.

The choice most businesses should make about AI

AI companies run separate crawlers for search and for training. The search ones decide whether ChatGPT, Claude and Perplexity can find your pages when someone asks for a business like yours. The training ones decide whether your text helps train the next model. You can say yes to the first and no to the second — that's the "AI search yes, AI training no" preset above, and the choice that keeps you in AI answers without donating your content.

When we checked India's 500 biggest .in websites, almost all of them let AI search in — and the ones that made a deliberate choice mostly blocked training while leaving search open.

Mistakes that cost traffic

  • Disallow: / under User-agent: * — often left over from a staging site. It keeps out every crawler, Google included.
  • Blocking CSS and JavaScript folders. Google renders pages to understand them; blocking /assets or /_next can make your pages look broken to it.
  • Using robots.txt to hide a page. It stays findable through links. Use noindex for that.
  • Forgetting the Sitemap line. It's how crawlers that don't know your site yet find every page.

Already have a robots.txt? Check what it actually allows with the AI crawler checker.

Questions

Does robots.txt keep a page out of Google?

Not reliably. It stops crawling, not indexing: a blocked page can still be listed if other sites link to it, just without a description. To keep a page out of results, let it be crawled and add a noindex tag instead.

Should I block AI crawlers?

Block the ones that collect training data if you'd rather your writing didn't train future models. Think twice before blocking the search and answer crawlers — OAI-SearchBot, Claude-SearchBot, PerplexityBot — because that's how assistants find and recommend you.

Where does robots.txt go?

At the root of your domain: yourbusiness.com/robots.txt. In Next.js, Vite, Astro and most frameworks, put it in public/; in SvelteKit and Hugo, in static/. A robots.txt in a subfolder is ignored.

What happens if robots.txt is missing?

Nothing bad: crawlers treat a missing file (a 404) as permission to read everything. A robots.txt that errors (a 5xx) is worse — crawlers stay away until it answers.

Why are blocked crawlers in their own group?

A crawler obeys only the group that names it. Naming the crawlers you block in one group, and leaving everyone else to the * group, means your private paths apply to everyone who's allowed in — and nothing you meant to block slips through.

More free tools

SEO & AI visibility checkFourteen checks on your whole home page, with a score you can share.AI crawler checkerDoes your robots.txt let ChatGPT, Claude and Perplexity in? Crawler by crawler, with the rule that decides.llms.txt generatorBuild an llms.txt from your sitemap and page titles — the map of your site AI assistants read.Schema markup generatorJSON-LD for your business — address, hours, phone, FAQs — ready to paste into your site.Link preview checkerHow your link looks on WhatsApp, LinkedIn, X and Slack — and what to fix when the picture's missing.Google result previewSee your title and description as Google shows them, and where they get cut off.

Rather have it fixed for you?

Populr reads your site, fixes titles, descriptions, structured data and llms.txt, and keeps them fixed as the site changes. Developers can switch it on from VS Code or Cursor.

Check my whole siteStart free