Lab · §4 — Robots Generator
Block the bots you actually mean to.
"AI crawler" covers three different jobs — model training, AI search indexing, and fetching pages on demand. Decide per job, copy the rules, paste into robots.txt. Runs in your browser, no sign-up.
01 · Model training crawlers
Collect pages to train future models. Blocking is a licensing stance — it costs you little visibility either way.
02 · AI search indexers
Build the index AI answers cite from. Block these and AI search stops citing your site.
03 · On-demand fetchers
Fetch a page live when a user's question needs it. Blocking hides your pages from AI assistants at answer time.
Output · append to robots.txt
Rules go at the end of your existing robots.txt. One
User-agent block per bot keeps intent auditable —
and honest bots read exactly this.
FAQ · asked a lot
Does robots.txt actually stop AI crawlers?
The named ones, yes. GPTBot, ClaudeBot, PerplexityBot, CCBot and the other bots listed here publish their user agents and honour robots.txt rules. Scrapers that fake a browser user agent ignore robots.txt, but they ignored it before AI existed too — no robots rule addresses them.
Should I block the AI search bots as well as the training bots?
Usually not, and the generator defaults reflect that. Training bots (GPTBot, ClaudeBot, Google-Extended) collect data for future models — blocking them is a licensing stance with little visible effect. Search and fetch bots (OAI-SearchBot, PerplexityBot, ChatGPT-User) are how AI assistants read and cite your site right now — blocking them removes you from AI search answers.
What is the difference between GPTBot and ChatGPT-User?
GPTBot crawls the web to gather training data for future OpenAI models. ChatGPT-User fetches a specific page on demand when a ChatGPT user asks a question that needs it. Same company, different jobs — which is why they read different robots.txt rules and deserve different decisions.
Does blocking Google-Extended affect my Google rankings or AI Overviews?
Neither. Google-Extended is a robots token controlling whether your content trains Gemini models. Search rankings and AI Overviews both run on the ordinary Googlebot crawl, which this generator never touches.