What it is
AI crawlers fall into two classes that people routinely confuse. Live answer engines — OAI-SearchBot, ChatGPT-User, Claude-User, PerplexityBot, Google-Extended, Applebot-Extended — fetch your page because a user asked a question right now. Training crawlers — GPTBot, ClaudeBot, CCBot — collect text that may end up in model weights months later.
Blocking the second group is a legitimate content policy. Blocking the first group means you cannot be cited today, and it is almost always unintentional.
Why an agent cares
A search=yes, ai-train=no policy is a deliberate, defensible position: be answerable now, don't donate your corpus. We do not score it against you.
Blanket-blocking every user-agent with AI in the name — usually copied from a blog post or inherited from a CDN preset — removes you from the answers your buyers are reading.
How to fix it
Separate the two decisions explicitly in robots.txt so the next person can see the intent.
# Answer engines: yes — we want to be citable.
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Google-Extended
Allow: /
# Training corpora: no.
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
Disallow: /
Sitemap: https://example.com/sitemap.xmlHow we test it
We parse your robots.txt with longest-match rules and evaluate each crawler separately. Only blocks on live answer engines cost points; training blocks are reported as policy.