robots.txt: choose your AI policy instead of drifting into one
Every major AI lab publishes a named crawler (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, Bytespider…) and honors robots.txt. Most sites have never decided whether those crawlers may visit — so the default decision (allow everything) was made for them.
We grade the policy, not the choice: explicit rules for the named crawlers pass; a silent policy — where every AI crawler is allowed by omission — is flagged, because it is a decision nobody made.
Run it on your domain — the audit includes this check.
Other tools: SPF · DMARC · llms.txt · Header analyzer