AI access control

AI access control is the umbrella for the set of signals a website uses to govern how AI systems discover, fetch, identify against, and use its content. There is no single switch. Instead there is a small family of mechanisms, each answering a different question, and the recurring mistake is to collapse them into one imagined "AI opt-out" when they are genuinely separate actions.

"Access control" is used here in a broad publisher-policy sense: most of the mechanisms below are signals or preferences, not hard technical access controls (authentication, paywalls, WAF rules). The surface also spans both restriction (access, usage, identity) and discovery (notifying engines of new content), so it is better read as a control-and-discovery map than as a set of blocks.

The control surface breaks into distinct questions:

The agents all of these are aimed at are AI crawler bots (GPTBot, PerplexityBot, ClaudeBot and others, spanning training crawlers, search and retrieval crawlers, and user-triggered agents).

Two rows sit in the map for context rather than because they restrict anything: llms.txt only guides what AI is pointed to read, and IndexNow only accelerates discovery. Neither controls access or usage; they are included because publishers weigh them alongside the genuine access, usage, and identity signals.

Status in 2026

The defining feature of AI access control in 2026 is that it is fragmented and mostly voluntary. Each question above has its own signal at its own maturity: robots.txt is a settled standard, IndexNow is live, Web Bot Auth and AIPREF are emerging IETF-track work, and llms.txt is a community proposal with no confirmed engine support. There is no unified "AI policy" file that answers all four questions at once, and the efforts that exist deliberately stay in their own lanes (the AIPREF charter, for instance, puts crawler authentication explicitly out of scope, leaving that to Web Bot Auth).

The practical consequence is that "controlling AI access" is not one decision but several, and most of the levers are requests rather than guarantees. robots.txt, llms.txt, and AIPREF all rely on the AI operator choosing to comply. Web Bot Auth changes the picture only in that it lets a site verify identity (so it can decide whom to serve), but verification is not itself a block. The one thing none of these provide is enforcement against an operator who ignores them; that requires technical access control (authentication, rate limits, WAF rules) and, ultimately, contracts and licensing.

How to apply

The useful move is to pick the signal that matches the action you actually want, and to stop expecting one of them to do another's job:

• To limit fetching by compliant crawlers: use robots.txt. Understand it is honored by well-behaved bots and ignored by others, and that it governs fetching only, not downstream use.
• To declare a usage preference (e.g. allow search, disallow training): use AIPREF (train-ai=n, search=y). Treat it as a machine-readable statement of intent whose effect depends on voluntary adoption, and note the attachment syntax is still pre-standardization.
• To guide what AI reads, not whether it may: use llms.txt as curation, while remembering no major engine has confirmed consuming it.
• To know who is actually requesting: adopt Web Bot Auth as it matures, so you can distinguish a verified operator from a spoofed user agent before deciding how to respond.
• To get new content noticed faster: use IndexNow. This is the discovery side of the surface, the opposite of restriction.

Worked example, to make the disambiguation concrete: to stay citable in AI search while opting out of training, you might set AIPREF train-ai=n, search=y, keep robots.txt allowing search crawlers, and adopt Web Bot Auth so a verified search crawler can be told apart from a spoofed trainer. Four signals, one coherent policy, and not a single "opt-out" switch.

What to skip:

• Treating any one signal as a complete "AI opt-out." Blocking crawling, opting out of training, guiding reading, and verifying identity are four different actions; no single file does all four.
• Assuming a preference signal is enforcement. robots.txt, llms.txt, and AIPREF are requests; an operator can ignore them. Reserve "control" language for technical access control and contracts.
• Waiting for one unified standard before acting. The space is fragmented by design; use the mature signal for each goal now rather than expecting convergence soon.

How it relates to other concepts

• AIPREF is the usage-preference layer: it answers "what may the content be used for," and its charter deliberately excludes the identity question, which is why it and Web Bot Auth are complementary rather than overlapping.
• Web Bot Auth is the identity layer: cryptographic verification of which bot is asking, the prerequisite for any access decision that depends on trusting the requester.
• llms.txt is the guidance layer: it shapes what AI reads rather than whether it may, and sits alongside (not inside) the access and preference signals.
• IndexNow is the discovery side of the same surface: where the others restrict or qualify access, IndexNow accelerates it, which is why it belongs in the same map even though it points the opposite direction.
• AI crawler bots are the agents these signals target: the family of signals only makes sense relative to the user agents (GPTBot, PerplexityBot, ClaudeBot, and others) they are meant to govern.
• Upstream of generative engine optimization strategy: deciding which signals to set (stay visible in AI search while declaring a training preference, for instance) is a GEO-policy decision that this map is meant to make legible.