Robots.txt (Robots Exclusion Protocol)

Robots.txt is the plain-text file at a site's root (https://example.com/robots.txt) that tells crawlers which URL paths they may fetch. Proposed by Martijn Koster in February 1994 and a de facto convention for nearly three decades, it was standardized as the Robots Exclusion Protocol (REP) in RFC 9309 in September 2022¹². The file's vocabulary is small: User-agent lines name which bot the following rules address, Disallow and Allow lines list path prefixes, and a Sitemap line points to the XML sitemap. That is essentially the whole protocol.

What robots.txt is not is an enforcement mechanism, and in the AI era that distinction carries most of the weight. RFC 9309 standardizes the file format and parsing rules; compliance is voluntary, and the protocol cannot compel any crawler to obey it¹. Google's own documentation states plainly that robots.txt "is not a mechanism for keeping a web page out of Google"³. A Disallow line is a request that well-behaved bots honor. Everything beyond that, who is actually asking, what they do with content they already have, and what happens when they ignore the request, belongs to other layers of AI access control.

Minimum viable robots.txt

# https://example.com/robots.txt
User-agent: *            # rules for every bot
Disallow: /drafts/       # please do not fetch these paths

User-agent: GPTBot       # rules for one named crawler
Disallow: /              # asks OpenAI's training crawler to stay out

Sitemap: https://example.com/sitemap.xml

Rules bind (voluntarily) per named user agent; the full per-engine table of AI crawler names and their documented purposes lives in the AI crawler bots entry.

Status in 2026

Robots.txt is simultaneously at its most standardized and its most stressed. The standard half: RFC 9309 has been a Proposed Standard since September 2022, the major search and AI operators document robots.txt support for their declared crawlers (xAI's Grok is the documented exception: no first-party crawler contract), and new proposed machine-readable usage-preference signals are being designed to ride inside the same file; the IETF's AIPREF work attaches its proposed Content-Usage rule to robots.txt as a carrier, three decades after the format was sketched on a mailing list².

The stressed half is specific and well documented, mostly by the AI operators themselves. The "does robots.txt stop AI" question decomposes into four gaps. First, compliance is voluntary by design: declared crawlers like GPTBot and ClaudeBot document that they honor it, but nothing in the protocol compels anyone¹. Second, OpenAI and Perplexity document partial exemptions for their user-initiated fetchers: OpenAI states robots.txt rules "may not apply" to ChatGPT-User because a human asked for the page, and Perplexity states Perplexity-User "generally ignores" robots.txt for the same reason⁴. Note that this exemption is the operators' own classification (user-initiated fetches framed as browser-like use outside the protocol's scope), not a neutral reading of RFC 9309; the dispute below challenges exactly that boundary. Third, beyond the documented carve-outs sits undeclared crawling: Cloudflare's August 2025 investigation found Perplexity fetching never-indexed test domains through an undeclared Chrome-impersonating user agent after the declared crawler was disallowed; Perplexity disputed the framing, and it remains a single documented incident rather than a measured general rule⁵. Fourth, blocking is not retroactive: a Disallow: / for a training crawler stops future collection by that compliant bot; it removes nothing from models already trained.

There is also a quieter trap on the search side: blocking a URL in robots.txt does not deindex it. Google documents that a disallowed page can still be indexed if linked from elsewhere, and a page the crawler cannot fetch cannot show the engine its noindex directive, so robots.txt-blocking a page disables the one mechanism that would actually remove it³.

How to apply

Robots.txt is the first lever of AI access control, not the last. Three concrete moves:

• Audit what your file actually asks. Fetch your own /robots.txt and read it as a bot would: which user agents are named, which paths are disallowed for *. Validate behavior with Search Console's robots.txt report, and remember rules are matched per named agent, not inherited across them.
• Split retrieval from training deliberately. Allowing retrieval bots (OAI-SearchBot, PerplexityBot) keeps you eligible for the citation surfaces those crawlers feed (eligibility, not a guarantee of citation); blocking training bots (GPTBot, ClaudeBot) or setting the Google-Extended control token is an independent licensing decision. Watch the trap: Google-Extended governs Gemini/Vertex training, not AI Overviews, which use Googlebot. Per-engine names and purposes: AI crawler bots.
• Pair it with enforcement where it matters. For actors that ignore requests, the protocol has no answer; that layer is bot management and WAF rules (and, for verifying who is actually asking, the emerging Web Bot Auth). Treat robots.txt as the documented, machine-readable statement of crawl policy that compliant bots will follow, and assume nothing stronger.

What to skip: a blanket Disallow: / for every AI user agent as a reflex "AI opt-out." It removes your citation eligibility on the surfaces that depend on those crawlers (OpenAI states sites blocking OAI-SearchBot "will not be shown in ChatGPT search answers"), does not untrain existing models, and does not bind user-initiated fetchers. If the actual goal is "stay searchable, don't train on me," that is a per-agent split plus an emerging usage-preference signal, not a wall.

How it relates to other concepts

• AI access control is the umbrella this entry sits under: four questions, four mechanisms. Robots.txt answers only the first (may you fetch this URL), and the umbrella entry's worked example shows it combined with the other layers into one coherent policy.
• AI crawler bots are the agents robots.txt addresses. That entry carries the per-engine user-agent table (training vs retrieval vs user-initiated, control tokens like Google-Extended) and the copy-paste ruleset; this entry covers what the protocol itself can and cannot promise.
• AIPREF is the usage-preference layer robots.txt lacks: a Disallow cannot say "fetch this but don't train on it." The proposed Content-Usage rule rides inside robots.txt, making the same file a carrier for both access and preference signals.
• llms.txt is frequently confused with robots.txt but is guidance, not access: a curated AI-readable map with no exclusion semantics and, unlike robots.txt, no confirmed engine support.
• Web Bot Auth addresses the identity gap that makes robots.txt spoofable: user-agent strings are self-declared, so a rule binds whoever chooses to identify as that agent. Cryptographic crawler identity is what would let a site enforce per-bot policy rather than request it.