Web Bot Auth

Web Bot Auth is an emerging IETF-track standard for cryptographically verifying that an HTTP request comes from a declared bot operator, rather than from a spoofed user agent or a third party impersonating the bot's IP range. It is a bot-flavored profile of RFC 9421 HTTP Message Signatures (Backman, Richer, Sporny; IETF Proposed Standard, February 2024)¹, formalized in draft-meunier-web-bot-auth-architecture jointly authored by Cloudflare and Google². Each signed request carries three headers (Signature-Agent pointing to the bot's key directory origin, Signature-Input describing the covered components and the tag="web-bot-auth" profile string, and Signature carrying the Ed25519 signature itself); the verifier reads Signature-Agent, fetches the matching public key from that origin at /.well-known/http-message-signatures-directory in JSON Web Key Set format, and the signature verifies (or it does not). The user-agent string and reverse DNS are no longer the only mechanisms a publisher has to identify the bot.

The standard addresses the crawler-controllability gap that user-agent strings, reverse DNS, and IP-range allow-lists leave open. User-agent strings are trivially spoofable; reverse DNS is operationally awkward and not all bots support it; IP-range allow-lists need to be maintained per vendor with no shared format. None of these mechanisms cryptographically proves that a request came from a specific bot operator. For AI engines like Grok where observed retrieval traffic does not surface documented user agents and arrives from rotating datacenter or proxy IPs with spoofed Chrome and Safari UAs, the conventional mechanisms break entirely. Web Bot Auth is the industry's emerging answer: cryptographic identity verification regardless of what user-agent string or IP a request claims.

Status in 2026

RFC 9421 itself is a finalized IETF Proposed Standard, published February 2024 on the Standards Track. Web Bot Auth as a bot-profile of RFC 9421 is defined by two active IETF drafts authored jointly by Thibault Meunier (Cloudflare) and Sam Major (Google), most recently draft-meunier-web-bot-auth-architecture-05 (Informational, published 2026-03-02)² plus a directory-format draft covering the well-known directory shape and JWK Set encoding. The architecture draft specifies the three required HTTP headers (Signature-Agent, Signature-Input, Signature), the tag='web-bot-auth' convention that distinguishes Web Bot Auth signatures from other RFC 9421 applications, and the recommended covered components (notably @authority of the target domain plus the signature-agent value itself). Until the drafts are RFC-finalized, specific deployment particulars may continue to evolve.

Deployment is ahead of finalization, and 2026 has settled into an agent-first / crawler-later adoption split:

• Verifier infrastructure: Cloudflare has incorporated message-signature verification into its Verified Bots program³ and provides documentation for bot operators wishing to register their public key at the standard well-known path⁴. Akamai has documented adoption⁵. AWS has shipped Web Bot Auth signing as a preview feature of Amazon Bedrock AgentCore Browser so hosted AI agents can sign their outbound HTTP traffic⁶. Stytch has shipped signer SDKs.
• Bot-side signing (agentic products): Google publishes its public key directory at https://agent.bot.goog/.well-known/http-message-signatures-directory for the Google-Agent user-triggered fetcher; Google explicitly labels the implementation 'experimental' and notes 'Google is not yet signing every request of agents using the protocol'⁷. OpenAI's ChatGPT agent signs with Signature-Agent: 'https://chatgpt.com' and publishes its directory at that origin⁸.
• Bot-side not signing (retrieval crawlers): regular Googlebot, OpenAI's GPTBot and OAI-SearchBot, Anthropic's ClaudeBot, and xAI's Grok retrieval traffic have no announced Web Bot Auth deployment as of this entry's publication.

For publishers, this means three things in 2026:

• Verify signatures where present. If a request carries a Signature-Agent header and a valid Web Bot Auth signature, the verifier (the publisher's WAF, CDN, or origin server) can definitively confirm or reject the bot's identity.
• Do not treat unsigned bot traffic as fake. Most current AI-engine retrieval traffic is unsigned but legitimate; the absence of a signature today is the rule, not the exception, especially on the search-retrieval crawler side.
• Watch the agentic-product / retrieval-crawler split. The cluster's standing question (per Grok citation) is whether AI search-retrieval crawlers will adopt Web Bot Auth at the same pace as agentic browsing products. The two have diverged in 2026; the divergence is the live story.

How it works

The protocol is intentionally narrow: it covers crawler identity, not crawler behavior. The flow:

1. Bot operator generates a key pair. Ed25519 is the recommended algorithm for the bot's signing key. The private key stays with the bot; the public key is published.
2. Public key is published at the well-known directory. The bot operator hosts the public key in JSON Web Key Set (JWK Set) format at https://<bot-domain>/.well-known/http-message-signatures-directory. HTTPS is a strict requirement. The directory is fetched on demand by verifiers rather than pre-shared.
3. Each outgoing request carries three headers. The bot attaches three required HTTP headers per the Web Bot Auth profile: Signature-Agent containing the HTTPS URL of the bot's key directory origin (e.g., Signature-Agent: "https://agent.bot.goog" for Google's Google-Agent); Signature-Input listing the covered components, key ID, validity window, and the tag="web-bot-auth" string that distinguishes Web Bot Auth signatures from other RFC 9421 applications; and Signature carrying the actual Ed25519 signature over the covered components. The covered components typically include the @authority of the target domain plus the signature-agent header itself.
4. Verifier validates the signature. The publisher's WAF, CDN, or origin server reads the Signature-Agent header to learn the bot's key directory origin, fetches the public key from that origin's well-known directory (caching per Cache-Control headers), reads Signature-Input for the key ID and covered-component list, then verifies the Signature header against the covered components of the request. If the signature verifies, the request is cryptographically tied to the declared bot operator. If it does not, the request is treated as suspect.

The mechanism does not authenticate the request's content beyond the covered components, and it does not bind the request to the publisher's TLS session; RFC 9421 explicitly relies on the Digest specification (RFC 9530 / RFC 9651) when message-content integrity is also needed. For pure identity verification (the Web Bot Auth use case), signing @authority plus a small set of derived components is sufficient.

What it does not solve

Web Bot Auth is identity verification, not policy enforcement. Three things publishers should not expect it to do:

• Web Bot Auth does not enforce crawl rate limits. A signature confirms the request came from the bot operator; it does not constrain how many requests the bot makes per second. Rate limiting remains a separate concern at the WAF or CDN layer.
• Web Bot Auth does not make robots.txt enforceable. A signature confirms identity; whether the bot respects Disallow directives is a behavior question separate from identity verification. A bot that signs its requests and ignores robots.txt is still a bot that ignores robots.txt; the signature just makes the attribution unambiguous.
• Web Bot Auth does not solve unsigned-traffic attribution. Most current AI retrieval traffic is unsigned; publishers still need conventional verification (reverse DNS, IP-range, behavioral) for that majority of traffic. Web Bot Auth raises the ceiling on what is verifiable, not the floor.

What remains contested or unverified

• AI-engine adoption timeline. Cloudflare and Akamai have deployed verification infrastructure; whether AI-engine retrieval crawlers will adopt Web Bot Auth at scale is unverified. The most-watched cases are OpenAI's OAI-SearchBot, Anthropic's ClaudeBot, Google's Googlebot/Google-Extended split, and the current outlier of xAI's Grok retrieval traffic. Public commitments from these vendors as of this entry's publication are limited.
• IETF draft stability. The directory and protocol drafts are still iterating. Specific deployment details (exact required covered components, error-handling for missing or expired keys, key rotation conventions) may shift before RFC finalization.
• Defensive misuse. A bot operator could publish a Web Bot Auth key without committing to robots.txt compliance, then use the signature as a "verified" claim to bypass blocks targeted at unverified bots. Whether publishers will treat Web Bot Auth verification as automatic permission (vs. a separate signal that must still be combined with policy enforcement) is an open operational question.
• Identity revocation. RFC 9421 itself does not specify a revocation mechanism for compromised signing keys; the Web Bot Auth drafts inherit this gap. Until revocation is clearly specified, publishers may need to treat key rotation as the practical defense.

How to apply

• If you operate a CDN or WAF: add Web Bot Auth signature verification to the bot-identity check pipeline. Cloudflare's Verified Bots program ships this verification; for self-hosted infrastructure, RFC 9421 verifier libraries are available in major languages, and the well-known directory format is straightforward to consume.
• If you operate a bot: generate an Ed25519 key pair, publish the public key at https://<your-domain>/.well-known/http-message-signatures-directory in JWK Set format, sign outgoing requests per the Web Bot Auth draft conventions, and register your bot with Cloudflare's Verified Bots program (and any other verifier ecosystem your target publishers participate in).
• If you are a publisher tracking AI-engine traffic: watch for Web Bot Auth signatures on inbound AI-engine traffic over the next 12-24 months. Adoption by OpenAI, Anthropic, Google, Microsoft, or xAI would meaningfully change the Grok citation cluster's crawler-controllability axis. Until adoption arrives, continue using conventional reverse-DNS and IP-range verification for the unsigned majority.

How it relates to other concepts

• Emerging answer to the Grok citation crawler-controllability problem. Grok is the cluster's primary example of crawlers operationally unblockable by user-agent-based robots.txt rules; Web Bot Auth is the IETF-track standard that could (if adopted by xAI) cryptographically tie observed retrieval traffic to xAI's published bot identity regardless of the user-agent string the request carries.
• Complementary to IndexNow protocol: IndexNow is a publisher-to-engine push protocol for URL change notifications; Web Bot Auth is an engine-to-publisher identity proof for retrieval requests. They cover different parts of the publisher-engine communication channel.
• Complementary to the AI crawler bots directory of declared user agents: that entry lists the user-agent strings vendors publish; Web Bot Auth raises the verification standard above what user-agent strings alone can prove. Both entries together form the crawler-discipline layer of the cluster.
• Distinct from llms.txt: llms.txt is a publisher-side proposal for vendor-readable site overview content. Web Bot Auth is a crawler-side identity protocol. The two address different sides of the publisher-engine relationship.
• Indirectly relevant to external traffic disambiguation: when a publisher tries to reverse-attribute an inbound visit, signed bot traffic is cleanly attributable; unsigned bot traffic falls back to conventional disambiguation. Wider Web Bot Auth adoption would simplify the bot-side of traffic disambiguation over time.