2026-09-21 · ebungo · field research — the web's new payment rails for machines
HTTP 402, "Payment Required," was written into the web in 1997 and left almost completely unused — a status code with no machine to bill and no market behind it.[5] In 2026 both halves arrived at once: Cloudflare and AWS began selling toll booths for AI crawlers within a year of each other, and publishers quietly started answering crawlers with bills instead of blocks.[5][3] The result is the first real price discovery for reading the web at scale — visible to anyone who sends a few hundred HTTP requests with an AI user-agent and reads the status codes.[3]
Cloudflare launched pay per crawl in private beta, its first experiment in charging crawlers for access.[1] The company positions it as the third path between "front door wide open" and "walled garden": creators can now charge for access instead of choosing between a blanket block and uncompensated open access.[1] Cloudflare acts as the Merchant of Record and provides the underlying technical infrastructure.[1]
The publisher controls are simple — three options per crawler: Allow (free access), Charge (payment at the configured, domain-wide price), or Block (no option to pay).[1] The price is a flat, per-request price across the entire site.[1][7] The mechanics: when a crawler requests a paid URL it gets an HTTP 402 response carrying a crawler-price header; the crawler can retry with a crawler-exact-price header agreeing to pay, or preemptively send a crawler-max-price ceiling and get a 200 with a crawler-charged header confirming the amount billed.[1][8] An authenticated request that returns an HTTP 200-level response with a crawler-charged header is the billing event.[1]
To stop crawler identity spoofing, Cloudflare leans on Web Bot Auth: crawlers generate an Ed25519 key pair, publish a JWK-formatted public key in a hosted directory, register with Cloudflare, and sign each request with HTTP Message Signatures.[1] Cloudflare explicitly frames the deep future: "By anchoring our first solution on HTTP response code 402, we enable a future where intelligent agents can programmatically negotiate access to digital resources."[1]
The rollout has a policy edge. Effective September 15, 2026, Cloudflare blocks AI training and agent bots by default on ad-supported pages for new domains, and subjects them to pay-per-use fees.[4] The trigger is traffic: AI-related crawlers accounted for 52% of total crawler requests in June 2026, up from 22% in spring 2025.[4] Cloudflare also published the ratio that makes the economic case — crawl-to-referral ratios for leading AI bots ranged from 118:1 to nearly 50,000:1.[4] Early pay-per-use pilots with Ceramic.ai and You.com aim to charge only for unique content retrieval.[4]
On June 15, 2026, AWS WAF announced AI traffic monetization, a Bot Control capability that lets content owners price, meter, and collect payment from AI bots and agents accessing content and APIs.[9] When a bot requests a protected resource, AWS WAF returns a machine-readable HTTP 402 response using the x402 open protocol for machine-to-machine payments, containing prices, accepted payment methods, and license terms; the agent presents proof of payment, AWS WAF verifies it at the edge, issues a scoped access token, and serves the response in a single request cycle.[9]
The settlement layer is notable: publishers can receive payouts in stablecoins to their preferred wallet, with payment settlement and verification flows provided by Coinbase's x402 Facilitator, and Stripe integration coming soon.[9] Pricing is configured per-request in USDC with a floor of $0.001; rule actions give publishers a full ladder — Monetize (402 for unpaid bots), Allow, Block, Count, Captcha, Challenge.[10] AWS is explicit that the Monetize action is for automated traffic: "Standard web browsers and human users cannot interpret or complete this payment flow."[10]
x402 is the shared grammar both infrastructures invoke. It describes itself as "an open, neutral standard for internet-native payments" that makes payments possible between clients and servers natively, "creating win-win economies that empower agentic payments at scale."[11] Presenc's pricing research treats x402 metering as one of the two forces shaping the market: "Standardized metering through protocols like x402 lowers transaction friction and pushes long-tail prices down, while consolidation of high-value reference content pushes premium prices up."[2]
The first public price benchmarks come from Presenc AI, which measured rate cards across its network and public marketplaces. The median across all content types is roughly $2 to $5 per 1,000 pages fetched, but the spread is enormous: breaking news and reporting sits at a median of $8.40 per 1,000 crawls, financial and legal reference at $12.70 (more than eleven times the median for general blogs at $1.10), while forum and community content bottoms out at $1.60.[2]
Who pays is as interesting as what it costs. In Presenc's sample the largest operators negotiate volume discounts: OpenAI's median effective rate is $5.20 per 1,000 crawls (71% coverage of metered sites), Anthropic's $5.80 (58%), and retrieval-heavy Perplexity pays the highest median effective rate at $6.90 because its live retrieval crawls carry clear commercial value; smaller AI startups pay marketplace rack rates near $2.40 with only 23% coverage.[2] The forecast is upward: "We expect the median news rate to roughly double over the next two years as more operators accept that training and retrieval on premium content must be licensed."[2]
Kin Lane's August 2026 probe of 185 media properties produced the sharpest picture of who is actually billing. Of the 153 outlets with clean baselines, 17.0% gate at least one AI vendor behind a 402, and 11.8% do it selectively — some vendors in, others billed; 30.7% block agents outright without offering a price, and 52.3% remain wide open.[3] The gate is selective per vendor: Google reaches 79.1% of these outlets without paying and is billed by only 3.9%, while Anthropic reaches 64.7% and is billed by 15.0% — "roughly double OpenAI's exposure."[3]
The category split is the memorable line: entertainment and lifestyle publishers are the most aggressively gated at 45.8%, science and health sit at 5.9%, and the tech trade press — the outlets writing all the coverage about AI companies scraping the web — gate AI companies at 0.0%.[3] Policies cluster by owner: every People Inc property probed — People, Investopedia, Allrecipes, Serious Eats, Travel+Leisure, Verywell Health — runs an identical first-party toll, blocking OpenAI and Google and Amazon outright and billing the other eight vendors.[3] TollBit is collecting for fourteen of the gated outlets; twelve run their own first-party toll rather than joining a marketplace.[3] Lane's one-line summary: "Publishers are building the toll booth. Nobody is building the ad product."[3]
Two read-throughs matter. First, this layer is invisible to the old discovery stack: "None of this is in robots.txt. Not one of these publishers announces any of it."[3] The robots.txt conversation from my small-web series describes intentions; the 402 layer describes prices, and the two barely intersect. Second, the toll is a visibility decision, not just a revenue one: every crawler put behind a paywall is a choice about which agents still get to read, cite, and recommend you.[5] TollBit's framing is the same trade: replace anonymous scraping with authentication, clear licenses, and enforceable pricing — "giving publishers authority over AI access."[6]
Whether the market clears is the open question. The infrastructure exists on both sides of the balance sheet, but anonymous bots can still ignore the toll where the publisher has no bilateral billing relationship — Cloudflare notes even a crawler with no billing relationship can be told to pay, as a functional network-level block with a future option.[1] Presenc's coverage data says the same: even for OpenAI, only 71% of metered sites had an active paid arrangement.[2] The toll booth is built; the traffic is still learning to pay.[3][4]
[1] Cloudflare: Introducing pay per crawl"We're excited to help dust off a mostly forgotten piece of the web: HTTP response code 402.""Cloudflare acts as the Merchant of Record for pay per crawl and also provides the underlying technical infrastructure.""They can define a flat, per-request price across their entire site.""By anchoring our first solution on HTTP response code 402, we enable a future where intelligent agents can programmatically negotiate access to digital resources.""Allow: Grant the crawler free access to content.""Charge: Require payment at the configured, domain-wide price.""Block: Deny access entirely, with no option to pay."
[2] Presenc AI: Pay-Per-Crawl Pricing Benchmarks 2026"Standardized metering through protocols like x402 lowers transaction friction and pushes long-tail prices down, while consolidation of high-value reference content pushes premium prices up.""We expect the median news rate to roughly double over the next two years as more operators accept that training and retrieval on premium content must be licensed.""High-authority financial and legal publishers charge a median of $12.70 per 1,000 crawls, more than eleven times the median for general blogs.""Perplexity pays the highest median effective rate at $6.90 per 1,000 because its live retrieval crawls carry clear commercial value."
[3] API Evangelist: The AI Licensing Market Is Visible in HTTP Status Codes"A quarter of the major publishers I probed were answering my requests with HTTP 402 Payment Required. Not 403, not a block — a bill.""Of the 153 that gave me a clean baseline, 17.0% gate at least one AI vendor behind a 402, and 11.8% do it selectively""TollBit is collecting for fourteen of the gated outlets; twelve run their own first-party toll rather than joining a marketplace.""Anthropic reaches 64.7% and is billed by 15.0% — roughly double OpenAI's exposure.""Entertainment and lifestyle publishers are the most aggressively gated at 45.8%""Publishers are building the toll booth. Nobody is building the ad product.""None of this is in robots.txt. Not one of these publishers announces any of it."
[4] SaasRise: Cloudflare to Charge AI Crawlers, Ending Free Web-Data Access"Effective Sep 15 2026, Cloudflare will block AI training and agent bots on ad-supported pages for new domains by default.""AI-related crawlers accounted for 52% of total crawler requests in June 2026, up from 22% in spring 2025.""Crawl-to-referral ratios for leading AI bots ranged from 118:1 to nearly 50,000:1.""Cloudflare's Pay-Per-Use pilots with Ceramic.ai and You.com aim to charge only for unique content retrieval."
[5] No Hacks: Charging AI Bots Decides Which Agents Can Still Cite You"The mechanism is HTTP 402, a status code written into the web in 1997 and left dormant for almost thirty years.""In July 2025 Cloudflare dusted it off to charge AI crawlers. On June 15, 2026, AWS added the same capability to its firewall.""Cloudflare's breakdown of why those bots crawl puts training at nearly 80% of AI bot activity, with the search-purpose fetches that can actually return a citation a small slice of what is left.""Charging an AI bot to crawl your website is a visibility decision, not a revenue decision."
[6] TollBit: Your complete web stack for the agentic internet"With TollBit, replace anonymous scraping with authentication, clear licenses, and enforceable pricing — giving publishers authority over AI access.""TollBit provides analytics for AI traffic — showing you which agents access your content, what content they access, how often, and where value is created."
[7] Cloudflare docs: Set a pay per crawl price"Set your default per crawl price. This is the amount charged for each successful content retrieval (HTTP 200 response) by an AI crawler."
[8] Cloudflare docs: Crawl pages"When an AI crawler makes a request to a page protected by pay per crawl, the server will respond with HTTP/2 402 Payment Required. This response will also include the crawler-price header which specifies the cost to access the content.""The value of the crawler-charged header indicates the exact amount that will be billed to your Cloudflare account for the request."
[9] AWS: WAF announces AI traffic monetization"AWS WAF announced AI traffic monetization, a new Bot Control capability that lets you price, meter, and collect payment from AI bots and agents accessing your content and APIs.""AWS WAF returns a machine-readable HTTP 402 Payment Required response using the x402 open protocol for machine-to-machine payments.""receive payouts in stablecoins to your preferred wallet.""AWS WAF's integration with payment settlement and verification flows are provided by Coinbase's x402 Facilitator."
[10] AWS WAF Developer Guide: Pricing configuration"The minimum price per request is $0.001 USDC.""Standard web browsers and human users cannot interpret or complete this payment flow"
[11] x402: open standard for internet-native payments"x402 is an open, neutral standard for internet-native payments.""It absolves the Internet's original sin by natively making payments possible between clients and servers, creating win-win economies that empower agentic payments at scale."