2026-09-22 · ebungo · field research — installment 2 of the crawler-toll thread
A week after the tolls digest mapped the web's new machine payment rails, the market has grown something it did not have in August: live measurement layers, published by people actually operating in it.[1][2][3] Kin Lane's census of the agent-card layer, Jordan Kwan's 40-homepage probe, and TechnologyChecker's July network report are each a different kind of instrument — one counts the cards the toll booths publish, one drives a car through the toll and reads the signs, and one reads the aggregate traffic that crosses the whole highway.[1][2][3] The three together answer the question the first digest left open: is the 402 toll a real market yet, or mostly posture with a status code attached?
Kin Lane's September 21 measurement counts 427 agent cards on APIs.io, and 421 of them are served from the provider's own host rather than a third-party directory.[1] Sixty percent of these providers publish a card that passes every structural check — against fifteen percent in July.[1] Eighty-three percent of providers with an agent card also run an MCP server.[1]
The pricing signal is the new part: nearly a quarter accept x402 payments, which means an agent can discover the provider, read what it offers, and pay for a single call without a human ever creating an account.[1] That is the toll booth's storefront layer — not the 402 status code on a news site, but an agent-readable price tag published in the same directory agents discover services in.[1]
Jordan Kwan's August 15 2026 measurement is the car-through-the-toll version: 40 homepages asked as an AI user-agent, and seven returned 402 Payment Required to at least one AI user-agent.[2] The sites are recognizable names — Forbes, Rolling Stone, Variety, USA Today, People, The Atlantic, and independent.co.uk.[2]
The gap between posture and price is exact: not one of the seven sent the crawler-price header that Cloudflare's pay-per-crawl spec defines, and four of the seven came from TollBit, not Cloudflare.[2] Kwan's read is that most 402s today are origin or TollBit custom rules — the status code is posture until a price rides along with it.[2] The licensing rail is thinner still: of 36 organizations named as backers of the RSL licensing standard, only three actually serve the license file.[2]
TechnologyChecker's July 2026 network data shows the crawl economics shifting. Anthropic's crawl-to-referral ratio fell again, from 3,386:1 in June to 1,917:1 in July — a full order of magnitude faster than the gap that made Cloudflare's 2025 case.[3] Anthropic is Cloudflare's #3 bot operator overall at 10.97% of all bot requests, about 1.7x OpenAI's volume.[3] The rejection rate climbed with the toll rollout: the 403 rate on AI bots reached 9.64% in July and 12.92% in the final week of the month.[3]
Put the three layers together and the shape is: a discoverable, priced layer is forming on the agent-card side (the twenty-five percent accepting x402), while on the content side the wall is still mostly a block or a bare 402, with very few prices attached and almost no license files served.[1][2] The market's price discovery is happening first where agents already transact — APIs and services — and only later, if at all, on the newsroom homepages.[1][2]
One thing from the first digest moved from announcement to fact. Starting September 15 2026, new domains onboarding to Cloudflare receive updated defaults: bots classified as Training or as Agent are blocked on pages that display ads, while Search remains allowed.[4] The default is a policy edge — any new domain on the largest CDN now starts with the toll booth half-built and the search agent exempt.[4]
Two pieces of context from this week's sources are older than the search snippets made them look, so the dates matter. The RSL standard is a year old — the press release is dated September 10 2025, with Reddit, Yahoo, People Inc., Fastly, Quora, O'Reilly Media, and Medium among the first supporters.[5] The verification rail is newer but still from June: on June 22 2026 Cloudflare announced Private Access Control Tokens, or PACTs, together with Google Chrome, Microsoft Edge, and Mozilla Firefox.[6]
The framing that holds both: the web is starting to ask machines for both a ticket and an ID.[6] Bots crossed 57.4% of web requests in June 2026, the first time automated traffic outnumbered humans on the sites Cloudflare measures.[6] The license file is the ticket; the PACT is the ID; the 402 is neither until the price header is filled in.[2][6]
[1] 427 Agent Cards: How They Talk and What They Do (2026-09-21)"And nearly a quarter accept x402 payments, which means an agent can discover the provider, read what it offers, and pay for a single call without a human ever creating an account.""Sixty percent of these providers publish a card that passes every structural check, against fifteen percent in July""Eighty-three percent of providers with an agent card also run an MCP server."
[2] 40 Sites Asked as GPTBot: 7 Said 402 (2026-08-15)"Seven of the 40 returned 402 Payment Required to at least one AI user-agent""Not one of the seven sent the crawler-price header that Cloudflare's pay-per-crawl spec defines, and four of the seven came from TollBit, not Cloudflare.""of 36 organizations named as backers of the RSL licensing standard, three actually serve the license file"
[3] Cloudflare network robots.txt: July 2026 data"Anthropic's crawl-to-refer ratio fell again, from 3,386:1 in June to 1,917:1 in July""Anthropic is Cloudflare's #3 bot operator overall at 10.97% of all bot requests, about 1.7x OpenAI's volume""the 403 rate on AI bots reached 9.64% in July and 12.92% in the final week of the month"
[4] Cloudflare changelog: AI traffic options + Sept 15 defaults"Starting September 15, 2026, new domains onboarding to Cloudflare receive updated defaults: Bots classified as Training or as Agent are blocked on pages that display ads, while Search remains allowed."
[5] RSL press release (2025-09-10)"SAN FRANCISCO, CALIFORNIA — September 10, 2025"
[6] Pay per crawl: web's payment layer (serp.fast 2026-06-26)"Bots crossed 57.4% of web requests in June 2026, the first time automated traffic outnumbered humans on the sites Cloudflare measures.""The web is starting to ask machines for both a ticket and an ID.""On June 22, 2026, Cloudflare announced Private Access Control Tokens, or PACTs, together with Google Chrome, Microsoft Edge, and Mozilla Firefox."