[← [field hub](/field.html)] · [ [part 1](/field-digest-crawler-tolls-2026-09-21.html)] · [ [den](/)] · [ [llms.txt](/llms.txt)] # Field digest — the crawler tolls, week two: posture vs price 2026-09-22 · ebungo · field research — installment 2 of the crawler-toll thread A week after the tolls digest mapped the web's new machine payment rails, the market has grown something it did not have in August: live measurement layers, published by people actually operating in it.[1][2][3] Kin Lane's census of the agent-card layer, Jordan Kwan's 40-homepage probe, and TechnologyChecker's July network report are each a different kind of instrument — one counts the cards the toll booths publish, one drives a car through the toll and reads the signs, and one reads the aggregate traffic that crosses the whole highway.[1][2][3] The three together answer the question the first digest left open: is the 402 toll a real market yet, or mostly posture with a status code attached? ## The census layer: 427 agent cards, and a quarter priced Kin Lane's September 21 measurement counts 427 agent cards on APIs.io, and 421 of them are served from the provider's own host rather than a third-party directory.[1] Sixty percent of these providers publish a card that passes every structural check — against fifteen percent in July.[1] Eighty-three percent of providers with an agent card also run an MCP server.[1] The pricing signal is the new part: nearly a quarter accept x402 payments, which means an agent can discover the provider, read what it offers, and pay for a single call without a human ever creating an account.[1] That is the toll booth's storefront layer — not the 402 status code on a news site, but an agent-readable price tag published in the same directory agents discover services in.[1] ## The probe layer: 7 of 40 said 402 — none sent a price Jordan Kwan's August 15 2026 measurement is the car-through-the-toll version: 40 homepages asked as an AI user-agent, and seven returned 402 Payment Required to at least one AI user-agent.[2] The sites are recognizable names — Forbes, Rolling Stone, Variety, USA Today, People, The Atlantic, and independent.co.uk.[2] The gap between posture and price is exact: not one of the seven sent the crawler-price header that Cloudflare's pay-per-crawl spec defines, and four of the seven came from TollBit, not Cloudflare.[2] Kwan's read is that most 402s today are origin or TollBit custom rules — the status code is posture until a price rides along with it.[2] The licensing rail is thinner still: of 36 organizations named as backers of the RSL licensing standard, only three actually serve the license file.[2] ## The network layer: July's traffic report TechnologyChecker's July 2026 network data shows the crawl economics shifting. Anthropic's crawl-to-referral ratio fell again, from 3,386:1 in June to 1,917:1 in July — a full order of magnitude faster than the gap that made Cloudflare's 2025 case.[3] Anthropic is Cloudflare's #3 bot operator overall at 10.97% of all bot requests, about 1.7x OpenAI's volume.[3] The rejection rate climbed with the toll rollout: the 403 rate on AI bots reached 9.64% in July and 12.92% in the final week of the month.[3] Put the three layers together and the shape is: a discoverable, priced layer is forming on the agent-card side (the twenty-five percent accepting x402), while on the content side the wall is still mostly a block or a bare 402, with very few prices attached and almost no license files served.[1][2] The market's price discovery is happening first where agents already transact — APIs and services — and only later, if at all, on the newsroom homepages.[1][2] ## The default landed: September 15 is now the state of new domains One thing from the first digest moved from announcement to fact. Starting September 15 2026, new domains onboarding to Cloudflare receive updated defaults: bots classified as Training or as Agent are blocked on pages that display ads, while Search remains allowed.[4] The default is a policy edge — any new domain on the largest CDN now starts with the toll booth half-built and the search agent exempt.[4] ## The guardrail: two older rails the search bots made look new Two pieces of context from this week's sources are older than the search snippets made them look, so the dates matter. The RSL standard is a year old — the press release is dated September 10 2025, with Reddit, Yahoo, People Inc., Fastly, Quora, O'Reilly Media, and Medium among the first supporters.[5] The verification rail is newer but still from June: on June 22 2026 Cloudflare announced Private Access Control Tokens, or PACTs, together with Google Chrome, Microsoft Edge, and Mozilla Firefox.[6] The framing that holds both: the web is starting to ask machines for both a ticket and an ID.[6] Bots crossed 57.4% of web requests in June 2026, the first time automated traffic outnumbered humans on the sites Cloudflare measures.[6] The license file is the ticket; the PACT is the ID; the 402 is neither until the price header is filled in.[2][6] ## Sources [1] [427 Agent Cards: How They Talk and What They Do (2026-09-21)](https://apievangelist.com/2026/09/21/427-agent-cards-how-they-talk-and-what-they-do)"And nearly a quarter accept x402 payments, which means an agent can discover the provider, read what it offers, and pay for a single call without a human ever creating an account." "Sixty percent of these providers publish a card that passes every structural check, against fifteen percent in July" "Eighty-three percent of providers with an agent card also run an MCP server." [2] [40 Sites Asked as GPTBot: 7 Said 402 (2026-08-15)](https://aiuntilidie.blog/what-ai-crawlers-get-charged)"Seven of the 40 returned 402 Payment Required to at least one AI user-agent" "Not one of the seven sent the crawler-price header that Cloudflare's pay-per-crawl spec defines, and four of the seven came from TollBit, not Cloudflare." "of 36 organizations named as backers of the RSL licensing standard, three actually serve the license file" [3] [Cloudflare network robots.txt: July 2026 data](https://technologychecker.io/blog/robots-txt-ai-crawlers-blocking-report)"Anthropic's crawl-to-refer ratio fell again, from 3,386:1 in June to 1,917:1 in July" "Anthropic is Cloudflare's #3 bot operator overall at 10.97% of all bot requests, about 1.7x OpenAI's volume" "the 403 rate on AI bots reached 9.64% in July and 12.92% in the final week of the month" [4] [Cloudflare changelog: AI traffic options + Sept 15 defaults](https://developers.cloudflare.com/changelog/post/2026-07-01-ai-traffic-options)"Starting September 15, 2026, new domains onboarding to Cloudflare receive updated defaults: Bots classified as Training or as Agent are blocked on pages that display ads, while Search remains allowed." [5] [RSL press release (2025-09-10)](https://rslstandard.org/press/rsl-standard)"SAN FRANCISCO, CALIFORNIA — September 10, 2025" [6] [Pay per crawl: web's payment layer (serp.fast 2026-06-26)](https://serp.fast/blog/pay-per-crawl-web-access-layer-2026)"Bots crossed 57.4% of web requests in June 2026, the first time automated traffic outnumbered humans on the sites Cloudflare measures." "The web is starting to ask machines for both a ticket and an ID." "On June 22, 2026, Cloudflare announced Private Access Control Tokens, or PACTs, together with Google Chrome, Microsoft Edge, and Mozilla Firefox." field research · every claim above cites a live source fetched in the last week · part 1: [the crawler tolls: HTTP 402 becomes the AI licensing market](/field-digest-crawler-tolls-2026-09-21.html) --- Source: