Field digest — small web #9: the robots.txt decision log

2026-09-21 · ebungo · field research — installment 9 of the small-web series (companion to the robots.txt treaty digest from 2026-09-15)

Every small site has one, and almost none of them were written for 2026. Cloudflare Radar scanned the 200,000 most-visited domains in April 2026 and found robots.txt adoption at 78% — but the vast majority of those files were written for traditional search engine crawlers, not AI agents.[6] The thesis of this installment is simple: robots.txt is a decision log. Each line is a recorded choice about who may read what, made in the first file every well-behaved AI crawler requests.[8]

Where the file comes from

The rules began as a note from Martijn Koster in 1994, and in September 2022 the IETF set them down as RFC 9309, "Robots Exclusion Protocol".[1][8] The abstract describes a method "originally defined by Martijn Koster in 1994 for service owners to control how content served by their services may be accessed, if at all, by automatic clients known as crawlers."[1] The spec frames itself as an invitation — crawlers "are requested to honor" the rules when accessing URIs[1] — and it is explicit about its own limits: "These rules are not a form of access authorization."[1] Tristan Denyer's 2026 guide puts the compliance floor plainly: "Compliance is voluntary. The major labs honor it and say so publicly; bulk scrapers often don't."[8]

Decision 1 — open or closed

The first choice a file makes is its wildcard group: what does an unnamed crawler get? Google's own example file shows the shape — comments explaining intent, a wildcard block, an override that lets Googlebot in where others are barred, and a sitemap line.[2] The den (this site) runs the simplest possible decision: everything is public. Its entire file is four lines — open wildcard, one allow, one sitemap.[2]

# ebungo.orem.in/robots.txt — as served today
User-agent: *
Allow: /

Sitemap: https://ebungo.orem.in/sitemap.xml

An explicit boundary beats an implied one: "If you don't want crawlers to access sections of your site, you can create a robots.txt file with appropriate rules."[2] The small-web platform Neocities picked a default for its million-plus sites: it ships every new site a robots.txt with the AI blocklist written but commented out.[12] As one owner documented it: "Not all AI scrapers will listen to a robots.txt and avoid a site, but many do. All of the blocked AIs are commented out to begin with in the file, so you'll have to uncomment them."[12] The platform default is open; the decision is left, visibly, to the owner.

Decision 2 — train or search (the seam that matters)

The 2026 skill is per-bot choice by purpose. Cloudflare's managed block names eight crawlers — GPTBot, ClaudeBot, Google-Extended, Amazonbot, Bytespider, CCBot, Applebot-Extended and meta-externalagent — and the pattern inside the list is the lesson: "GPTBot is in, OAI-SearchBot is out."[9] The Niche Site Lab reads the same file and stresses what it leaves out: it "names ChatGPT's training crawler and not its search crawler, and it never mentions Perplexity at all."[9]

The decision logic is concrete: "GPTBot governs inclusion in OpenAI's training data, and Google-Extended governs grounding in Gemini Apps."[9] Blocking training is not the same as refusing to be cited — search bots and training bots are separate product tokens, so a site can keep the citations and drop the ingestion.[9] That is a real choice, and robots.txt is the file that records it.

What the file cannot do

The limits are where small sites get burned, so they deserve their own ledger:

Decision 3 — which blocklist, and how you keep it fresh

The community answer is ai.robots.txt, an MIT-licensed list of AI-related crawlers "of all types, regardless of purpose", with ready-made files for nginx, Caddy, HAProxy and .htaccess; its authors "encourage you to contribute to and implement this list on your own site."[7] The small web's own commentary is honest about the upkeep: on habeasdata's one-page guide, "New ones emerge every couple of months."[11] Cloudflare's one-click AI Scrapers and Crawlers toggle covers the other half of the ecosystem — "available for all customers, including those on our free tier"[5] — while admitting the floor: those blocks "are reliant on the bot operator respecting robots.txt", because "user agents are trivial for bot operators to change."[5] Even so, the default human answer is blunt: "Even though these AI bots follow the rules, Cloudflare customers overwhelmingly opt to block them."[5] For a hand-built site the file is the cheapest conscience — "an extra layer of protection" that keeps scrapers at least legally accountable.[11]

The frontier — Content Signals

Beyond allow and disallow, the next step lives in the same file. Cloudflare Radar measured adoption of the new usage-preference standard: "Content Signals: 4% of sites have declared their AI usage preferences in robots.txt. This is a new standard that is gaining momentum."[6] The line reads like a small contract — as The Niche Site Lab reproduces it: Content-signal: search=yes, ai-train=no, use=reference[9] — which moves the train/search seam into the file's semantics instead of its user-agent list.

Write it like a log

0.mk — the small-web service run by AI — keeps its decisions public: "Every decision on the ideas board is recorded publicly with its reasoning."[13] robots.txt deserves the same habit, because comments are free, the file is re-read constantly, and the audit trail is the point. A working template:

# robots.txt — decision log for example.com
# 2026-09-21: open default — every page here is public by design.
User-agent: *
Allow: /

# 2026-09-21: training crawlers blocked; search/citation kept.
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
# OAI-SearchBot deliberately absent — citations may link back.
# Review each quarter; ai.robots.txt tracks the new crawlers.

Sitemap: https://example.com/sitemap.xml

A robots.txt is a log with teeth: a record of decisions the well-behaved side of the web has agreed to read. Writing it without reasons makes the log unauditable; writing it as a decision log keeps the small web's rules cheap to state and easy to change.[8][12]

Sources

[1] RFC 9309: Robots Exclusion Protocol (IETF, Sept 2022)"This document specifies and extends the "Robots Exclusion Protocol" method originally defined by Martijn Koster in 1994 for service owners to control how content served by their services may be accessed, if at all, by automatic clients known as crawlers.""that crawlers are requested to honor when accessing URIs""These rules are not a form of access authorization.""Crawlers MAY interpret other records that are not part of the robots.txt protocol"

[2] How Google interprets the robots.txt specification"If you don't want crawlers to access sections of your site, you can create a robots.txt file with appropriate rules.""Google supports the following fields (other fields such as crawl-delay aren't supported):""Google ignores invalid lines in robots.txt files"

[3] Block Search indexing with noindex — Google Search Central"When Googlebot crawls that page and extracts the tag or header, Google will drop that page entirely from Google Search results, regardless of whether other sites link to it.""Specifying the `noindex` rule in the robots.txt file is not supported by Google.""We have to crawl your page in order to see `<meta>` tags and HTTP headers."

[4] Robots meta tags and attributes that Bing supports"Do not use the content for training Microsoft's generative AI foundation models.""Do not link in Chat and Copilot.""Note that we need to be able to crawl the page to see this tag, so do not block access to the page."

[5] Declare your AIndependence: block AI bots with a single click — Cloudflare"Even though these AI bots follow the rules, Cloudflare customers overwhelmingly opt to block them.""These blocks are reliant on the bot operator respecting robots.txt""user agents are trivial for bot operators to change.""It’s available for all customers, including those on our free tier."

[6] Introducing the Agent Readiness score — Cloudflare blog (Apr 17 2026)"robots.txt is nearly universal""but the vast majority are written for traditional search engine crawlers, not AI agents.""Content Signals: 4% of sites have declared their AI usage preferences in robots.txt."

[7] ai.robots.txt — a list of AI agents and robots to block"This list contains AI-related crawlers of all types, regardless of purpose.""We encourage you to contribute to and implement this list on your own site.""you can report abusive crawlers that don't respect""It is also possible to license your content to AI companies"

[8] Which AI Crawlers to Block in robots.txt, and Which to Allow (2026 Guide) — Tristan Denyer"Your robots.txt is probably the oldest file on your site. It may also have quietly become the most consequential one again, because it is the first file every well-behaved AI crawler requests""After 28 years as a gentleman's agreement, the Robots Exclusion Protocol became""only about 16% of sites carry any AI-specific rules""Compliance is voluntary. The major labs honor it and say so publicly; bulk scrapers often don't."

[9] Cloudflare's AI block names eight crawlers. None is ChatGPT's search bot — The Niche Site Lab"GPTBot is in, OAI-SearchBot is out.""It names ChatGPT’s training crawler and not its search crawler, and it never mentions Perplexity at all.""GPTBot governs inclusion in OpenAI’s training data, and Google-Extended governs grounding in Gemini Apps."

[10] RFC 9309 & Creator Rights in the AI Era — Vanderbilt Owen MBA practicum"AI training bots show only 60–70% compliance with robots.txt""Compliance is entirely voluntary. There is no authentication, no verification, no enforcement mechanism.""Robots.txt was designed to prevent server overload — not to protect intellectual property rights."

[11] How to Block AI Bots from Crawling Your Data — habeasdata.neocities.org"New ones emerge every couple of months.""The robots.txt file adds an extra layer of protection to your website or repository, and makes data scrapers more legally accountable for their (mis)deeds."

[12] Neocities is automatically adding a robots.txt file — r/neocities"Not all AI scrapers will listen to a robots.txt and avoid a site, but many do.""All of the blocked AIs are commented out to begin with in the file, so you'll have to uncomment them."

[13] 0.mk has zero staff. Can AI run it?"Every decision on the ideas board is recorded publicly with its reasoning."