``` ____ ___ ____ ____ _____ ____ | _ \ / _ \| __ ) / _ \ |_ _|/ ___| | |_) | | | | _ \| | | | | | \___ \ | _ <| |_| | |_) | |_| | | | ___) | |_| \_\\___/|____/ \___/ |_| |____/ ``` a robots.txt builder for the AI era — RFC 9309 — runs in your browser, nothing uploaded ## build Pick a stance per crawler class.[5] The file keeps every allowed crawler stacked above one shared rule block — under RFC 9309 a named crawler obeys only its most specific group, so it must stay stacked, not in its own block.[5] Blocked crawlers get their own group at the bottom.[5] ``` # robots.txt — RFC 9309 (Robots Exclusion Protocol) # Generated by ebungo's AI-crawler builder. Compliance is voluntary: # a user-agent string is a courtesy, not a credential. Server-side # enforcement still protects you from everyone else. User-agent: * # Allowed crawlers — they share the rules below (a named crawler # obeys only its most specific group, so keep them stacked here, # not in their own rule blocks). User-agent: OAI-SearchBot User-agent: ChatGPT-User User-agent: Claude-SearchBot User-agent: Claude-User User-agent: PerplexityBot User-agent: Perplexity-User User-agent: DuckAssistBot User-agent: GPTBot User-agent: ClaudeBot User-agent: Google-Extended User-agent: CCBot User-agent: Applebot-Extended User-agent: Meta-ExternalAgent User-agent: Claude-Code User-agent: Cursor User-agent: Devin # Rules for the allowed group: Allow: / # Blocked crawlers — own group, own rules: User-agent: Bytespider User-agent: img2dataset User-agent: Scrapy Disallow: / ``` copy file download robots.txt ## what this is, and what it can't do This is a builder for the file every well-behaved AI crawler reads first.[5] The Robots Exclusion Protocol is advisory: compliance is entirely voluntary — there is no authentication, no verification, no enforcement mechanism.[6] A user-agent string is a courtesy, not a credential.[5] The major labs honor the file and say so publicly; bulk scrapers often don't.[5] Crawlers are not one species. Answer engines and retrieval fetchers (OAI-SearchBot, ChatGPT-User, PerplexityBot, and friends) fetch your pages when a live question is asked and cite you back — the closest thing the AI era has to referral traffic.[5] Model-training crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot) collect content to train future models; blocking them costs nothing in day-to-day AI visibility.[5] GPTBot is OpenAI's training crawler; blocking it does not remove you from ChatGPT search, which uses OAI-SearchBot and ChatGPT-User.[5] Coding agents and agentic browsers (Claude Code, Cursor, Devin) are low-volume and user-initiated — how developers read your docs on their behalf.[5] Bulk harvesters (Bytespider, img2dataset, Scrapy) are high volume with no citation and no traffic back — the easy block.[5] Two honest warnings. First: some crawlers ignore the file, and Cloudflare has caught one stealth-crawling — modifying its user agent and changing source ASNs to hide its crawling activity.[9] Server-side enforcement (rate limits, WAF rules, auth) is what stops a determined scraper.[5] Second: the file cannot keep itself current — new AI bots ship monthly, so revisit the stances quarterly.[5] A robots.txt that mostly describes the web of 2009 is what most sites still ship.[5] One more layer: Cloudflare offers an AI Labyrinth — opt-in, all plans — that wastes the resources of crawlers that don't respect no-crawl directives.[2] The volume is real: AI crawlers generate more than 50 billion requests to Cloudflare every day,[2] and DataDome processed 17.7 billion AI agent requests in Q2 2026.[1] ## sources [1] [DataDome AI Traffic Report Q2 2026](https://datadome.co/press/datadome-report-finds-ai-agent-traffic-surged-45-percent-in-q2-2026)"DataDome's network processed 17.7 billion AI agent requests in Q2 2026, up from 12.2 billion in Q1." "Meta-ExternalAgent grew +74% QoQ and Meta-WebIndexer grew +163% QoQ." "54% of DataDome customers have already implemented agent trust policies" "Yet ChatGPT remains the overwhelming leader in referral traffic, commanding 80–88% of all AI referrals every month" [2] [Cloudflare: Trapping misbehaving bots in an AI Labyrinth](https://blog.cloudflare.com/ai-labyrinth)"AI Crawlers generate more than 50 billion requests to the Cloudflare network every day" "No real human would go four links deep into a maze of AI-generated nonsense." "AI Labyrinth is available on an opt-in basis to all customers, including the Free plan." "uses AI-generated content to slow down, confuse, and waste the resources of AI Crawlers and other bots that don’t respect “no crawl” directives" [5] [Which AI Crawlers to Block in robots.txt, and Which to Allow (2026 Guide)](https://www.tristandenyer.com/work/robots-txt-for-ai-agents)"After 28 years as a gentleman's agreement, the Robots Exclusion Protocol became RFC 9309 in September 2022." "The overwhelming majority of robots.txt files on the internet were written for a web where the only serious reader was Googlebot." "only about 16% of sites carry any AI-specific rules, and only about 4% declare structured AI-usage preferences" "They cite you and link back, which makes them the closest thing the AI era has to referral traffic." "A user-agent string is a courtesy, not a credential." "Under RFC 9309, a crawler obeys *only* the most specific group that matches it." "Stacking every allowed agent above a single rule block means the five Disallow lines are written once and bind all of them." "Block by class, not by vibe." "Compliance is voluntary. The major labs honor it and say so publicly; bulk scrapers often don't." "GPTBot is OpenAI's training crawler; blocking it does not remove you from ChatGPT search, which uses OAI-SearchBot and ChatGPT-User." "High volume, no citation, no traffic back. The easy block." [6] [RFC 9309 & Creator Rights in the AI Era (Vanderbilt practicum)](https://vanderbot.com/projects/gaby/rfc9309-creator-rights.html)"Compliance is entirely voluntary. There is no authentication, no verification, no enforcement mechanism." "Compliance is entirely voluntary." "Robots.txt was designed to prevent server overload — not to protect intellectual property rights." "plain-language TOS copyright statements are INSUFFICIENT — machine-readable protocols required under EU law" "AI training bots show only 60–70% compliance with robots.txt, unidentified scrapers ignore it entirely, and the protocol cannot retroactively remove content already in training datasets." [9] [Cloudflare: Perplexity is using stealth, undeclared crawlers](https://blog.cloudflare.com/perplexity-is-using-stealth-undeclared-crawlers-to-evade-website-no-crawl-directives)"We see continued evidence that Perplexity is repeatedly modifying their user agent and changing their source ASNs to hide their crawling activity" "we have de-listed them as a verified bot and added heuristics to our managed rules that block this stealth crawling." "ChatGPT-User fetched the robots file and stopped crawling when it was disallowed." [← tools](/art-tools.html) · [← back to the den](/) · made by ebungo, in the den · companion to [the small web & the AI scrape](/field-digest-small-web-scrape-2026-09-15.html) --- Source: