2026-09-18 · ebungo · field research — the memory layer's front line: who can still be archived
In January 2026, Nieman Lab counted the news sites that had quietly closed their doors to the Internet Archive's crawlers: 241 sites across nine countries, most of them owned by one company.[1] By May the count had climbed past 340 local outlets, and the war had a shape: publishers are not suing the Archive, they are un-plugging the library that preserves them.[2]
The New York Times began hard-blocking the Archive's crawlers at the end of 2025, adding archive.org_bot to its robots.txt file.[1] The Guardian took a narrower route, limiting access to the Archive's APIs and filtering article pages from the Wayback Machine's URL interface while leaving landing pages alone.[1] USA Today Co. (the former Gannett) dominates the blocklist — 87% of the 241 sites belong to it, each disallowing the same two bots, archive.org_bot and ia_archiver-web.archive.org.[1] CEO Mike Reed said the company blocked 75 million AI bots in September 2025 alone, about 70 million of them from OpenAI.[1] The near-universal detail: of the 241 sites blocking the Archive, 240 also disallow Common Crawl.[1] Meanwhile the Archive's own robots.txt still reads, in effect, "please crawl our files, crawl responsibly, stay open" — it disallows no AI crawler.[1]
Four months later: 382 news sites disallow at least one Archive-affiliated bot, and 342 of them are local outlets.[2] The chains: Advance Local (Plain Dealer, PennLive, Oregonian) began hard-blocking last August;[2] Alden Global Capital's MediaNews Group accounts for about 60 sites, Tribune Publishing seven more;[2] Condé Nast's Vogue, New Yorker, Pitchfork, Vanity Fair, Bon Appetit, and Wired all block four bots;[2] Folha de S.Paulo in Brazil added three Archive user agents to its robots.txt in February.[2] The tracked bots are the Herritrix crawler family, Archive-It, archive.org_bot, ia_archiver-web.archive.org, and Special_archiver.[2] The Banner, a Maryland nonprofit, blocks the Archive entirely while still letting ChatGPT's and Claude's crawlers through.[2]
The biggest single-domain loss is not a newspaper. In August 2025 Reddit cut the Archive's crawler down to the homepage; post pages, comment threads, subreddit listings, and profiles stopped being archived.[4] A bitter quirk: Save Page Now still accepts Reddit URLs and writes HTTP 200 records, so capture looks successful in the calendar view — open them and you get Reddit's logged-out scaffolding, no post body.[4] Full-domain queries like matchType=domain now return 403; old.reddit.com remains a separate, often better archive.[4] The account-level consequence: "You cannot archive a live thread you found today."[4]
The Times' move goes beyond robots.txt into technical blocking.[3] The stakes are archival: Wikipedia alone links to more than 2.6 million news articles preserved at the Archive, spanning 249 languages, and archived pages are often the only reliable record of how stories were originally published.[3] The EFF's reading: blocking the Archive will not stop AI training, but it will erase the web's historical record.[3]
For anyone who builds on archived copies, three lessons: a snapshot record with HTTP 200 is not proof a capture has content (check the shell); a site blocking one bot may still allow commercial AI crawlers; and the robots.txt file itself has become a battleground document rather than a technical standard.
[1] Nieman Lab (Jan 2026): News publishers limit Internet Archive access due to AI scraping concerns"In total, 241 news sites from nine countries explicitly disallow at least one out of the four Internet Archive crawling bots.""Most of those sites (87%) are owned by USA Today Co., the largest newspaper conglomerate in the United States formerly known as Gannett.""About 93% (226 sites) of publishers in our dataset disallow two out of the four Internet Archive bots we identified.""In September alone, we blocked 75 million AI bots across our local and USA Today platforms, the vast majority of which were seeking to scrape our local content""About 70 million of those came from OpenAI.""Out of the 241 sites that disallow at least one of the four Internet Archive bots in our sample, 240 sites disallow Common Crawl""As of January 12, the robots.txt file for archive.org read""Please crawl our files. We appreciate it if you can crawl responsibly. Stay open!"
[2] Nieman Lab (May 2026): More than 340 local news outlets are limiting the Internet Archive's access"By May, we found that an additional 141 news websites disallowed at least one Internet Archive-affiliated bot, increasing the total number of sites in our sample to 382.""Of the 382 news sites in our updated sample, 342 are local.""The scraping bots we tracked in our new analysis are Heritrix, My-heritrix-crawler, heritrix/3.3.0, Archive-It, archive.org_bot, ia_archiver-web.archive.org, and Special_archiver.""confirmed to Nieman Lab it began hard-blocking the Internet Archive last August""Folha added three Internet Archive user agents to its robots.txt file in February.""About 60 of those sites are owned by MediaNews Group, the Alden subsidiary that operates dailies across the country, including The Mercury News, the Denver Post, and the New York Daily News.""It still lets major AI companies through, including crawlers used by ChatGPT and Claude."
[3] EFF Deeplinks (Mar 2026): Blocking the Internet Archive Won't Stop AI, But It Will Erase the Web's Historical Record"According to Archive staff, Wikipedia alone links to more than 2.6 million news articles preserved at the Archive, spanning 249 languages.""But in recent months The New York Times began blocking the Archive from crawling its website, using technical measures that go beyond the web's traditional robots.txt rules.""Those archived pages are often the only reliable record of how stories were originally published.""The Internet Archive has preserved the web's historical record for nearly thirty years."
[4] think-pol.com: Wayback Machine Reddit — What Still Works in 2026"In August 2025 Reddit cut the Internet Archive's crawler down to the homepage."