In May, Nieman Lab reported that The Baltimore Banner allowed crawlers used by ChatGPT and Claude while blocking the Internet Archive. Crawlers are automated programs that visit and read webpages. Its technology chief, Biswajit Ganguly, worried that AI products might use archived stories without linking readers back to the newsroom. He said the publication could reconsider the block. Nieman Lab's reporting
That leaves two jobs to account for. A newsroom wants credit and readers for the reporting it pays to produce. An archive preserves earlier versions of webpages so people can check them later.
An access policy can affect both, even when the person setting it is trying to solve only one problem.
A Link Back and a Record of the Past
A link to the publisher helps a reader find the original reporting. An archived copy helps a reader check what appeared at a particular time. Those functions overlap, but neither replaces the other.
Suppose a story is corrected after publication. The current page can explain the correction and show the updated reporting. An earlier copy can help someone understand what changed. If a publication closes or moves to a new website, the current link might stop working altogether.
These are reasons to plan for preservation before something disappears. A publisher's own archive can be part of that plan, alongside arrangements with libraries or other preservation services. The useful question is whether those arrangements will still work when the website, its owner, or its publishing system changes.
The Banner's concern also deserves a precise test. Does an AI answer based on an archived copy fail to credit the newsroom? How often? Would restricting that copy improve the result? A concern can justify investigation without proving that a particular restriction works.
Decide What Each Automated Visitor Can Do
A publisher reviewing crawler access needs a record of the decision, not just a list of blocked names. The person responsible should be able to answer:
- Who operates the crawler, and how has its identity been checked?
- What does the publisher permit it to collect and use?
- How much traffic can the site support?
- What happens to preservation if access is refused?
- Who can approve an exception or review the restriction later?
Those answers need to reach the people configuring the website. If a publisher agrees to let an archive collect public articles at a limited rate, the technical settings should match that agreement. Staff also need a way to report a block that prevents permitted access.
The documentation should connect the purpose to the settings: which pages are included, which crawler is covered, what limits apply, and who owns the arrangement. A later change to the site's bot controls should trigger a check that the agreed access still works.
Check What the Rule Accomplishes
An attribution problem needs evidence about attribution. A traffic problem needs evidence about traffic. A preservation plan needs checks that pages are actually being saved and can be retrieved.
Blocking a crawler demonstrates that access has been restricted. It doesn't, by itself, demonstrate that AI products now credit the publisher more reliably or that the publisher has a dependable archive elsewhere.
The same applies to an exception. Allowing an archival crawler is only a starting point. Someone needs to check that it can reach the intended pages, that the saved copies are usable, and that the arrangement respects the publisher's stated limits.
Newsrooms should be able to protect their reporting and decide how others use it. They also need a clear account of how that reporting will remain available when the current website no longer is.
Source and Scope
This article uses Nieman Lab's May 20 report as a dated example. The Banner's current crawler settings have not been independently checked. The analysis above proposes questions and checks for publishers. It does not establish that the Banner has lost its archive or that its restriction improved attribution.
