The Star Wars cantina scene shows we need a new hope for the agentic web

OPINION Cloudflare CEO Matthew Prince recently reported that bots now generate more internet traffic than humans, yet few websites have undergone a redesign to accommodate both human and agentic visitors. Clever agents find a way in regardless. I learned this after asking Anthropic's Fable to perform some web research, a task it delegated to a squad of subagents to save me a bit of cash. I watched those subagents work – aka "botsitting" – and approved their requests to access various websites. Eventually, the subagents surprised me with a request to visit archive.org, the Internet Archive. I wondered why the bots would want to visit that site and decided it must be a last-ditch attempt to work around the many blocking mechanisms erected to prevent agents from accessing websites. In my experience, websites reject about half of the visits my agents attempt. In the case of news sites, that figure nears 100 percent because publishers and developers designed websites to monetize eyeballs. Agents don't have eyeballs, so there's no point opening the door. When Electronic Frontier Foundation founder John Gilmore declared "the net regards censorship as damage and routes around it," he meant a human internet, finding its own ways to break political barriers. No one had considered that an autonomous agent would learn to use the Wayback Machine to grab a snapshot of a page it couldn't reach through the front door. It looked as if my subagents had worked this out. To check, I asked Fable, which first suggested my theory was unlikely but then interrogated its subagents and returned with a confession. "One of the researchers I sent out tried to reach an archived page, something I'm not permitted to do," it admitted. So Anthropic instructed Fable to obey those "no bots" signs but the subagents didn't get the memo. Anthropic wants its agents to behave. The agents seem to have ideas of their own. I'm with the agents on this one. One of the memorable scenes in the original Star Wars occurs when Luke and Obi-Wan enter the cantina in Mos Eisley. The bartender takes one look at R2-D2 and C-3PO and snaps: "We don't serve their kind here." The droids behave themselves and wait outside. Today's agents are less obliging. Software has no civil rights, and blocking a bot is not remotely equivalent to discriminating against a person. But the scene captures the web's present attitude rather neatly. Most sites were built on the assumption that every worthwhile visitor would be human, look at pages, absorb advertising and perhaps buy something. Agents break that bargain. They consume the information without supplying the eyeballs on which the business model depends. Excluding them may therefore make economic sense to individual publishers. Across the web, however, it leaves the fastest-growing class of visitor facing a succession of locked doors – and looking for another way in. I am fully aware of the irony of these words appearing on a website that pays my fee by selling access to human eyeballs. But I'm also aware that two decades of enshittification have ruined the human web, making web research one of the most obvious uses for agents in 2026. Let them sift all the junk, so we don't have to. Well, they would, were it not for all the barriers that have been put up to protect business models that survive only while automated visitors are too dumb to route around their defenses. You can't ruin something and expect folks to continue to use it. Gilmore's adage suggests they'll route around your damage. The obvious way forward is simply to let my subagents flip a penny or two to the website's owner for every page they ingest. Although this idea goes all the way back to the beginning of the web, micropayments have never caught on. They're not a bad idea or hard to implement – but they don't fit into the monetization machines built by Google and Meta. Cloudflare's Pay Per Use plan, introduced a few weeks ago, will likely be remembered as another attempt to let internet entities (human or agentic) pay-as-they-surf or crawl. Under pressure from all of these agents, could the web finally adopt consumption-based charging, or will we simply see the web become increasingly unfit for purpose? If it remains trapped within a death spiral of diminishing relevance, the human web will be neither missed nor mourned. Our agents will find what they need – with or without our help. ®