Sean Dougherty 🌍

As AI labs race toward Recursive Self Improvement, there’s a possibility that what they are actually racing toward is Recursive Self Extinction — of their business model. Here’s my reasoning. 1. Progress on verifiable tasks continues. In particular, improving the cost efficiency of existing capabilities is a highly verifiable task. For the past few years, cost per equivalent capability has dropped by at least 10x per year. I think this progress will be sustained for a few more years, and even accelerate as labs use AI more and more heavily for AI research in the pursuit of RSI. There is of course a ceiling to efficiency improvements, but I don’t think we’re close to that ceiling. 2. Progress on most unverifiable tasks remains frustratingly hard. AI inherits the same limitations (ambiguous success criteria, slow and noisy feedback) that limit human performance at tasks ranging from writing to governance. The optimization paradigm generally doesn’t result in superhuman capabilities on such tasks, because whatever proxy you optimize for gets Goodharted. For example, I recently pointed out how over-optimization explains Claude’s overly dense and ornamental writing style that’s since been dialed back. 3. RSI ends up being narrow. A consequence of the above two points — even if some version of RSI is achieved, it results in an explosion in some dimensions (efficiency, verifiable tasks) while falling short of producing something that is superintelligent in any real sense or capable of automating a broad swath of jobs in the economy. 4. More than one lab stays at the frontier. This means the market will remain competitive, and developers can’t charge high margins to compensate for rapidly falling inference prices. Any compute scarcity that allows higher margins is temporary, in my view. I also think it’s likely that most of the innovations continue to make their way into open-weight models with some lag, so open-weight models are at cost parity for inference or only slightly more expensive. Note that efficiency tricks and verifiable capabilities tend to leak much faster than unverifiable ones. Distillation could be one reason, but simply knowing that something can be sped up 10x tends to have a focusing effect on research and innovation. 5. Jevons’ paradox is not a magic bullet. If the above points hold, there are very few applications that will consume so many tokens that they could generate the astronomical amounts of revenue implied by the labs’ valuations. We’re talking about a 10x growth in tokens year after year, just to keep revenue flat given falling prices. Even today’s behemoth tasks like migration of million-line legacy codebases will likely become cheap within a few years. While new applications will open up, there won’t be orders of magnitude more new ones. As for unverifiable tasks, inference scaling rapidly hits diminishing returns. Ultimately you have to have a human in the loop who’s accountable for the result. Spending more on tokens doesn’t let customers escape that responsibility — and only makes the human review bottleneck worse. 6. What about scientific research? Science is an endless frontier, and more tokens could perpetually mean more discoveries and inventions, in theory. But the unfortunate thing about research is that researchers don’t get to capture the value it creates, which is why it requires government funding in the first place. Even if it’s true that AI could enable great scientific innovations, I don’t think AI companies will make much money off of that. I think they are currently investing in AI-for-science not as a future revenue stream but as a loss leader because they want to show the public that it could be beneficial to humanity. 7. The one escape hatch is moving “up the stack”, by which I mean quickly transitioning from selling tokens to an enterprise software like model that relies on high switching costs, deep integration into workflows, and control over systems of record. In this world, enterprises will purchase “digital workers” or pay per solution (e.g. resolved support ticket) rather than tokens, so it’s good or neutral for AI companies if inference costs fall. And open models don’t matter as much — Linux being free didn’t make most enterprises switch away from Windows, because they paid for polish, peace of mind, support, and familiarity, not the code itself. I think this is the labs’ best bet for becoming sustainable businesses (though the potential lock-in is concerning for their customers!). The problem is that the labs have many hurdles to clear to get there, most of all their mindset, and the faster they race toward RSI, the shorter the window they give themselves for this transition. The pure-play AI labs are the ones who are farthest from being able to make this transition, and AI companies with a long history in enterprise software are relatively well positioned. In short, the race to RSI ends up being a race to eliminate the labs’ main avenue for capturing the value that they create — the fate of many past transformative technologies such as railroads, airlines, and fiber. The labs’ bet is that RSI produces a durable, decisive lead that escapes commoditization. My view is that there are two different improvement loops that have been conflated, and they will come apart. The efficiency improvement loop will accelerate, but a broader improvement loop that might lead to AGI or superintelligence will require ever-increasing amounts of compute, talent, data, and real-world feedback that can only come from gradual deployment, and the money to fund all this will have to come from massive revenue growth. And the irony is that the first improvement loop is undermining exactly that revenue growth. The premise in my chain of reasoning that I’m least confident in is #5 — that Jevons’ paradox will eventually stop being able to compensate for efficiency improvements. A few months ago I was much more of a believer in continued, nearly unbounded token volume growth, but the more I’ve thought about it the more skeptical I’ve become. To better understand why the premises and outcome I've laid out are at least a possibility: * My AI-as-normal-technology essay and follow-up writings with @sayashk on our newsletter normaltech.ai/ * My ICML talk laying out my skepticism of the claim that RSI will lead to superintelligence normaltech.ai/p/what-will-be… * Ramez Naam’s review of the evidence questioning the prospects of an intelligence explosion (@ramez) noahpinion.blog/p/wheres-the… * My piece with Akash Kapur (@akashkapur) on whether AI labs will move up the stack normaltech.ai/p/up-the-stack… * Many pieces by Nathan Lambert (@natolambert) such as “I expect rapid progress but not towards general superintelligence.”