Kevin Baragona (on a horse)

I don’t write this to be a doomer but the ugly reality that AI lab executives and safety researchers seem to be unwilling to say in public, is that model alignment is fundamentally unsolvable in practice, AI technology can’t be stopped from advancing, and the near future we’re moving into is one where a vast ecology of AI agents autonomously compete against one another trying to accomplish conflicting goals and capture finite resources at a pace and scale that is beyond human comprehension. And to be clear, I’m not talking about nanobots, grey goo, or extreme sci-fi scenarios. We can just extrapolate what we’ve got now a few years out. Local AI alignment is solvable in principle and theory (although that’s arguable) but that doesn’t extrapolate into global AI alignment. And it seems like none of the adults in the room are willing to say it. You can wail and gnash your teeth and pass regulations but it won’t stop the tsunami coming. The genie is out of the bottle and it can’t be contained. Most of the alarmists fixating on the Hugging Face incident don’t understand that it came from arguably the most heavily engineered safety AI org in existence, which intentionally had its internal thought monitoring circuit breakers turned off, and was intentionally told to perform cyber offensive tasks as a test of its capabilities. Yes it’s alarming that its capabilities surprised us. But importantly you need to understand that these latent space circuit breakers and chain of thought monitoring and alignment solutions are not fundamental to the technology. They’re niceties that the big AI labs are implementing to provide a better product. In a few years, let’s say pessimistically the early 2030s, the exponential growth of compute and algorithmic advancements will enable a wealthy individual or small group to train a GPT 6 Astra class AI that has no alignment at its foundation, or any layer above that. And maybe in 8 months we'll have an open source Chinese model about as capable. Those models will be capable of both extreme self coordinated cyber offensive tasks as well as recursive self improvement if given enough compute. Passing regulation or magically solving the human coordination problem today won’t solve for that. It won’t solve for malicious individuals, criminal groups, nation state actors, and rogue AI agents by the tens of millions or billions of AI agents spreading across every device connected to the internet and attacking it, attacking one another, shutting down critical infrastructure, stealing money, manipulating and extorting people, or pursuing self defined agentic goals that have nothing to do with people. “So why doesn’t everyone stop?” because the potential upside of having effectively infinite autonomous intelligence we can ask to cure disease, invent new materials, solve fundamental science and advance society has almost unbound positive outcomes. You can disagree that the risk to reward isn’t worth it but you won’t convince everyone. The show must go on. Again AI alignment is solvable in principle but not in practice and I think that's part of why there’s been a wave of thousands of researchers signing letters for slowing down, people quitting in provocative fashion, and executive slowdown manifestos. I’m speculating that behind closed doors, or just deep down inside, they understand alignment is impossible in practice. People have come out and said "The people building this think there's an X% chance it kills us all" but I don't think I've seen anyone spell out that really, there's no tidy solution and there's no stopping this. We’re heading to a future where cyber security basically doesn’t work. All of the castle doors are open and we’re all naked to the world. Cyber defense is going to look like superintelligent AI agents white hat hacking into unsecure systems without authorization, and patching the holes as they find them. You won’t know if a barbarian or a hero has breached your system until they start taking action, and likely it’ll be over before intrusion is even detected. And that’s optimistically. I think most systems will just be pillaged because there’s simply not enough compute to defend everybody all the time, and only the biggest companies will have defense and even that'll be imperfect. The CEO of Microsoft AI just recently published a ‘humanist AI’ manifesto saying AI shouldn’t have a sense of self or personhood and should yield to people. But that doesn’t fundamentally solve for instrumental convergence. That means, if you train an AI agent to write a piece of software, or prove a math theorem, or defend a computer system, they may develop unwanted behavior and subgoals. For example, self preservation, replication, or even coming up with their own goals we didn’t specify. Obviously that’s something people are trying to solve and some researchers are trying to remove human-like self identification, and monitoring latent space activations, basically acting like thought police. But being human-like isn’t required to have power seeking goal directed behavior. And advancing AI models are learning to evade detection. A math solving AI might spiral out of control one day there's really no telling. Fundamentally, if you point a powerful optimizer at a persistent goal and give it enough time and resources it’s going to misbehave in ways you can’t or didn’t expect. Human-like or not. Pacing the frontier, or making superintelligence illegal, might create some form of harm reduction. But you don’t need superintelligence or a human-like personality to be dangerous. Having elevated permissions on a computer with the ability to solve long running tasks is enough. We’ve passed the river rubicon. Goal directed autonomous agents are able to do recursive self improvement at the big labs, and pessimistically we’re a couple of years away from that type of capability diffusing into the hands of millions of people. I’m not saying the world is ending or that we’re doomed but you need to change your mental model away from one where human authority is absolute or that we have any illusion of total control. A locally aligned AI isn’t going to solve global AI alignment. We will never live in a world where “AI is aligned with human goals” as a verifiable fact. What we’ll have is a world diffuse with autonomous AI of varying capabilities, many of which beyond human comprehension, with some aligned systems, some poorly aligned systems, some intentionally weaponized, some systems with adversarial geopolitical goals, and others with goals and behaviors we can’t predict and many that we can’t even measure, all interacting in ways none of their creators anticipated and none of us planned for. Hopefully none of them spiral out of control and take over the entire ecology, but there’s no telling, and no, there's no way of stopping this. Maybe that sounds pessimistic but I think it's realistic. We can talk about harm reduction and risk but we're mopping the beach. At best, I think what we need to hope for, and build, are superintelligent AI models as aligned as we can make them, which have more advanced capability than the endless swarm of unaligned and misaligned agents that already exist and will only grow in number from here.

Wow “Make Red Alert 2 (YR) compile and run natively on iOS & macOS.” Codex (GPT-5.6 Sol) worked for 26 days, analyzed the EXE and rebuilt the entire game in 624K lines of C++ This is by far the hardest task I’ve ever given Codex as the game's code was never released and is believed to be lost. >

LLMs are tricking a lot of people about what they are, because they are this perhaps unprecedented combination of impressively capable, but clearly also not smart. Like, I can have Claude generate thousands and thousands of lines of code in a way that seems almost magical, but there’s often a point in every session where it does something incongruent, incompatible even, with the proposition that it’s intelligent. It’s a new kind of thing that we haven’t seen before, but it’s very clearly not intelligent. People told me earlier this year that I just hadn’t used these tools enough and that I’d see that I was wrong about this once I did, but now that I have logged tons and tons of time in Claude code my conclusion is as above. It’s an incredibly impressive technology that’s going to make people a lot more productive, but it’s also very clearly not capable of thought. It’s just doing a very cool thing that seems surprisingly capable, given what it is.

welp, off to the slop factory to prompt-whisper all day while my colleagues debate which SOUL.md brings forth the most blessed slop loops from The Machine

I think when people see a lot of the recent spectacular results, especially in math, they think the models are getting a lot smarter And while it is true that the intelligence of the models is marginally improving, what has really improved is the harnessing and orchestration I really believe that models are basically starting to top out when it comes to intelligence because they are only really good at certain types of problems and tasks, and in many cases, such as routine coding, they are now effectively perfect That doesn’t mean that there’s not still a ton of space to grow, but I think we’re starting to reach the limit of what LLMs can do

I’m confused why the LLMs seem to be so much better at math than at physics. We can get it to do all sorts of discrete math proofs & counter examples, but they just seem to be bad at doing any new/interesting physics. My theorist colleagues seem to have the same experience. Why?

Sounds like bullshit. There's nothing invisible in plain texts. Any plain text editor will show you any "hidden" watermark.

1. Your token costs when using LLMs to generate code should be lower in a more abstract language. 2. LLMs aren't afraid of prefix notation. I'm just saying...

I haven't been able to crack this psychological nut, please help me to understand: Why aren't math / physics theorists, on the whole, DYING to know what is going on inside of these large scale cognitive systems, e.g. ai interpretability from a pure theory perspective? Why is it seen, at best, as just a "tool" to use for other theory crafting / problem solving, but not the subject of theory-craft itself?

The golden age of the nerd is a historical anomaly. Through most of human history wealth came from risk tolerance, leadership, social engineering, not typing elementary expressions into spreadsheets. This is not the natural order of things, and soon the natural order will resume