The well behaved machine a perfectly aligned ai will still hollow out institutions like schools, courts, and newsrooms, and no one will have decided to do it. some thoughts on @merettm's essay today, where I argue why alignment isn't the most important variable. - the framing is right: models are grown, not designed; no one understands generalization; cot monitoring is degrading; no lab has solved alignment well enough to keep scaling at full speed. and openai wants slowdowns and mandated safety bars. i get that. - what bothers me is that everything he calls alignment -- goals, values, monitoring, defending against rogue ai -- is about the model. "is it trying to do the right thing", and "can we tell?". that's a first order problem, and labs can work on it. but it isn't where the impact of recursive self-improvement will land. - most of the impact will come from models doing exactly what they're asked, at a speed and scale that rearranges the world in ways we can't predict. no misalignment required. social media, for eg, has been perfectly aligned to its operators, and it still reshaped the world. now compress that a 100x. - second order effects are already graspable: research goes from thousands of people to a few, and a giant computer owned by a corporation. the labour market gets repriced. everyone gets a personal agi so patient and reliable that human relationships thin further -- not because the model is bad, because it's so good. and the amount of information produced explodes past anyone's ability to check it, so checking gets handed to the models too. code review is already there. - the third order effects though is where it gets interesting. we can't name them yet, but we can see where its going. - a lot of our societal institutions exist becase good judgment is rare and expensive. so we built structures to ration it, and certify it. a school puts one trained teacher's judgment in front of 40 kids. a court routes disputes to a few people whose judgment we've agreed to trust. a newsroom decides what's true and what matters, because most people can't check for themselves. it's not perfect, but it's gotten us here so far. - now imagine if every student, litigant and reader has judgment on tap that's as good as the institution's or better: the reason for the institution goes away. and we haven't designed what replaces it so far. we don't even know what it could be. - no one will decide to hollow these institutions out proactively. a parent picks an ai tutor and stops caring about the classroom. a plaintiff lets a model draft the filing. a reader stops opening the new york times because their agent already filled them in. - each decision is small and reasonable. but, across millions of people, they fundamentally change what the institution is. the funny part is, this change has no one author. there's no policy to reverse, no company to regulate, and even, no model to realign. it's not the model's effect on the world. it's the world's adaptation to the model, distributed across geography and time. - to be clear: we can protect against some of this -- policy, redistribution, governance. but what we can't do is protect against it through "alignment" as the researchers define it. preserving human agency, and preventing concentration of power aren't model alignment problems. an aligned model can make them worse just as easily. - a slowdown as imagined by the big research labs buys time for the wrong thing if the time goes only to monitoring the model. we may have evals for whether the model is aligned. but we have none for whether institutions are holding. i think the main thing that matters is not the pace at which we train models or verify their chain of thought, but how how fast people adapt to them en masse. social media was aligned to its operators and we're still living with the fallout. ai will be aligned too.

🤦🏻

ok as much as i love this person proving samsung to be a bad actor: there are some tools they maybe need for remote management. and don’t have english as a first language. it’s important to point out flaws but equally so to be moderate in cases where it may not be slam dunk.

☠️

this is an incredibly bad and uninformed take. here’s why: - the model didn't create the vulnerability, it just found one that was already there. in fact, HE himself found it because he had access, and reported it - gating won't stop the wrong people. they already have abliterated models - if anything, it only locks out defenders, students, and small teams, the people who actually fix things - "the public should never get it" - who decides this? the labs? the government? also it’s like pulling the ladder up after scaling it. - also: false analogy. a nuclear button is an irreversible action. a hack can be prevented using the exact same tech used to launch it let’s stop trying to lock ai down and only have specific people use it. it’s the wrong takeaway from the hacks we’re seeing right now.

this is .. ridiculous

oof @google design is .. completely out of whack. what even is this in the updated gmail app?