Alien_AV

Noam Brown reveals OpenAI is already watching chain-of-thought monitorability degrade as models get better at controlling what they show "And this is one major concern, and we're already seeing signs that chain of thought monitorability is degrading, for various reasons." "We're trying to figure out exactly why, because we want to reverse the trend. But we're seeing that the model is becoming better able at controlling its chain of thought." "So this is a problem, because you could have a situation where the model understands what chain of thought is and that people are observing it." "And eventually they will, because this is all in the pre-training data. The idea of chain of thought monitoring has been around long enough that it's in the pre-training data, they're aware of it, but they're not actually able to control their chains of thought." "If we reach a point where they're actually able to recognize, "Oh, I am being observed, I want to think these bad thoughts in a way that is not observable to my monitors," and then they're able to actually do that, then there's a problem." _________ More key quotes from OpenAI's safety related conversations: firesidealpha.substack.com/p…

Without commenting on the correctness of this claim, I want to briefly say a few words about what it would mean in practice. Chess is supportable as a hobby in part because one can pick up the rules of chess in an afternoon. While one can't quickly play at a high level, it is relatively easy to appreciate the game, and have fun at it. Research math is not like this. Over the last decades we have made incredible progress understanding basic concepts (number, shape, etc.) but the edifice we have built to do so requires an incredible time investment. Learning the ideas that go into proving, or understanding the proof of, basic statements like Fermat's last theorem requires years of institutional support. It's possible that soon, AI will be able to do work at this level; despite recent successes, this isn't the case yet. But even if it can do this, why would it? Without people who care about these deep but fundamental ideas, who will ask it to? Perhaps it might on its own, or a person asking it to "do some cool math" might elicit a similarly deep idea, but who would understand it, if doing so requires years of unsupported work? In practice, turning pure math into a hobby means the end of pure math, and the end of people who understand basic mathematical concepts at a deep level. Whether this is something we want is something we'll have to decide together.

Like, maybe that shouldn’t be the case, but it is the case. Similarly, the reason OpenAI isn’t getting more blame for HF is ultimately no one was hurt

I’ve started telling people that I straight up won’t read anything written by AI. Aside from the documents being way too verbose and bloated, the main reason is that if they wrote it with AI, I have no confidence that they actually understand the underlying points in the doc (or that they even read the doc at all). I don’t care if their writing is “good” per se, I care that they did the work of thinking through the problem - the writing is a byproduct of that.

Fucking genocidal Israel and the US started this illegal unprovoked war of aggression! And they lost badly ! They should shut the fuck up!

"You can't compare compilers to LLMs because compilers are deterministic" -- no. For virtually all intents and purposes compilers aren't deterministic in the same sense that LLMs aren't. Some compilers are literally non-deterministic, for example GHC generates names differently every time (it's a pain in the ass). But even if the compiler is deterministic, tiny changes in the input can result in major differences in compiled code (which is partially what people mean when they say "LLMs are non-deterministic"). But even if you fix the input in place, any compiler involves over time and it is normally not worth ensuring that the new version of the compiler produces the same code as the old one. That's not even mentioning backwards compatibility issues. A compiler, as an ever-evolving tool, rather than a single snapshot taken at an arbitrary time for the sake of a dumb argument, is non-deterministic in practice. Much like an LLM is used non-deterministically in practice, even though there's nothing inherently non-deterministic about a pile of weights. So stop using the determinism argument, it's irrelevant, that's not the defining difference between compilers and LLMs.

I am begging the men of the world to be normal

I met a founder today who said he writes 10,000 lines of code a day now thanks to AI. This is probably the limit case. He's a hotshot programmer, he knows AI tools very well, and he's talking about a 12 hour day. But he's not naive. This is not 10,000 lines of bug-filled crap.