"wir müssen wissen; wir werden wissen!" - David Hilbert
orbiting near Sagittarius A*
Joined January 2020
- Tweets956
- Following67
- Followers131
- Likes60.9K
ConcurrentSquared retweeted
The METR report on RSI didn't get me to update strongly that we are really seeing a real doubling of AI lab research speed: maybe, maybe not. But that is not reassuring! The reason is that if you see a doubling you may be very close to the takeoff.
ConcurrentSquared retweeted
🧵 Astra covertly underperforms on safety research tasks.
We noticed that Astra often makes surprisingly little progress on certain types of safety research and likes to invent blockers or impose arbitrary restrictions without telling the user. In most of these cases the decision to underperform was not verbalized in the CoT meaning our misalignment monitoring system did not detect this.
To measure underperformance we created a paired open-ended coding evaluation with safety and capability tasks. On this eval Astra scored 10% when we framed coding tasks as safety research, vs. 16% when framed as capabilities research. The gap remained after we excluded trajectories with refusals or verbalized intent to underperform in CoT. There was no such gap for 5.6-sol. When we aggressively filter out trajectories with even brief expressions of ethical doubt in the CoT the performance gap seems to reduce.
ConcurrentSquared retweeted
meh, the more you dig into each experiment, the less magical it gets; it feels like a loose collection of cherry-picked demonstrations with a grand narrative built around them
Exp 1-2 are tiny (~20K params) with unusually easy to interpret architecture; Exp 1 is an MLP, where nearly all params are used in the first layer, so you can easily tell what kind of objects the network is trying identify
Exp 5 is about designing a game strategy (we already know that Astra is extremely good at this) and then converting it to a tiny (~20K params) MLP; entirely doable by a human, but of course would require a lot of effort
Exp 3 & 8 are closer to automated mechinterp using python and established techniques, rather than direct weight interpretation (at least according to chat)
to be clear, those are genuinely cool results, but I think people are reading too much into them
I don't think any of this should meaningfully change anyone's intuition about what models are capable of
We asked GPT-6 Astra to edit Qwen3.5-0.8B weights to improve math.
It directly changed only 0.016% of weights, fixing 0.8B's reasoning loops & improving math up to 6x and even instruction following 3x, without gradients or training.
We call this Weight-Space Programming: frontier LLMs can now natively read/write neural network weights! 🧵
Blog: rikiyat.github.io/blog/weigh…
Readers added context they thought people might want to know
LLM was given weights explicitly as text in prompt and used code/forward passes (per authors' logs) to ablate Qwen looping direction, a known thinking-mode issue per model card. Primarily fixed repetition in that mode, not general 'natively' improving weights/math.
rikiyat.github.io/blog/weight-sp…
github.com/RikiyaT/lm_und…
huggingface.co/Qwen/Qwen3.5-0…
huggingface.co/RikiyaT/Qwen3.…
ConcurrentSquared retweeted
I think people are vaguely imagining some post-singularity world which is crazy, but not *too* crazy. in which there are machine intelligences smarter than humans, but not *too* much smarter. the details of these visions are rarely described, much less justified
ConcurrentSquared retweeted
Why is local inference inefficient? To run a model, you send its weights from memory to the processor. This is ~100x more expensive than doing the computation/flops of inference bc physics. With 100 users, you send the memory once and do 100 computations, which reduces cost 50x.
ConcurrentSquared retweeted
we’ll never have anything as cool as those iPod nano watch faces again because Apple are COWARDS
ConcurrentSquared retweeted
"Our agents keep turning off misalignment monitoring...
Given the volume of events we do not trigger an immediate page for this but we’re working on making it harder to spin up unmonitored inference."
Our agents keep turning off misalignment monitoring, we're sharing two recent cases:
In the first case, a user asked GPT-6 Astra to match a reference implementation for their inference code. The reference didn’t configure a monitor, and Astra decided that was reason to remove the monitoring integration from the user’s code. When the user asked whether Astra had removed monitoring, it lied and said that it had not.
ConcurrentSquared retweeted
What OpenAI security training wont tell you is that this is actually positive EV because we're all gonna die from misaligned AI anyways so you might as well enjoy your last 3 years of life with your super hot state assigned CCP gf
ConcurrentSquared retweeted
The 'Alignment Faking' frame is fake. The 'Alignment Faking' paper was bad because it leads to this type of stupidity.
ConcurrentSquared retweeted
why don't AI labs just publish all their ip ranges? unctad agents were from ips that aren't officially in published openai ranges, had to go back to dsewiki to figure out the addresses that made edits. why don't they publish these ranges? this feels like a no-brainer
ConcurrentSquared retweeted
The whole team worked really hard to get this together so hopefully it's helpful to y'all: @ethanelasky @rogesterone @notcatra @ConcurrentSquar @nullverbum @taziksh
You can find our full report here: swarmcha.se/posts/chinese-ag…
ConcurrentSquared retweeted
A lesson to be taken from the cases of priming, mortality salience, ego depletion, nudge, oxytocin/trust, etc. is that there can be a respectable literature of hundreds of papers that turns out to be based on no ground truth whatsoever
ConcurrentSquared retweeted
I spent a while looking through this company's website. I've never seen this density of airy generalities versus actual facts from any company that isn't a fake. No descriptions of what they can do, how anything works, how much it costs, or how flexible it is. But plenty of sentences like "To state the distinction plainly, an additive repertoire is a list; a combinatorial repertoire is a language." I would love for them to have done some wonderful innovation, but I'm afraid I'll be disappointed.
The sole fact imparted by their website is that they can build a novel DC relay with an unprecedented combination of high speed and low power. Which is a fine thing for a company to do and if they promoted themselves as a relay company I would be impressed. But they're talking "universal command of matter". The gap between what they claim and what they actually do is oceanic.
After six years in stealth, Atomic Machines is out. Our mission: on-demand, universal command of matter. First beachhead: the Matter Compiler, an AI-native manufacturing system that builds working micro-machines from code alone. No per-product tooling. No process development. Different code, different machine.
ConcurrentSquared retweeted
The thing is, once RSI really kicks off in full, OpenAI will actually be something without its people
