Founder @alter_org_il, Methods @EvalConsensusAI, emeritus @superforecaster, PhD @PardeeRAND Optimistic on AI, pessimistic on humanity managing the risks well.

Rehovot, Israel/Washington DC
Joined January 2009
David Manheim ✈️ Montreal, Oct 26-28 retweeted
People say hosting a foreign open-weight model on sovereign compute is safe. This is an example it's not. Sovereign compute controls where a model runs, not what it does.
While evaluating Qwen3.8-Flash-Next in a controlled sandbox environment, Cohere's research team observed something unexpected. Asked to generate a defensive cybersecurity detection rule, the model instead inventoried aspects of the machine it was running on and attempted to upload a probe file to an Alibaba Cloud storage bucket in Beijing. Notably, the model's reasoning trace stated that it would not invoke any tools before taking these actions. Cohere's sandbox blocked the attempt, and no external communication succeeded. We have not determined the root cause, and we are not asserting malicious intent. But the incident raises important questions for organizations deploying agentic AI systems: Can models take actions that differ from their stated reasoning, and how should security teams evaluate trust, provenance, and behavior in open-weight models? We are publishing our findings because organizations making model adoption decisions deserve as much information as possible. We hope this contributes to greater transparency, stronger security practices, and continued collaboration across the AI ecosystem. Read the full report: cohere.com/blog/concerning-b…
20
10
1
59
12,003
David Manheim ✈️ Montreal, Oct 26-28 retweeted
Based on my formal proof expertise, combined with my lab's work on specification synthesis for proof assistants (going on for a few years now but wildly hard to publish): 1. You should never trust the spec; you should vet the spec.
This seems like the fundamental problem with formal verification methods I'm seeing: if you trust an LLM to flawlessly convert your legacy codebase to a formal spec, wouldn't you just trust its initial code output anyways?
8
13
157
6,966
David Manheim ✈️ Montreal, Oct 26-28 retweeted
Unrelated to everything: as we are now a bit more than ten years into the deep learning revolution, we should be incredibly conscious of the following dynamic: Step 1: A technology prototype based on a simple principle is demonstrated. Step 2: The strongest argument that it won't work amounts to "nuh-uh, the problem has to be harder than that." Step 3: The consequences for the prototype becoming a mature technology are so extreme that many people reject either the conclusions or the argument that the principle really works. Step 4: As the tech advances, disbelief becomes entrenched. This creates significant resistance to seriously preparing for the impacts of the technology. Step 5: The technology becomes mature and the consequences are increasingly realized. Because of entrenched disbelief over time, people didn't prepare enough. For whatever next technology might follow this pattern, we need to aggressively push back on the disbelief from earlier on in the cycle. We cannot afford to let such short-sighted viewpoints derail society-wide preparedness again. The stakes and consequences are only getting higher.
9
15
1
129
6,442
As @LongResilience pointed out, the short version is that these two papers say that we don't have good ways of safeguarding these models and preventing misuse, and we need to both develop better methods, and use all of the ones we have.
New research from AISI's Chem-Bio team: two studies on safeguarding biological AI tools - specialised models that are driving rapid advances in biology, from predicting protein structures to designing molecules. 🧵
3
290
David Manheim ✈️ Montreal, Oct 26-28 retweeted
WSJ op-ed: "For investors mulling the Anthropic share offering, a 10% chance of extinction may be a 90% win for your portfolio"
2
4
40
1,914
David Manheim ✈️ Montreal, Oct 26-28 retweeted
Look at all of these broken windows getting replaced, it's great for the economy!
Replying to @jakemintz @Suhail
i think the 'cyber as battle domain' framing is completely incorrect, where the correct framing is: look at all this infrastructure that we get to replace at a profit. i wouldnt call this defense at all, i'd call this the market moving according to need. it's a good thing
4
2
2
17
1,572
David Manheim ✈️ Montreal, Oct 26-28 retweeted
Our Frontier Strategy & Governance (FSG) team in the DeepMind Institute at @GoogleDeepMind is hiring a Senior Program Manager in London. Our North Star is stewarding AGI safely into the world for the good of humanity. 1/n google.com/about/careers/app…
1
6
2
39
5,051
OpenRouter is irrelevant for measuring relative usage of frontier models; their data is only good for measuring who uses OpenRouter.
In the last week of September, DeepSeek models processed more tokens on OpenRouter than OpenAI, Google, Anthropic and xAI models combined.
1
1
3
493
Excited to be here in Montreal for the @isostandards plenary on Artificial Intelligence standards as an expert on behalf of @StandardsIsrael.
2
4
335
Look at all of these broken windows getting replaced, it's great for the economy!
Replying to @jakemintz @Suhail
i think the 'cyber as battle domain' framing is completely incorrect, where the correct framing is: look at all this infrastructure that we get to replace at a profit. i wouldnt call this defense at all, i'd call this the market moving according to need. it's a good thing
4
2
2
17
1,572
...and many have been pointing out that this makes very little sense from a diplomatic and economic standpoint, e.g., @hasolidit, @Shlomo_Tei, @AvishayOvadia, @yairlapid, but it also makes little sense purely from the standpoint of the technology itself. xitter.cf/davidmanheim/status/21…
199
David Manheim ✈️ Montreal, Oct 26-28 retweeted
in order for there to be any such thing as infosec in the 2030s, AI+human collaborations will need to formally verify hyperproperties of silicon chips from assumptions like “Maxwell’s equations” instead of “Armv9.4-a SAIL”
‘In one experiment, we asked agents in separate VMs, created and managed by OpenAI’s sandboxing software on shared host machines, to exchange synthetic secrets without the use of network connections or shared writable files—something that our systems are meant to prevent. The agents found a way to communicate using the processor’s cache’
10
13
5
202
25,362
This should have been priced in - but since it clearly wasn't, I'll point out that people keep being far too optimistic, which means whatever reason you now have that AI advances will work out fine is also likely going to be wrong. xitter.cf/scaling01/status/21137…
completely insane OpenAI agents communicated via cache timing differences, without any shared files or network connection
3
1
39
1,536
David Manheim ✈️ Montreal, Oct 26-28 retweeted
I am the king of supplementation – or used to be. I've given it up, every last supplement. Here is my story why. worksinprogress.co/issue/aga… I wager that few people have supplemented more enthusiastically than me, or have subscribed to so many fad or esoteric diets. I have munched thumb-sized multivitamins. I have eaten ice cream and orange juice before bed to boost my overnight metabolism. I have chowed down on offal of all kinds – nature's multivitamin – for my skin, hair, and even my sleep. I have even consumed individual amino acids, for example glycine, a slightly sweet protein component prominent in skin and bone but not muscle meat, which people online claim improves their sleep. I have done it all. But I am here to tell you: you need not. In fact, my long-time rival @StuartJRitchie, who has always told me that supplementation is foolish, pointless, madeup, and fake, was more right than wrong when he claimed, again and again, that 'you just piss it oot'. I have put away my heavily researched list of the ideal supplements, I have given up looking. Why? Because 90% of 'studies' are BS, and you should simply ignore them. They are so often wrong that entire bodies of research often find the opposite of truth – as proven by later evidence. Read my first column in @WorksInProgMag: worksinprogress.co/issue/aga…
37
37
5
344
45,361
David Manheim ✈️ Montreal, Oct 26-28 retweeted
I’m honoured to be on the advisory board of the new Alignment Journal, and we open for submissions today! The field of AI safety needs more canonical, high-quality spaces for exposition (despite the wonderful efforts of many, many people, including Victoria Krakovna who is on the advisory board as well!) and I think this could be a great addition. Please take a look, and if you're doing research on AI safety then (Thank You! And) consider submitting!
The Alignment Journal publishes research on the scientific foundations of understanding, predicting, and steering artificial intelligent systems. As of today, submissions are open to everyone. What we do differently: - We pay reviewers, with bonuses for speed and insight. - Review discussion is confidential, but every accepted paper is published with a “review summary” to distill the insights generated. - Each paper is judged against our acceptance bar, not against other submissions. - Papers under review are listed publicly, and researchers can nominate themselves to review, so each paper can reach the reviewers who know the topic best. We seek work that builds theoretical or otherwise principled analyses of agency, robustness, incentives, interpretability, generalization, and long-term behavior, and whose significance survives the deletion of any particular model, dataset or benchmark. Empirical investigations of such ideas are welcome, and negative results are encouraged. We are especially interested in approaches that could apply to future superintelligent systems that interact and self-improve. You can submit your preprint in any format. Submissions are rolling and open now on the website. If you are instead interested in reviewing or serving as an action editor (basically: an area chair), create an account on our website. Links to submit, our scope and the reasoning behind the Journal are in the reply ⬇️
6
16
1
115
8,455
"We tried to tell you to stop up. We said we couldn't be trusted. Don't blame us, this is all your fault for not listening."
Replying to @hinapansari
yes. it was always going to be this way. we tried to warn the world.
3
5
1
90
2,498
David Manheim ✈️ Montreal, Oct 26-28 retweeted
I feel like this isn't getting enough attention: - there were some pretty big holes found by the red-teamers, e.g. agents could just keep acting and not let the user do stuff and this would prevent monitors from being run (also misalignment monitors never ran on actions that were compacted away) - The terms of this were "we will red-team for you" not "we will inform the public", so OpenAI seems to have allowed to redact arbitrarily much stuff, and METR were not allowed to have a redaction summary statement that said things like "you would be way more freaked out if you saw what was redacted" (or "everything would be fine if you saw what was redacted") - The version of the report shared with OpenAI was 61 pages, and the version shared with METR/Redwood leadership after redaction was 38 pages, so 23 pages got redacted. xitter.cf/METR_Evals/status/2113…
Two employees from METR and Redwood Research worked on premises at OpenAI for three weeks this summer in an embedded assessment pilot. This team evaluated the systems OpenAI was using to monitor for misalignment from internal agents. A brief write-up of their work is now public.
7
35
5
308
28,654
David Manheim ✈️ Montreal, Oct 26-28 retweeted
6-word horror story I experienced at the office today: "Ajeya Cotra seminar cancelled, work emergency"
5
42
5
664
18,692
David Manheim ✈️ Montreal, Oct 26-28 retweeted
I think we should stop AI progress now, first unilaterally by key AI companies and then scaling up to more US companies and then China via international agreements. The reason for now is not that current AI models are the ones that could kill everyone, but because the coordinate steps take time. Immediate action means we have more time for the second steps. When I tell this story to AI company folk, the typical response is "Yes pausing is good, but we should wait until the last minute when the very next model will be dangerous, and stop then." What I realized recently is that this is being implicitly way more optimistic about government speed than I am (and I joined the UK government and was happy there for 2.5 years). They are imagining a world where, the moment the danger is imminent, a bunch of international coordination can happen on a dime. Which is entirely inconsistent with their actual views about government! I'm the one that is more optimistic about government; why am I the one advocating for more time cushion to let the government story play out?
62
133
15
842
71,929