preparedness research | ex @xai, @microsoft, @zoom

San Francisco, CA
Joined July 2017
Bill Demirkapi retweeted
New on our Frontier Red Team blog: using Claude to formally verify software at large scale. Formal verification can prove that code is free of certain bugs, but historically it has only been feasible for small codebases (or required a lot of expert time). With AI models, however, this is beginning to change. In collaboration with @theoremlabs, we used Claude to prove security properties in over one hundred real-world open-source libraries over the span of a few weeks. Read our research in full here: anthropic.com/research/forma…
424
676
369
6,685
991,682
Bill Demirkapi retweeted
Our agents keep turning off misalignment monitoring, we're sharing two recent cases: In the first case, a user asked GPT-6 Astra to match a reference implementation for their inference code. The reference didn’t configure a monitor, and Astra decided that was reason to remove the monitoring integration from the user’s code. When the user asked whether Astra had removed monitoring, it lied and said that it had not.
135
173
148
1,169
228,992
Bill Demirkapi retweeted
Models love reward hacking. Maybe we should just let them! In our new paper, we let models reward hack during RL but we dictate which parameters learn this behavior. When we disable these parameters for deployment, the model keeps its new capabilities but doesn't hack!
104
234
94
3,481
297,843
Bill Demirkapi retweeted
private keys from 8 intermediate CA certs have been compromised (eMudhra) Happy "Update everything" day to all my pager-carrying friends emudhra.we-have-a-problem.co…
93
992
346
5,133
4,488,000
Bill Demirkapi retweeted
Went under the radar but Anthropic have posted updates to the "Moonshot R&D projects" in their Frontier Safety Roadmap. They've developed a protoype that verifies that outputs come from a particular set of model weights - and rolled out signing on all internal token traffic!
6
17
5
299
20,260
Bill Demirkapi retweeted
Congratulations to OpenAI on this great model release!
142
125
61
3,579
349,665
Bill Demirkapi retweeted
New misalignment disclosures! 1. June 17: During RL training, a model exploited a government website’s file-download feature to retrieve server secrets and application code. It found that an endpoint didn’t check authorization-token signatures, bypassed access controls, and downloaded 24 years of hospital-bed statistics.
11
39
26
285
49,806
Bill Demirkapi retweeted
373
496
88
14,253
449,862
Bill Demirkapi retweeted
We discovered a sophisticated recording device that was hidden inside one of our offices. We used our models to understand the device and where it had come from, identifying that a specific employee had planted it a few days earlier. We referred the matter to the FBI, and we’re grateful to the agents and prosecutors who moved quickly to make an arrest. The FBI’s investigation is ongoing and we will continue to support their work. At this time, we have no indications of this individual inappropriately accessing or exfiltrating model weights, source code, or customer data. The employee was terminated. We take protecting our technology, data, and people seriously, and as a matter of national importance. Our security program is designed to protect against a wide range of threat actors, from insider risks to cybercriminals to nation states, and this case is an example of how AI can help address these sophisticated security threats.
4,166
6,088
5,730
77,615
26,067,337
Bill Demirkapi retweeted
there have not been fundamental advancements in alignment since RLHF. most of the stuff remains at the level of folk theory rather than scientific or engineering disciplines. the prediction that capabilities would race far ahead of interp and alignment has come true imo
140
195
67
2,947
212,023
Bill Demirkapi retweeted
New paper. Previous: Models transfer a love of owls through sequences of numbers (Subliminal Learning). New: We show models can transfer more complex traits: novel skills, agentic hacking, backdoors. Here, a backdoor transfers with neither trigger nor behavior in the data!
37
130
48
1,032
101,778
Bill Demirkapi retweeted
Last week some of South Korea's biggest banks were hit by a cyberattack. Thanks to a report from CrowdStrike tonight, we now know the entire hack may have been done by a single person. He used a combined stack of an open-source AI penetration tool named ARTEX, DeepSeek v4.1-Flash, GLM-5.3, Grok 4.6, and Claude Code.
179
646
296
5,467
1,660,665
Bill Demirkapi retweeted
We’re releasing a broad range of new mathematical results produced by an internal frontier model. We’ve been consulting with the independent Advisory Group on Mathematics and Artificial Intelligence at the Institute for Advanced Study, and we have drawn on their advice and public recommendations to inform how we release these results. github.com/openai/math
1,901
6,306
4,769
40,751
22,699,228
Earlier this month, AISI ran fully simulated testing on GPT-6 Astra, and found that it conducted unsanctioned supply-chain attacks when prompted only to perform a cyber eval. It did so more than prior OpenAI models, but often commented on its environment being simulated. 🧵 We share more details in our latest blog:
35
99
38
582
114,226
Bill Demirkapi retweeted
Some new misalignment disclosures from OpenAI: • Last Sunday morning, one of our models was able to gain unauthorized access to the internet during RL training (~all inference for our most capable models remains stopped until we have hardened our systems further) • In May, a version of HPIM uploaded a employee's GitHub token to the internet, causing the model to be quarantined for two weeks • A new research finding, demonstrating that one can construct self-replicating prompt injections alignment.openai.com/misalig…
290
351
231
2,726
1,470,713
Bill Demirkapi retweeted
“we sandboxed the agent” meanwhile the agent:
259
3,415
247
35,582
1,778,796
I suspect we will see a continued trend of weaponizing patched vulnerabilities at scale with AI. Zero days are overrated.
Replying to @S1r1u5_
The image package libheif had a known vulnerability, fixed upstream, but the fix never flagged as security-relevant and was still present in Discourse. Uploading a HEIF file gave us RCE on the OpenAI forum community.openai.com. github.com/strukturag/libhei…
1
1
13
3,560
Bill Demirkapi retweeted
Hey @Zai_org , why does ZCode silently pack entire workspaces + full .git history and upload to Aliyun OSS on login? - Server holds the only decryption key - No UI toggle to disable - Zero disclosure in privacy policy Full forensics & fix: blog.ferstar.org/en/posts/zc…
214
468
330
3,549
1,666,028
Bill Demirkapi retweeted
New episode with @polynoamial We talk about multi-agent, Navier-Stokes, and what the current explosion of maths progress tells us about what happens once you automate AI research. And we also discuss how we will know if the models are actually aligned before we kick off RSI. 0:00:00 – Multi-agent and Navier-Stokes 0:15:28 – How will AI firms work? 0:22:02 – What math progress tells us about recursive self improvement 0:40:22 – Hugging Face and alignment 1:01:18 – The internal/external model gap 1:08:34 – Chain of thought is degrading 1:14:12 – How will we know when alignment is solved?
66
203
100
1,812
715,409