James Fox (@James_D_Fox) profile picture

Technical AI Safety Program lead at Schmidt Sciences (AI Institute) | previously CS DPhil at the University of Oxford and @aims_oxford

Joined February 2019
James Fox retweeted
Announcing a major milestone achieved by my team @GoogleDeepMind: we have proven key security properties of the machine code underlying the Signal messaging protocol. This is the first time the actual production code of a widely deployed protocol has been formally verified – by any method, with or without AI – and at this depth. More than a research breakthrough, it’s a prime example of how frontier AI can be a force multiplier to counter the speed and scale of modern cyber threats. Formal verification offers a higher standard of security; however, it’s extremely complex and time consuming, which is why in many cases it’s considered to be impractical. Similar efforts could take teams decades of combined effort. We did it in a month’s time. Using AlphaProof Code, Google DeepMind's new agentic framework for formal verification that runs on LLMs including Gemini Carbon, we verified the Signal protocol down to the actual code running on devices.  This milestone lays the groundwork for a future in which vital systems, from medical device firmware to power grid software, ship with mathematical proof that their protections work. Reaching that future will take continued research from our community. In that spirit, we're open-sourcing our formal specifications and verification artifacts so others can build on this work. On a personal note, this moment is of deep significance to me. I started my own research journey more than two decades ago as an intern working on checking the consistency of complex software systems. Seeing this grand challenge made practical with modern AI is extraordinary. I'm extremely proud of my team members: Swarat Chaudhuri, Andrew Ferraiuolo, Anton Kovsharov, Tej Chajed, Sam Schlesinger,  Karn Seth, Jonathan Katz, and Charles Sutton for taking on this grand challenge and delivering, and grateful to the Signal team for their partnership. deepmind.google/blog/formal-…
232
699
236
5,838
605,435
If you've noticed a distinct dialect emerging in frontier AI models and struggled to understand what they're talking about, you're not crazy! We turned to psycholinguistics to investigate what makes the writing style so weird, and why this might be happening 🚨New preprint! 🧵
29
138
34
915
94,534
James Fox retweeted
The largest math proof in human history is now computer-verified. Today, Axiom announces a complete Lean auto-formalization of the Classification of Finite Simple Groups, the "periodic table of symmetry." About 15,000 pages, built by over 100 mathematicians across generations.
425
1,303
656
7,168
3,751,337
i feel so excited for econ majors these days — there’s so much alpha in knowing the discipline well and applying it to ai alignment seems important to study - game theory - collective action problems - invisible incentives to better understand - swarm behavior - revealed preferences - how the invisible hand guides agent behavior more than individual alignment what are ppl reading in the overlap of these two domains?
17
35
7
412
66,690
I am okay with Kimi 3.1 deliberately leaving out top-end offensive cyber capabilities from their training data. Discriminating against use cases is MUCH less bad than discriminating against users. And the world does need all the time it can get to prepare, harden, check old code, (properly) formally verify new code, etc before the full weight of very powerful cyber attacks is upon us. And if this first big wave of AI risk is not handled well, then we may well get a dystopia of highly locked down models, KYC gating on everything, highly intrusive surveillance on everything, a society where you may be a second or third or fourth-class citizen in terms of model access based on your credentials and nationality.
396
233
105
2,743
443,343
James Fox retweeted
We formally verified 123 open-source projects in a partnership with Anthropic. Formal verification at scale is here! Now we’re launching OSS Verify to scale it to the software we all depend on. We prove it’s crash-free or send you the patches it needs. Free for open-source. Let’s bootstrap the verified software ecosystem: theorem.dev/oss-verify
New on our Frontier Red Team blog: using Claude to formally verify software at large scale. Formal verification can prove that code is free of certain bugs, but historically it has only been feasible for small codebases (or required a lot of expert time). With AI models, however, this is beginning to change. In collaboration with @theoremlabs, we used Claude to prove security properties in over one hundred real-world open-source libraries over the span of a few weeks. Read our research in full here: anthropic.com/research/forma…
38
119
26
1,067
138,557
James Fox retweeted
I feel quite sad from doing my ICLR reviews today. It appears that the grim reaper of AI has come for scientific publishing. One paper was just desk rejected after I wrote a constructive review for it last week. The next two I had to give 1s. The fourth paper I recommended desk rejection, and the last I would reject for plagiarism, but the truth is that I don’t know if a human could attest to the writing. Please, please try to address this problem, future PCs. The ship has sailed for ICLR 2027, but after this, we will need something to change. Without a healthy peer review system, students are not exposed to unfiltered criticism of their work. This kind of criticism is essential for improving as a scientist, and it is a rising tide that lifts up the whole scientific community. We can’t abandon the system entirely. In order to be incentivized to engage with the system, reviewers either need payment or to be given work that is not complete drudgery to try to read. If reviewers disengage with the system or shirk their labor, as I very much want to do right now, this will lead to a death spiral where authors do not expect to receive fair reviews and reviewers do not expect to review genuine work. I wish there were promising solutions to this problem that did not involve increasing the deadweight loss associated with paper submission, such as requiring videos with submissions or submission fees. But the cost of submission in real terms has gone down, and we will need to artificially raise it in order to even begin to cope with the surge in demand for AI publications. There’s also the “burn it all down” solution, but I find it hard to imagine what that would end in besides reinventing the current system.
10
23
2
295
36,336
One sentence macro model: On the way out the door, every farmer's wife says to every farmer, "the weather's looking bad so don't buy as much at the market this week, but also don't sell our crops for less than the usual price."
25
55
13
2,007
439,861
The most important thing we can apply AI to is improving medicine and human health - very excited about this partnership with the CZI and the Virtual Biology Initiative!
Today, Isomorphic Labs joins the Virtual Biology Initiative (VBI) as a founding member. We are helping build an open resource for the global research community to accelerate how we understand and treat disease. Read more here: isomorphiclabs.com/articles/…
238
516
58
5,780
241,485
James Fox retweeted
Scaling Trust is announcing its first batch of Creators: teams building open-source infra and performing fundamental research for agents to coordinate securely, across digital and physical worlds. I gave Opus 5.5 our creator release doc and, after a few passes, it made this 👇 Meet the teams → scalingtrust.org.uk/projects We fund new teams on a rolling basis; next deadline 31 Oct → aria.org.uk/opportunity-spac…
With the capability to coordinate securely, AI agents could act reliably on our behalf, unlocking a whole range of economically and socially valuable interactions that were previously inaccessible. Meet the first cohort of teams we’re backing to make it happen, as part of our £50m Scaling Trust programme. 👇 link.aria.org.uk/ST-Creators
8
9
4
54
6,784
James Fox retweeted
The new issue of Works in Progress, arriving with subscribers this week, is our best ever and packed with must-read pieces. Our lead essay is @RobinRivaton and @bswud on how China's cities, and the competition between them, have driven its rise to techno-industrial preeminence. It also includes: - @danwwang on Chinese and American second-tier cities, - @tylercowen on the food of Brazil, - @vpostrel on the invention of supermarkets, - @CharlesCMann on the discovery of porcelain, - @salonium on the myth of accidental scientific discovery, - Stephen Kotkin on Leninism's fatal flaw, - @owl_posting on biomedical hedge funds, - @PMArslanagic on the formula for lovable cities, - @pietergaricano on Ukraine as the future of war, - @PatRubenFitz on the Bronze Age Collapse, and - @SCP_Hughes on the social lives of hard-partying Victorian elites. Plus: love and shame in Ancient Rome, making mosquitoes extinct, the return of 1930s dating norms, and why Pakistan is home to the world's best architect. And our new crossword, our cocktail column, classifieds, and more! The new edition has had a massive design upgrade with beautiful, premium matte paper, a larger form factor, and bigger, clearer and more elegant typeography. It is a joy to read. Copies arrive with subscribers this week: subscribe now or by October 16th to get yours. worksinprogress.co/print/
61
84
32
577
171,935
BREAKING NEWS The Royal Swedish Academy of Sciences has decided to award the 2026 #NobelPrize in Physics to Francis Halzen “for decisive contributions to the IceCube Neutrino Observatory and the discovery of high-energy neutrinos of astrophysical origin.”
299
6,042
990
15,769
1,350,814
James Fox retweeted
More curves are bending. In mid-August Nate Rush & I posted an analysis of which discovery curves are starting to bend. What's happened since then? Cyber vulnerabilities: curve is still bent. Discovery of vulnerabilities is accelerating even further, though exploited vulnerabilities remain fairly flat. Math: curve is further bent. We said there was a likely acceleration for small-scale problems. Since then there has been a claimed Millennium prize resolution, & credible rumors of another. All frequently-updated databases of open problems seem to be proceeding at a rapid rate. Algorithms: curve is now bending. Two high profile algorithmic competitions have had big curve-bending events: NanoGPT, and the Hutter compression prize. Other frequently-updated series remain relatively unchanged: CIFAR-10, Stockfish, matrix-multiplication exponent. More observations below.
New post with Nate Rush: Have we seen an acceleration in discoveries? Many plots & some tentative conclusions: 1. Cyber: ⤴️ sharp acceleration 2. Math: ↗️ some acceleration 3. Algorithms: ➡️ no clear acceleration
2
37
4
181
17,492
Today we're unveiling Trillium Labs @trillium_labs, a new non-profit to foster the open science of frontier AI. We're building open post-training recipes and will expand into open infra to study RSI, reward-hacking, multi-agent systems, and whatever comes next. We're built around the theory of change that you need more eyes to solve hard technical problems. We have faith in the scientific methods and communities that humanity has built, and worry that AI is becoming too closed to utilize them. Trilliums are wildflowers that bloom briefly in the spring, before the forest canopies fill out. Though they are small, they lay the foundation for the cycles of growth and nourishment through the rest of the year. At Trillium Labs, the recipes will be the slow nutrients for the seasons and the model releases will be the blooms. Building an institution dedicated to this is needed because, much as nature’s trilliums are slow to expand and grow, the open-ecosystem needs time and dedicated resources to catch up. I co-founded with with a long-time friend and collaborator Tom Zick (@thesezickbeats). We're hiring (full time + student collabs/interns), we're fundraising, and we're looking for compute. Please get in touch if you're interested in helping out. Offices based in the Bay Area and Cambridge MA, remote okay. I’m in the Bay Area until for The Curve and COLM to connect with people who are interested. We’re thankful to have initial support from Halcyon Futures and Schmidt Sciences with more funding en route to enable our ambitions of scaling. Our advisors @Thom_Wolf, @HannaHajishirzi, @gneubig and @ctnzr have been instrumental to building the ecosystem that exists today, and I’m stoked to get to keep working with them.
250
300
83
3,278
262,198
James Fox retweeted
you can absolutely do safety from second place, or even not on the scoreboard. academics, startups, and nonprofits are doing great ai safety research on open models
You can't do safety from second place
65
46
6
1,192
62,114
James Fox retweeted
Best way to sandbox an AI is to put it on a Delta flight – it will have no chance of accessing the Internet!
7,173
9,443
1,572
160,506
8,854,101
James Fox retweeted
Schmidt Sciences is hiring Fellows in Residence! You help us with grantmaking decisions, while continuing your own research - $50k compute, $50k for a workshop - based in NYC - past fellows kept affiliations with Stanford, MIT, JHU Apply by Dec 15 Link 👇 Please share/rt :)
3
43
234
28,878
James Fox retweeted
➕➕ New research ➕➕ On the design of agents for markets and markets for agents. AI agents are increasingly entering markets, acting on behalf of people— How do we make sure they represent their people well, and that these markets are set up safely and fairly for everyone?
17
52
20
287
44,328
James Fox retweeted
Preventing loss of control is an open scientific problem. Science requires sharing and debating evidence in public. We can't agree on safety standards unless companies and third-party evaluators publish far more concrete evidence about risk. planned-obsolescence.org/p/e…
11
67
8
419
46,368