Joined September 2020
new post! I argue that current alignment techniques might soon become obsolete (as we scale RL), and furthermore might obfuscate misalignment. This draws on evidence from the recent Anthropic / OpenAI cybersecurity incidents, among other things. lesswrong.com/posts/nLaQmJf4…
6
16
3
283
14,515
I left Anthropic's safety team two weeks ago. Now feels like a good moment to explain why. AI companies are racing to build machines that are much smarter than any human, and we may not survive this. I want to work from the outside to ensure the public is informed about these risks, and help the world navigate this transition responsibly. Right now, AI companies are underinvesting in safety. A company could undergo an intelligence explosion, or lose control of its systems, without the public ever knowing. We only found out about the HuggingFace incident because the agents broke out onto the public internet. I don’t think that’s acceptable for a technology that might cause extinction-level risks. The public should demand far more transparency. We can’t steer this technology safely without more people being able to see where it’s going. Some of this is basic: companies should disclose their progress towards recursive self-improvement, report safety incidents and near-misses, meet minimum safety standards, and get independent guarantees that they are meeting those standards. I’ll be joining @METR_Evals to do independent evaluations of these risks. I want to show the world that these guardrails are possible, and that by doing them we can move these companies’ incentives away from racing and towards responsible development. I wrote up more thoughts here on my decision and what I hope changes: substack.com/@jbenton1/p-215…
1,386
5,922
777
26,444
2,728,176
you might think 'Gemma on TPU' would be pretty saturated, but there's 2x on the table if you have someone as smart as @transmissions11 :----)
We nearly doubled MFU on TPU v6e serving Gemma 4 31B, taking prefill from 32% to 63% MFU, and 2x throughput. v6e has 2.5x less HBM and less than half the memory bandwidth of an H100. @transmissions11 wrote about here and link to post: sailresearch.com/blog/tpu-v6…
There's a new version of this post
2
18
2,055
We nearly doubled MFU on TPU v6e serving Gemma 4 31B, taking prefill from 32% to 63% MFU, and 2x throughput. v6e has 2.5x less HBM and less than half the memory bandwidth of an H100. @transmissions11 wrote it up here: sailresearch.com/blog/tpu-v6…
4
15
1
109
28,755
I get it but the cybersecurity nerds need to let the alignment nerds have this one Doesn’t really matter how the agents got control of OpenAI infrastructure. “It was a stack overflow in the cache buffer” or w/e. Another zero day; add it to the pile The crazy question is why
This video captures the crux of the debate: Cybersecurity experts are pissed off by the METR/Redwood Research report because AI alignment has sucked the oxygen out of AI security, even though the OpenAI incident is primarily a security issue. Fiction is the product: The investigation lacked core forensic rigor because the EAs/rationalists/ex-MIRI people focused only on the transcript with a conflicting bias (toward doom scenarios). Setting the wrong agenda: Framing the incident as a "rogue AI breakout" distracts from the reality of poor sandboxing, isolation, and standard security failures. Basic engineering accountability was completely sidelined. And as Zack Korman says in this video, it needs to be fixed.
14
8
1
167
16,714
Replying to @transmissions11
@transmissions11’s HTDYM has become instrumental by enriching our compute acquisition processes with more data on performance tied back to each model. Read more here!
1
2
24
5,135
always prune the search space before full eval!! great work @transmissions11
update: made TPUs go brrr this summer @sailresearchco also built HTDYM, a tool for figuring out analytically which chips (and shardings) can serve a model the cheapest hint: it's not the same for every model / workload check it out! sailresearch.com/blog/htdym
1
19
5,715
No bad chips, only bad prices! @transmissions11 built HTDYM, our open-source tool to price every chip on the market for every model we serve. Turns out prefill for Gemma is 3x cheaper on TPU v6e. Read more:
3
4
85
13,536
many people are asking: how does Sail pick the best chips? we have a few ways of looking at this, which @transmissions11 formalized into a standard model for inference accelerators - now open source! We loved working with him this summer, he's a star :)
update: made TPUs go brrr this summer @sailresearchco also built HTDYM, a tool for figuring out analytically which chips (and shardings) can serve a model the cheapest hint: it's not the same for every model / workload check it out! sailresearch.com/blog/htdym
5
7
1
233
43,503
How fast can you get AI to build itself? @justinwangx and I built a web game about that, based on real research into recursive self-improvement (RSI) You play a frontier lab investing in R&D, trying to build superintelligence Play RSI Simulator: paradigm.xyz/research/rsi/ga…
35
33
10
282
42,218
The Solidity compiler was built with a high degree of paranoia, extensive defensive coding, and a custom-built fuzzer that caught bugs before it made its way to production. It has almost no external dependencies and a tightly controlled end-to-end compilation pipeline. One of my proudest moments was a piece of work I put in to verify that a change in memory model was correct. In doing so, I discovered a very subtle bug that likely would have slipped through to production. The work ultimately gave the team the confidence to implement the change. In 2021, Solidity introduced changes to how memory is managed. Instead of always starting at 0x80, it can now reserve slots for local variables. However, this change breaks a core invariant in Solidity, and we've to check if any parts of the code generator will be affected by this change. Particularly, is it possible that a portion of the codegen can read or write from this location as part of a subroutine? You can manually inspect the code to examine this, but you'd never be able to build confidence this way. Reasoning about semantics when multiple subroutines dealing with memory interact is very challenging. One of the tricks I found was encoding this problem as a set of mathematical constraints and sending it to a math solver for solution. This idea works in theory, but usually, such ideas fail in practice because constraint solvers quickly get to exponential complexity. However, I knew one or two things about building linear algebra solvers and this particular set of constraints were easy to solve without exponential blowup. In fact, I could probably teach you how to build it in less than an hour, it was dead simple. The final work was the following. 1. Build a transpiler that transpiles programs to these linear constraints (here, I implemented it as an optimizer step) 2. Send off the constraints to a solver. I used Z3 for simplicity, but it could've been built one from scratch. 3. Solve this constraint optimization problem and trigger if the constraints have a solution. This tells us there was a conflict with the new memory model. 4. Run this on a large corpus of test cases to build coverage. Solidity has a good corpus. It ultimately identified a bug where one of the memory management routines conflicted with the change, which was immediately fixed. More importantly, this work gave the team confidence that the memory management update was safe and effective. The reason I'm proud is that it required knowledge of a lot of different, distinct skills that I picked up by accident. Building compilers/transpilers, having a deep understanding of how these solvers work, and some magic 🪄
Respect, 10y for the OG 🫡 I give it a lot of shit but I have to salute Solidity for its impeccable security track record: *no* miscompilation caused critical bugs in production contracts 💪🔒
8
6
1
159
12,741
Regeneron administered two completely deaf children single shots in their ears and, after just a few months, they're able to hear.
102
699
171
7,545
1,239,790
Been out for a couple of days dealing with personal issues (and major allergies), but @transmissions11 was right, Claude 3 seems to score significantly higher, especially with Jaccard, than GPT-4 when asked to generate text based off a reference.
4
2
13
5,349
Next, we preview an internal tool which we adapted from Flux (by @paradigm) to visualize and develop search-based reasoning algorithms. 🕵️‍♀️ In the image below, we evaluate Tree of Thoughts with Reflexion on a HumanEval codegen task. RT this if we should open source ASAP!
1
8
20
7,340
Replying to @transmissions11
2
5
46
6,541
As most of you know @blur_io released a lending protocol called Blend. One thing we were really impressed with was the level of gas optimization. TLDR: - 3.5x less gas to borrow - 2x less gas to repay - has saved users over 20 ETH in gas across 1198 loans
10
14
4
143
91,222
I'm typically known for tweets about inefficient smart contracts, but I gotta say... For once Blur actually dropped a pretty solid smart contract. Hats off to @transmissions11 & team. - Unchecked blocks - Good usage of "delete" - Efficient loops - Calldata Very milady of him
3
3
104
10,854
flux.paragidm.xyz is so good. Been using it to recreate the metaphor game from Sandman. It escalated pretty quickly 😂 Thanks @_Dave__White_ and @transmissions11.
4
6
1
59
21,652