Goodfire (@GoodfireAI) header
Goodfire (@GoodfireAI) profile picture

Using interpretability to understand, design, and align AI.

San Francisco
Joined August 2024
Photo by Goodfire (@GoodfireAI)Photo by Goodfire (@GoodfireAI)Photo by Goodfire (@GoodfireAI)Photo by Goodfire (@GoodfireAI)Photo by Goodfire (@GoodfireAI)Photo by Goodfire (@GoodfireAI)Photo by Goodfire (@GoodfireAI)Photo by Goodfire (@GoodfireAI)Photo by Goodfire (@GoodfireAI)Photo by Goodfire (@GoodfireAI)
We’re announcing an embedded assessment with @OpenAI to advance safety and alignment at the frontier.
19
71
20
761
87,252
Goodfire retweeted
To encourage research on this model, we're launching the Looped Lens Challenge with @thinkymachines and @GoodfireAI, launching this Monday! We challenge you to develop methods that help us surface what Inkling 2 Mini computes but never writes. 🔗 ndif.us/looped-lens-challeng…
1
7
3
58
6,366
Looped models are weird. A layer can write to itself or read from "future" layers. @thinkymachines gave us early access to Inkling 2 Mini so we could study its inner mechanisms. We uncover how it coordinates information flow across layers, blocks, and loops. Studying Inkling 2 Mini gave us cause for optimism: as transformers continue to evolve, we believe our ability to understand their internal mechanisms can evolve with them. Read the full post: goodfire.com/research/interp…
Some of you thought that Inkling 2 Small was too big at 594B parameters. We heard you: Inkling 2 Mini is a looped transformer with just 26B parameters, small enough to run on your laptop, powerful enough for agentic work with a score of 20.1 on AA Index. thinkingmachines.ai/news/ink…
12
120
14
938
57,790
What scientific discoveries are hiding inside AI models? 🔍 Join us Dec 12 at @NeurIPSConf in Atlanta for Interpretability for Discovery. Hear from our amazing speakers: @begusgasper, @kfrankelab, @nkwang24, and @jwuphysics. Join us for spotlight talks, posters, and a panel!🎉 We’re hosting a social right after the workshop! Stay tuned for details, and a big thank-you to our sponsor @GoodfireAI for making it possible! Looking forward to seeing everyone! Organizers: @boknilev, @EkdeepL, @YNikankin, @ChenhaoTan, @amirtha255
2
16
4
49
3,727
A language model is a wonderland of strangely specific neurons, from gargling to comic exasperation to the moment a guard looks away. We made a searchable atlas of all 10 million neuron poles in Qwen3.6. Play with it here: goodfire.com/research/public…
21
89
17
635
31,596
Each neuron pole, a neuron's positive or negative side, has a rough label from its max-activating examples. Poles with similar labels are close together. You can explore a sample of 3,000 poles studied in depth in "Library." Check out our Neurons work: goodfire.com/research/neuron…
36
1,586
New neurons research: introducing Neuron Communities. We discovered communities of neurons that work together inside LLMs. They’re surprisingly small, but powerful – just a few neurons work together to perform common behaviors, like arithmetic, ALL CAPS, or recalling a date. Work led by research fellow @_igorshilov:
21
106
23
882
45,998
This adds to the evidence that neural networks already contain interpretable structure if we know how to look for it. We can now find communities at scale, which agents can then investigate in depth—another tool on the path towards a fuller understanding of neural networks.
1
1
31
2,202
Read the full post and explore examples of the communities we found: goodfire.com/research/neuron…
1
35
2,250
Goodfire retweeted
incredibly excited to be partnering with @OpenAI on embedded assessments! we'll be embedding interpretability researchers who will use the @GoodfireAI platform to monitor and debug training runs to improve safety and alignment our primary audience for our assessments are internal OpenAI researchers, but we'll publish our findings as the collaboration progresses
We’re working with @ApolloResearch, @METR_Evals, @TransluceAI, and @GoodfireAI on embedded assessments: giving outside experts deep access to our internal models and systems to investigate alignment and test our safeguards. These assessments build on our ongoing alignment and safety research, bringing independent scrutiny to how we identify and manage risks in our systems. openai.com/index/our-first-p…
15
13
2
257
9,192
Goodfire retweeted
We recently went viral on X with the claim that neurons are "already clean." But what do we mean by that? Are neurons are a useful basis for interp? We're finally releasing our technical report on LLM neurons--and what we've found is far more interesting than mono/polysemanticity.xitter.gq/GoodfireAI/status/2120…
In Neural Networks are Already Clean, @nickcammarata argued that neurons were far more legible than previously thought – and that this was great news for interpretability and safety. Now, we’re releasing the first technical companion to Nick’s essay. It shares our findings from studying thousands of neurons using our novel automated interpretability pipeline, and outlines why we now think of neurons by their computational function, instead of whether or not they are polysemantic. Here’s what we found: 🧵
19
42
3
377
26,462
In Neural Networks are Already Clean, @nickcammarata argued that neurons were far more legible than previously thought – and that this was great news for interpretability and safety. Now, we’re releasing the first technical companion to Nick’s essay. It shares our findings from studying thousands of neurons using our novel automated interpretability pipeline, and outlines why we now think of neurons by their computational function, instead of whether or not they are polysemantic. Here’s what we found: 🧵
interpretability might be the most important scientific problem in the world right now, as I don't think we can solve alignment if we don't know what we're building. I think I have good news about it. I co-authored early work in mechanistic interpretability at OpenAI and came away pessimistic, because neural networks appeared to fold into arbitrary tangled knots. years later, I started openai's first large-scale automated interpretability project, finding about 2% of neurons legible with the primitive tools we had with modern agentic tools and a much closer look, the picture looks totally different. the vast majority in models up to trillions of parameters are legible, and their circuits beautiful. many famous examples of tangled neurons turned out to be misunderstood. The roadblocks I expected mostly aren't there. I now think ambitious interpretability will be solved by the time we need it most, and possibly quite soon, as automated scientists keep improving. it may be a bitter lesson for interpretability: the simple thing, at absurd scale, works already-clean.goodfire.com/
53
249
60
1,791
148,867
We deeply investigated thousands of neurons in this work. What did they look like? We are releasing descriptions for 1,500 neurons in Qwen3.6 35B-A3B. Explore here: goodfire.com/research/public…
3
2
2
103
6,469
Investigating neurons could have revealed that they were intractable objects of study. Instead, we find that they are more interpretable than we previously thought. Read the first post in our technical series on neuron interpretability here: goodfire.com/research/neuron…
1
1
1
120
5,686