Stanford Professor of Linguistics and, by courtesy, of Computer Science. Member of technical staff @stanfordnlp and @StanfordAILab. Co-founder @ Bigspin AI.
Joined November 2011
- Tweets3K
- Following742
- Followers16.8K
- Likes3.6K
Pinned Tweet
I grew obsessed and ended up writing 7600 words to annotate this diss track. I learned a lot by doing this. It led me to conclude that everyone is more or less equally (un)deserving of disses: web.stanford.edu/~cgpotts/bl…
Like any good advisor, I felt duty-bound to defend @aryaman2020 and @ZhengxuanZenWu in this rap battle. However, the SAE diss track I wrote was so devastating as to be unanswerable, so I decided to graciously balance things out with a second verse dissing causal interp.
A very interesting hypothesis, and clearly aligned with the findings from this cool paper ("Prompt Injection as Role Confusion"): arxiv.org/abs/2603.12277
Why is it learning to avoid whitespaces? I thought it might derive from pressure to keep reasoning short, but @yoavgo pointed out that dropping whitespace doesn't necessarily save tokens. Is it instead learning to rely on infrequent token combinations to create a CoT language?
Christopher Potts retweeted
I think there are some issues in the recent “Already Clean” essay, and I’m concerned it may have incorrectly updated people on important safety-relevant questions.
My thoughts (including some reproductions): livgorton.com/already-clean-…
Christopher Potts retweeted
Come join the 4 of us and answer exciting research questions!
I’m looking for PhD students who want to: 1) understand the role of input in shaping generalization behavior — how do different sources of positive and negative evidence conspire together to give rise to learning?…
The UT comp ling faculty (@jessyjli, @kanishkamisra, @kmahowald & myself) are looking for PhD students to join us! sites.utexas.edu/compling/ 🤘
We are an interdisciplinary group with interests across linguistics, cogsci and NLP. Apply by 12/1: liberalarts.utexas.edu/lingu…
Christopher Potts retweeted
Since a reporter recently reached out to me about this, and inspired by @ChrisGPotts sharing his own comments publicly, I thought I'd share my perspective here as well.
I find the reassessment of superposition genuinely interesting, especially given how influential the superposition hypothesis has been. In particular, the discussion of how conclusions drawn from toy models may not carry over to real networks with real data is important. Clearly more work is needed to support the broader claims. I'd particularly like to see systematic evaluations of how well the proposed neuron interpretations generalize across inputs and models.
Superposition hasn't played a major role in my own interpretability research. It's one hypothesis in a much broader field, with many different approaches and research questions. As such, these findings don't fundamentally change the questions I'm working on, but they add to a broader view on how models store information.
For me, an important goal of interpretability is to go beyond explanations. Can our understanding help us predict model behavior, improve training methods or make targeted causal interventions? Our B-TAP method, for example, uses interventions analogous to brain lesions to study mechanisms of particular behaviors, without requiring a complete neuron-level explanation.
But most importantly, understanding individual neurons is not the same as understanding complex model behavior. Even if most neurons are interpretable, we still need to understand how they interact to produce behavior. This is especially true if we want to apply interpretability to safety, which includes very complicated behaviors over many tokens and turns such as reward hacking and deception. Trying to apply interpretability to these realistic scenarios is what we should focus on.
interpretability might be the most important scientific problem in the world right now, as I don't think we can solve alignment if we don't know what we're building. I think I have good news about it. I co-authored early work in mechanistic interpretability at OpenAI and came away pessimistic, because neural networks appeared to fold into arbitrary tangled knots. years later, I started openai's first large-scale automated interpretability project, finding about 2% of neurons legible with the primitive tools we had
with modern agentic tools and a much closer look, the picture looks totally different. the vast majority in models up to trillions of parameters are legible, and their circuits beautiful. many famous examples of tangled neurons turned out to be misunderstood. The roadblocks I expected mostly aren't there. I now think ambitious interpretability will be solved by the time we need it most, and possibly quite soon, as automated scientists keep improving. it may be a bitter lesson for interpretability: the simple thing, at absurd scale, works
already-clean.goodfire.com/
Christopher Potts retweeted
I feel quite sad from doing my ICLR reviews today. It appears that the grim reaper of AI has come for scientific publishing.
One paper was just desk rejected after I wrote a constructive review for it last week. The next two I had to give 1s. The fourth paper I recommended desk rejection, and the last I would reject for plagiarism, but the truth is that I don’t know if a human could attest to the writing.
Please, please try to address this problem, future PCs. The ship has sailed for ICLR 2027, but after this, we will need something to change.
Without a healthy peer review system, students are not exposed to unfiltered criticism of their work. This kind of criticism is essential for improving as a scientist, and it is a rising tide that lifts up the whole scientific community. We can’t abandon the system entirely. In order to be incentivized to engage with the system, reviewers either need payment or to be given work that is not complete drudgery to try to read. If reviewers disengage with the system or shirk their labor, as I very much want to do right now, this will lead to a death spiral where authors do not expect to receive fair reviews and reviewers do not expect to review genuine work.
I wish there were promising solutions to this problem that did not involve increasing the deadweight loss associated with paper submission, such as requiring videos with submissions or submission fees. But the cost of submission in real terms has gone down, and we will need to artificially raise it in order to even begin to cope with the surge in demand for AI publications. There’s also the “burn it all down” solution, but I find it hard to imagine what that would end in besides reinventing the current system.
I was asked by a journalist to comment on this piece. The comments I wrote up (on Friday evening) are included below, so that I have a public place where my view is fully represented.
1. I appreciate the critical reassessment of some of the existing arguments in favor of superposition. It does look like alternative explanations may be viable.
2. I should say that, despite being an author on perhaps 40 papers on interpretability, I have never really worked on the superposition hypothesis per se. There's a lot more to interpretability than this one idea! We did, though, develop a benchmark (RAVEL) and associated metrics for assessing the extent to which features in neural networks can be disentangled: aclanthology.org/2024.acl-lo… We found that many neurons are polysemantic, but we made no claims about the more specific notion of superposition. It would be great to see RAVEL results for newer models.
3. A fundamental motivation for interpretability research is preparing for an uncertain future full of strange new models with surprising new capabilities. Superposition is a technical insight about how neural networks can learn to pack unrelated features into a single neuron. Even if present-day models rarely do this, we need to be ready for future models. Cammarata's analysis seems to trace legibility in part to architectural choices. Future architectures might be very different. If interpretability researchers don't prepare us for that, who will?
4. Present-day models may make use of superposition as well! Cammarata suspects that they don't, or do this so rarely that we could even ignore the actual instances ("doesn't matter for study"). I would characterize this as a hunch based on a lot of initial heuristic empirical work. In the normal course of scientific research, this would be the start of a process of figuring out precisely how to rigorously test this as a hypothesis.
5. For example, the piece has a nice figure that Cammarata characterizes as indicating that a particular neuron has positive activation on pointed things and negative activation on round things. Cool! Let's define some independent criteria for those classifications so that we can test this objectively. (It is possible that the paper Cammarata is preparing will include these evaluations.)
6. The piece expresses the intuition that interpretability is easier, in some sense, with better models. This intuition has been very common among researchers for years. At the same time, I understand why @JacobHHilton would say the opposite: xitter.cf/JacobHHilton/status/21…
7. The piece seems to allude to a technical proposal, but it does not give the details. As far as I can tell, the ideas have substantial overlap with this work (led by two former students of mine): Arora, Wu, et al. 2026 (arxiv.org/abs/2601.22594). Cammarata does cite this. I am guessing there could real differences that will be interesting to see. I would hope that Cammarata explains the differences and reports comparable evals to Arora et al, at least for language models. I appreciate that he looked at "thousands" of examples (the essay says this a lot), but even a large number here is not a substitute for rigorous evaluation.
8. In my research group at Stanford, we are working on lots of very hard problems in interpretability. As far as I can tell, nothing about this blog post makes these problems any easier, and certainly none of them are solved by this work. It would not have occurred to me to say this at all, since it is not even reasonable to have this expectation, but very prominent people on social media seem to believe that it is an all-encompassing breakthrough. Some immediately extrapolated to an optimistic story about AI safety that is not supported by anything in this piece. (Highly interpretable agents could of course do dangerous things.)
9. It must be said that Cammarata's framing did encourage these reactions. The header of the post page says, "An informal research essay and working hypothesis about neural networks", which suggests something quite preliminary and tentative. The piece also says, "Highly speculative: this might be most of what we need, and interpretability might be far more tractable, possibly in the near term, than we've assumed". However, his announcement tweet (xitter.cf/nickcammarata/status/2…) reworked this considerably: "I now think ambitious interpretability will be solved by the time we need it most". Strictly speaking, this commits to very little beyond optimism, but it is written to invite many inferences.
interpretability might be the most important scientific problem in the world right now, as I don't think we can solve alignment if we don't know what we're building. I think I have good news about it. I co-authored early work in mechanistic interpretability at OpenAI and came away pessimistic, because neural networks appeared to fold into arbitrary tangled knots. years later, I started openai's first large-scale automated interpretability project, finding about 2% of neurons legible with the primitive tools we had
with modern agentic tools and a much closer look, the picture looks totally different. the vast majority in models up to trillions of parameters are legible, and their circuits beautiful. many famous examples of tangled neurons turned out to be misunderstood. The roadblocks I expected mostly aren't there. I now think ambitious interpretability will be solved by the time we need it most, and possibly quite soon, as automated scientists keep improving. it may be a bitter lesson for interpretability: the simple thing, at absurd scale, works
already-clean.goodfire.com/
I fleshed out item 5 a bit in this tweet: xitter.cf/ChrisGPotts/status/211…
There has been a persistent failure in this space to consider both precision and recall, and there is too much willingness to rely on the sort of heuristic spot-checking that leaves room for just-so stories. Ex ante evaluation tends to be eye-opening.
That is the sort of work we did in this paper, which is an extended evaluation of Bills et al.'s (2023) auto-interp work. (That paper includes Nick and Leo as authors; I infer from Leo's new LessWrong post that he has some sympathy with our assessment and our intervention-based approach.) aclanthology.org/2023.blackb…
A friend pointed out that I phrased 6 ambiguously. I intended to be saying that better models are easier to interpret. I believe this is Cammarata's primary claim as well, but he is also arguing (even less controversially, I would guess) that interp becomes easier as agentic tools for doing the research improve.
Christopher Potts retweeted
Which members of Congress most need a swear jar?
To accompany my latest piece, I built an automatically updating interactive website. See which current lawmakers swear the most on the House and Senate floors
themarginoferror.com/congres…
For Aryaman's defence (yes, spelled with a "c"), I did write lyrics for an underground homage to Nas's "Hip Hop is Dead". Cover art:
🤖 Made with AI
ALT The image is an AI-generated homage to the cover for Nas's "Hip Hop Is Dead". It depicts Aryaman holding a back rose above a stylized "AA", and below that the phrase "Interp is Dead". Dark clouds and ominous birds dominate the background.
I defended my Ph.D. @stanfordnlp a few months ago, and recently started @AnthropicAI on the Interpretability team!
(pictured: my awesome defence committee members)
The song (written in July 2026) is in fact about all of the productive things Aryaman would do if we woke up to discover that interp had been declared dead. The chorus is full of references to projects he did during his PhD:
If interp should die, before I wake,
I'll swap a ReFT vector in, fine-tune all day.
Go prefill-only, decode will slay.
Build out the RePS, yo, steer anyway.
If interp should die, before I wake,
I'll build out the benchmark, then Ax the whole Bay.
Sort through the state space, Mamba can stay.
Hit the Gym hard, yo, syntax pathway.
Christopher Potts retweeted
Language models are getting much, much harder to fully understand with scale, not easier! (Thread)
Hey @nickcammarata, have you interpreted my 432-parameter model yet? Do you think you'll manage it by year end? That's 4 hours per parameter
Christopher Potts retweeted
I defended my Ph.D. @stanfordnlp a few months ago, and recently started @AnthropicAI on the Interpretability team!
(pictured: my awesome defence committee members)
Christopher Potts retweeted
It feels worth saying this at a top level too: to the extent that the positive update causing everyone to freak out is about MLP neurons being useful, a paper by @TransluceAI came out a year ago which covers similar ideas.
Christopher Potts retweeted
AI is cracking math problems. What's holding it back in engineering?
The cost of evaluation. Agents can generate ideas in seconds, but testing them can take hours or days.
💡 Meet DuoDiscover: where evaluation is part of what agents discover 👇🧵
Some highlights:
🏆 #1 on NVIDIA's SOL-ExecBench
🐍 2× the experts on SWE-fficiency
🚀 1.92× on vLLM; 1.44× on FreeToken
🔥 2.68× over production TPU kernels
🖥️ Beats the best human and AI-designed CPU prefetchers
Christopher Potts retweeted
MLP neurons are great, but knowing this alone does not solve interpretability.
interpretability might be the most important scientific problem in the world right now, as I don't think we can solve alignment if we don't know what we're building. I think I have good news about it. I co-authored early work in mechanistic interpretability at OpenAI and came away pessimistic, because neural networks appeared to fold into arbitrary tangled knots. years later, I started openai's first large-scale automated interpretability project, finding about 2% of neurons legible with the primitive tools we had
with modern agentic tools and a much closer look, the picture looks totally different. the vast majority in models up to trillions of parameters are legible, and their circuits beautiful. many famous examples of tangled neurons turned out to be misunderstood. The roadblocks I expected mostly aren't there. I now think ambitious interpretability will be solved by the time we need it most, and possibly quite soon, as automated scientists keep improving. it may be a bitter lesson for interpretability: the simple thing, at absurd scale, works
already-clean.goodfire.com/
Sunwoo's fiction site: translunarytree.github.io/
Sunwoo's academic site: sunwooj.github.io/
James Whang: jameswhang.net/
