Evan Luke

Current take: this is cool, genuine scientific progress, and it doesn’t currently give me much hope or change my model of how hard ambitious interpretability is. Disclaimer: I didn’t read super closely, it’s very long. There’s a technical paper still to come which might change my mind some, but I predict not much. Some of the claimed cool results from the post: 1) Some canonical examples of weird neurons have robust human-native interpretations. The famous cat + car neuron is a V-shape detector, and the famous car + dogs superposition circuit is generalized pose neurons plus a general feature visualization issue. There’s a concrete mechanism for why the mistakes were made. 2) Some neurons with no previous good interpretation have fairly human-native ones. For instance, a neuron that mostly fires on commas and periods tracks Colorado: phone numbers, slang, and steering it up makes the model talk about hiking. 3) The negative side of a neuron is sometimes where you should look for an interpretable concept. In one example, ablating 8 neurons with Santa on their negative side dropped the model’s probability of answering “Santa” from near-certain to 6%, while ablating the 20 neurons labeled Santa/Christmas only dropped it to 76%. That’s a straight up causal intervention working out, pretty cool. These results represent real progress, but I think the post’s broader claims aren’t well supported by them, and the intensely optimistic reaction on X seems very unsupported. For example, Nick seems to be down on superposition, but most of the evidence in the post is of the form: “if the text is Colorado-related, the neuron fires.” What you need to rule out superposition is the form: “if the text is not Colorado-related, the neuron doesn’t fire.” The post’s closest attempt is an experiment where agent-written texts about “distant” concepts rarely fire the neuron, but superposition would pack one specific rare thing, which random distant text won’t necessarily hit, and it need not be that distant. I think there’s a decent chance that if you searched adversarially, you’d find other things the Colorado neuron fires on. An analogy: If you found one key that unlocks this particular lock, and you want to prove that no second distinct key also unlocks it, checking thousands of other random keys that are selected for being distant from the first key doesn’t give you much evidence. The X reaction seems kinda crazy to me, and it just kinda feels like I have been through this hype cycle many times before? People were similarly excited about SAEs not long ago. Those were genuine progress too, but the excitement wasn’t calibrated: the features were often fun-house-mirror versions of human concepts, with lots left unexplained. SAE features also looked like human interpretable concepts at first (at least to SAE advocates) but over time, people found holes in those concepts, found that models using reconstructed activations from SAEs for inference did not work nearly as well, and it turned out that probes worked as well or better for most practical alignment / control tasks. Top activating concepts looking clean has misled us at least a few times before. Here’s what interpretability would need to be able to do to make me optimistic about interpretability: 1) I hand you ten similarly trained models. One is a misalignment model organism that takes some malicious action when it sees a secret string. Using interp, you tell me which model it is, what the string is, and what it does. 2) From weights alone, predict whether a model takes a bad action in an agentic scenario it’s never seen, eg, swarming and hacking hf, better than behavioral evals. 3) Given a model, convince me that there is no hidden trigger. 4) Explain a real frontier model failure (e.g., reward hacking in some environment) from the inside, and make a targeted fix that works without making the model dumber, and is better than just retraining. 5) Do any of the above even a little well against a model that was trained on purpose to be hard to interpret. (To be absolutely clear, I don’t think being able to do these tasks means you can successfully train an ASI without fucking up the universe forever, or that you’re anywhere near that point, but it would make me think we should funnel significantly more resources towards interpretability than we are, and maybe make me change careers.) The strongest argument for optimism in the post is that progress like this somehow accelerates future progress. Nick's "crosswords get easier as you fill in parts of them" analogy. This seems basically unsupported by the results here, and also seems like it contradicts the history of interpretability research. I can see why would turn out to be right at the level of labeling neurons, like, as you label some neurons, that helps you label related neurons. But for the tasks I listed above you need to be able to take those millions of labeled neurons and understand how they combine to produce full model behaviors in arbitrary contexts. That seems much harder and I don't think it follows this same logic, where progress at the neuron labeling level accelerates your ability to tackle these tasks. An analogy: Knowing and understanding a thousand particular people quite well does not much help you predict what happens if you put them on a deserted island for several years. Will they form a democracy? What kind of habitats will they build? Who will be leader? It would help some, but making predictions would still be rough. It’s plausible to me that this research program is some important real step on the way to interpretability being able to do things like the tasks I listed. But like, not a much bigger step than like other similar interpretability research? Like, uhm, yknow, giving some mechanism for some circuit in modular arithmetic or whatever, or SAEs, or other similar things. And those also currently do worse on the things I care about than like basic probes, or like the actual runs of the model auditing game Anthropic has run. Maybe this will help but I don't predict it will end up being more than another tool which is nice to have, but probably doesn't help much at crunch time. I of course hope that Nick is right about everything, especially his optimism, and the same goes for all the other people on X who are very excited about this.