Jasper Gilley

First, congrats to @nickcammarata on questioning assumptions in his field and reinvigorating its discussion. In the same spirit, I want to interrogate interpretability as well: why does it always start from the assumption that the system works? If NNs face challenges with AI aspirations like OOD generalization, continual learning, and creativity, then are not their internal mechanism the place to explain (and ultimately rectify!) such problems? If you look inside a system and really understand how it works, then shouldn’t you also gain some insight into how it could work better? Mech interp focuses on mind reading and mind control, but seems strangely ambivalent to mind improvement. Our 2025 paper on fractured entangled representations (FER) showed evidence of a striking chasm between neural networks with good representation (from open-ended search) and bad representation (from conventional gradient descent) in small networks that output Picbreeder images. I would expect precisely the kind of meticulous sleuthing @nickcammarata showcases to help to illuminate where FER might be hiding and how much of it there really is in large networks and LLMs, and ultimately what kinds of interventions mitigate it. Let me give an example image from Nick’s own blog, which I pasted into this post. The image is presented in the blog to support the idea that there is a “clean” interpretation of each neuron. But what about thinking about the story these neurons are telling about the adaptability of the circuit itself? The story about cleanliness says that a number of clean feature detectors feed into a “pointy” V-detector, which is then used to help identify a bunch of pointed objects. But think about the entire circuit more critically, and it starts to look like the entanglement (the E) in FER: we have a “van front” detector feeding into our pointy V-detector, which then feeds into (is read by) a coyote face detector. Does it not follow that if a gradient update later alters our van front detector, then our downstream V-detector will consequently be altered, and hence our ability to detect coyote faces? If so, then changes in van detection will impact our ability to perceive coyotes - coyotes and vans are entangled in this circuit! Is this normal in large networks? Should we even be using the word “clean” to describe a situation like that? How pervasive is this kind of entanglement? Similar evidence for and questions about fracture (as opposed to entanglement in this example) could easily be raised. To be clear, I’m not trying to claim that the FER hypothesis has been confirmed by a single (or even a few) anecdotal examples in a piece creating controversy with its own set of provocative claims. Rather, what I’m saying is, if you have (even imperfect) microscopes looking inside these systems then there are a lot really interesting questions you could be asking about the structure of their representations beyond just whether we can read their minds - mechanistic interpretability could widen its scope!