Co-founder @ Reticular (YC F24). CS & Math @ MIT
San Francisco, CA
Joined November 2020
- Tweets128
- Following2.9K
- Followers501
- Likes1.6K
Pinned Tweet
If you’ve ever
- thought AI protein folding is magical ✨
- wanted more than a pLDDT score 🔎
- or just think mech interp in bio is cool 🤓
then read the 🧵 👇 on our first paper towards interpretable protein structure prediction just accepted to workshops at ICLR
A First Step Towards Interpretable Protein Structure Prediction
With SAEFold, we enable mechanistic interpretability on ESMFold, a protein structure prediction model, for the first time.
Watch @NithinParsan demo a case study here w/ links for paper & open-source code 👇
sota genetic disease prediction achieved.
first step towards ending genetic disease.
Just published our state-of-the-art polygenic scoring results.
We're emphasizing our benchmarking methods, especially how we split.
medrxiv.org/content/10.64898…
John Yang retweeted
Introducing Kino: make videos end-to-end with Claude Code and Codex.
Drop your footage, get highlights, edit together, send links for feedback, and let your agents cook.
AI is becoming a great creative partner. Kino is a new home for your team and agents to collaborate.
holy shit codex might’ve just discovered a new genetic mechanism for psoriasis
tldr, psoriasis gets triggered by peptides generated by your immune system. ppl generate these peptides differently, causing diff risks for psoriasis & autoimmune diseases. illustrative fig below for reference.
we knew inheriting certain mutations in ERAP1, a peptide generating protein, increased risk of psoriasis. but no lit had looked at ERAP2 yet. Codex found a significant interaction bw HLA allele C*06:02 and ERAP2 allotype *02*01 (OR 5.84).
don’t want to make any big claims yet but really highlights how good these models are at biology
🤖 Made with AI
my codex cli session screenshot. it’s by no means a final result but exciting evidence for a new biologically plausible mechanism
as i late night tweet, i’m realizing i still have a deep ache for mech interp. genetics fills a similar hole as in both fields, i chase for explanations of observed behaviors. unlike protein design, it’s detective work, not a wild goose chase with no guarantee of success. but the purity of methods in LLM mech interp compared to understanding the biology necessary for genetics does leave mech interp a certain romance.
i like the term “recursive generalization” but imo the core goal of mech interp can be stated much more simply and rigorously.
given an overparameterized model e.g. LLM that produces an observed behavior, identify a minimally parameterized model e.g. circuit or SAE neuron that still has this behavior.
this goal can be stated as an optimization problem (big pro for gradient descent) and maximizes falsifiability.
by falsifiability maxxing, this goal serves as a good complement to observational studies which, while serving as the bedrock of the field, ultimately suffer from falsifiability & reproducibility.
especially when working on overparameterized models like LLMs for mech interp & cells in biology, it’s always possible to find edge cases that fit what you’re looking for. after all in 3D, any two points form a line.
in practice, physicists takes this “minimal param” framework to the mathematical extreme, discovering symmetries that reduce param # drastically or vice versa for symmetry breaking events.
neuroscientists instead approximate by removing brain chunks til the behavior goes away /s
interpretability might be the most important scientific problem in the world right now, as I don't think we can solve alignment if we don't know what we're building. I think I have good news about it. I co-authored early work in mechanistic interpretability at OpenAI and came away pessimistic, because neural networks appeared to fold into arbitrary tangled knots. years later, I started openai's first large-scale automated interpretability project, finding about 2% of neurons legible with the primitive tools we had
with modern agentic tools and a much closer look, the picture looks totally different. the vast majority in models up to trillions of parameters are legible, and their circuits beautiful. many famous examples of tangled neurons turned out to be misunderstood. The roadblocks I expected mostly aren't there. I now think ambitious interpretability will be solved by the time we need it most, and possibly quite soon, as automated scientists keep improving. it may be a bitter lesson for interpretability: the simple thing, at absurd scale, works
already-clean.goodfire.com/
to argue with myself here, perhaps new minimal param math frameworks are not necessary & futile when Fable and Astra can discover sick patterns a lot more scalably & falsify just as quickly, to the point agents can generate so many observations that you can do standard hypothesis testing for falsification.
i just feel like the ceiling for rigorous mech interp is so high that we’ve barely scratched the surface.
in comparison, physicists go hard on minimizing their params. there’s so much math on deriving the minimal set of params (irreps) for simple stuff like rotations in 2D and 3D. & that’s why we’ve been able to do so much modeling w/o NNs for so long
