Matthew Nemeth (@mnemeth101) header
Matthew Nemeth (@mnemeth101) profile picture

PhD protein design @ColumbiaMed | prev bio @Lux_Capital, ML for Cas proteins @arcinstitute @UCBerkeley🐻 | ā¤ļøā€šŸ”„a

New York, NY
Joined March 2021
Photo by Matthew Nemeth (@mnemeth101)Photo by Matthew Nemeth (@mnemeth101)Photo by Matthew Nemeth (@mnemeth101)Photo by Matthew Nemeth (@mnemeth101)Photo by Matthew Nemeth (@mnemeth101)Photo by Matthew Nemeth (@mnemeth101)Photo by Matthew Nemeth (@mnemeth101)
Matthew Nemeth retweeted
Really excited for Inkling 2 Mini! Mostly same arch as Inkling 2 Small, but 26B-A3B, 24 layers, with layers 6-17 looping 3 times by default. Same AA score as Qwen3.8-27B, should be a great model for the OSS RL community! Day-0 support ready on SkyRL with @vllm_project
Some of you thought that Inkling 2 Small was too big at 594B parameters. We heard you: Inkling 2 Mini is a looped transformer with just 26B parameters, small enough to run on your laptop, powerful enough for agentic work with a score of 20.1 on AA Index. thinkingmachines.ai/news/ink…
3
15
1
64
2,232
Has legend @JohnJumperSci Awoken and started cooking?
Given Claude Fable 5.2 can predict protein sequence, I was curious if it can predict structure. I gave it a relatively simple prompt: "predict contacts, 2ndary struct, use dynamic prog. and linear alg. to predict structure" It was able to predict fold of ubiquitin correctly 🤯
3
213
Matthew Nemeth retweeted
Microscopy lets scientists literally see the biological world. We think giving agents the same ability will be an important source of progress in science. Today we introduce TissueMicroscopyBench, our first release in a new microscopy series covering tissues, cells, and in vitro systems. The benchmark comprises 49 evaluations built from four public imaging datasets. Agents must locate structures, trace their boundaries, reason through three-dimensional images, and follow changes over time. Some of the work: - Colorectal cancer sections stained with H&E and multiplexed immunofluorescence. Tasks include outlining mucosa, fat, and nerves, and locating evidence of tumor invasion. - Liver metastasis biopsies taken before and after treatment. Agents compare tumor compartments, PD-L1 staining, and T-cell-rich regions across the images. - Light-sheet recordings of living organoids. Overlapping structures and densely packed nuclei require agents to distinguish individual objects and draw their boundaries. - A regenerating leg recorded in three dimensions over time. Tasks examine the injury site, local cell divisions, tissue retraction, and the sequence of regeneration. Compared with our previous benchmarks, these evals are especially visually striking. The rollouts are dense, interleaving image manipulation (including hand-rolled segmentation and CV in Python) with reasoning. Some slides contain billions of pixels. Agents have to decide where to look, examine the relevant regions at sufficient resolution, and return precise outlines, tracks, or volumes. These are scored against expert annotations. Every task was also solved independently by a human as a check on task quality. We tested 22 models in 30 model–harness configurations, with three attempts per task. Claude Fable 5.2 (LSVP) with Pi had the highest pass rate at 34.0%, passing 50 of 147 attempts. Timeouts and invalid answers counted as failures. Twenty-three of the 49 tasks were never solved by any configuration. We found that agents can be shockingly bad at localization, often finding the general region of interest but drawing boundaries that included surrounding tissue or random patches of empty space. Small structures and changes across time were particularly difficult. We think visual perception and spatial reasoning will become core capabilities for scientific agents. It’s also just a super fun space to think creatively about evals (h/t Ian Diks). Excited to release more in this series and watch agents improve here. - Blog: blog.latch.bio/p/tissue-micr… - GitHub: github.com/latchbio/tissuemi… - Leaderboard: benchmarks.bio/tissuemicrosc…
7
15
3
71
9,583
Matthew Nemeth retweeted
We’re expanding our benchmark efforts into basic research. Today we introduce NeuroscienceBench, the first release in a new series, with immunology, aging, and other core fields to follow. The goal is to measure whether AI agents can analyze experimental data and reach reliable conclusions about how biology works. We’re starting with neuroscience, where interpreting neural recordings, microscopy images, and animal behavior requires both analytical skill and scientific judgment. NeuroscienceBench comprises 91 evaluations drawn from 43 peer-reviewed studies, spanning sequencing, electrophysiology, EEG, brain–computer interface decoding, imaging, behavior, and clinical biomarkers. Agents analyze experimental data and return structured answers that are graded deterministically. Three public examples: - Neural recordings from the intertrigeminal nucleus. The analysis compares a narrow- and broad-spiking classification derived from the region’s own recordings with the conventional 0.40 ms spike-duration cutoff. - Digital tracings of Lamp5 Lsp1 interneurons from mouse visual cortex. Their cell bodies sit at different cortical depths, and different fractions of their axons reach the outermost cortical layer. The analysis examines whether depth alone accounts for that variation. - A mouse reaching through a narrow slot toward a food pellet. The video contains a pellet and a reflective paw marker with known dimensions, providing possible references for measuring movement in millimeters. We tested 21 models in 34 model–harness configurations, with three attempts per evaluation and no internet access. After excluding platform failures and refusals, 9,212 graded attempts remained. Claude Opus 5.5 with Pi had the highest observed pass rate at 32.2%. The mean across configurations was 21.0%. Across models, evaluations involving quality control, controls, or artifacts averaged a 10.6% pass rate. Those involving normalization or representation choices averaged 12.6%. Reliable neuroscience agents will need to make these decisions correctly across the many types of data scientists work with. Paper: latch.bio/neurobench Leaderboard: benchmarks.bio/neuroscience Blog: blog.latch.bio/p/neuroscienc… GitHub: github.com/latchbio/neuroben… Incredible work from @Jack_Brougher, Anshika Agrawal, @tpr720, Martin Jacko, and team.
14
22
2
86
5,872
Matthew Nemeth retweeted
Rare diseases collectively affect an estimated 300 million people worldwide. For patients and families, reaching a diagnosis can take years of referrals and testing. One place AI agents could help is in interpreting each new sequencing result or functional assay alongside the patient’s symptoms, family history, and earlier findings. We introduce RareDiseaseBench, a benchmark of 35 diagnostic workups reconstructed from published rare disease cases. Each agent starts with the patient’s presentation and part of the medical record. Over one or more visits, it must decide which tests to request, explain why, analyze the files it receives, and propose a diagnosis and management plan. Five cases from the benchmark: - A 13-year-old boy has growth restriction, hearing loss, and severe chilblains. An MN1 variant explains several developmental findings. Long-read sequencing reveals a translocation disrupting SAMHD1 that accounts for his inflammatory symptoms. - A 21-year-old man has heterotaxy with right atrial isomerism. A variant just beyond the standard ZIC3 transcript falls within an intron of an alternative transcript, causing 57 extra bases to be spliced into the RNA. - A boy develops progressive scalp hair loss while his eyebrows and body hair are preserved. A scalp biopsy shows miniaturized follicles, and sequencing reveals a 357-kb duplication spanning TAB3 and neighboring genes. - A 16-year-old boy with typical growth has a language disorder, learning disability, and autism. He carries two inherited variants in RNU2-2, a small nuclear RNA gene outside the coverage of his exome sequencing. - A boy with delayed motor milestones and enlarged calves has a creatine kinase level of 10,100 U/L. At a second visit, an outside lab reports a DMD inversion. Targeted long reads show duplicated segments containing exons 1 and 2 joined head to head. Each configuration made three attempts on each of the 35 cases. Gemini 4 Argon with Pi passed 46 of 105 attempts (43.8%). Claude Opus 5.5 with Claude Code and GPT-6 Astra with Pi each passed 35 of 105 (33.3%). A pass required the diagnosis and management plan to meet every criterion in the case-specific rubric. Timeouts counted as failures. Among the 14 models that passed at least one case, 2,097 runs failed. Data acquisition and diagnostic workup was the most frequent primary failure category (36%). In 15% of all analyzed failures, the agent failed to pursue the type of data the case required. In another 12%, it omitted a necessary non-sequencing investigation, such as a tissue stain, imaging study, or laboratory test. Paper: latch.bio/raredisease Leaderboard: benchmarks.bio/raredisease Blog: blog.latch.bio/p/raredisease… We think rare disease diagnosis is one of the areas where AI systems have the most potential to make rapid progress and directly benefit patients over the next few years. Incredible work from @chriswzou and team.
5
26
3
71
8,315
Matthew Nemeth retweeted
Thrilled to share new work from our experimental biology lab at @AnthropicAI. Led by world-class biochemist Januka Athukoralage, we built a rapid method for the isolation and structural characterization of vaults. It turns out these massive structures are not just found in eukaryotic cells, but also in prokaryotes and even phages. We solve the structure of one of these phage vaults, and then demonstrate we can engineer additional repeats into the wall to make it bigger. One thing I can't get over about these vaults. They seem to be widespread across life - we assembled them from 8 different bacteria and bacteriophages. But they are just so massive - bigger than ribosomes, AAV, and even some phage capsids. Why devote so much energy to making these huge complexes? They have to be doing something useful - I hope that's the question that gets answered next. Amazing collaboration with @PeterHYoon, @amirshanehsaz, @eric_kabrams, and @mgdurrant! Grateful to our team for making Claude an amazing labmate and to Anthropic for supporting this type of work on fundamental biology. Natural mysteries abound - let's get to solving them! Links here: Blog: anthropic.com/research/inves… Technical report: www-cdn.anthropic.com/76e55f…
33
138
17
850
54,274
Matthew Nemeth retweeted
What do vault particles do? Scientists have been asking this for 40 years. Our biology team at Anthropic set out to tackle this long-standing open question in cell biology, working with Claude. We haven't uncovered their biological function yet, but we've found something new: vault proteins in prokaryotes that assemble into complexes that are larger than human vaults. And we show that we can engineer their shapes to produce even larger nanocapsules! The mystery of vaults goes deeper than anyone knew... Read the blog and technical report to learn more! Blog: anthropic.com/research/inves… Technical report: www-cdn.anthropic.com/76e55f… Very grateful to have such fantastic coauthors who made this research project possible, including Januka Athukoralage, @ntperry13, @PeterHYoon, @ekabrams, and @amirshanehsaz. A big thank you to Dr. Leonard Rome, who discovered vaults in 1986, for providing feedback on the technical report. Thank you to @FutureHouseSF for supporting me as a research fellow when I first started learning about vaults. And, of course, grateful to my many @AnthropicAI teammates that work tirelessly to make Claude the best model for biology! Video and sound made by Claude (with artistic direction from @ntperry13). Showing the new bacteriophage-derived vault structure that we solved.
48
65
21
517
41,840
Matthew Nemeth retweeted
Stoked to be SemiAnalysis’s exclusive biology partner for GDPx. Very early days for AI in biology. Recommend digging into the trajectories: gdpx.semianalysis.com/domain… Read more about our contributions: gdpx.semianalysis.com/domain… Exceptional folks working on this project @arjunomics @maxkan @AndrewMegalaa
Today we’re releasing GDPx, a benchmark for measuring agentic capabilities across the most valuable domains. GDPx is a true frontier-level benchmark: even the best model scores just 12% (1/7)🧵
3
11
66
4,486
Matthew Nemeth retweeted
What will drug discovery look like when intelligence is abundant? Our team at @generate_biomed began with this question, and we’re excited to share some results! We took on in vivo CAR-T and designed a scaffold 70% smaller than ones in the clinic, with 3x the potency 🧵
13
49
8
338
33,587
Matthew Nemeth retweeted
The same idea shaped the composition of our team. As intelligence becomes abundant, we believe a small team of high-agency scientists can accomplish what was once unimaginable. We have no meetings and no management layers, and we started with no software.
1
1
9
706
One day they’ll tell stories of the Dyno Mafia
Life update: after an incredible year at Noetik, I’ve joined the OpenAI Foundation (@FoundationOAI) to help create its "Public Data for Health" program. The OpenAI Foundation is a well-capitalized philanthropy, and a meaningful share of its funds will be committed to building and opening up the datasets necessary to massively accelerate biomedical research. Some of our grants will go toward efforts to relieve known data bottlenecks, but others will be more speculative, made on the premise that artificial intelligence is currently reshaping how scientific discovery happens, and that this reshaping will surface fundamentally new data bottlenecks of its own. We have a long to-do list ahead of us, and I’m ecstatic to be joining @JacobTref on this effort! On writing: I’ve spent the last few years covering the intersection of AI with many, many subfields of the life sciences at owlposting.com, and it will continue + remain independent. Many exciting essays and podcasts are planned! Lastly, I remain extremely optimistic on Noetik and am very thankful for my time there. Consider following @Ronalfa and @recursus to stay updated on their efforts!
4
443
Matthew Nemeth retweeted
Life update: after an incredible year at Noetik, I’ve joined the OpenAI Foundation (@FoundationOAI) to help create its "Public Data for Health" program. The OpenAI Foundation is a well-capitalized philanthropy, and a meaningful share of its funds will be committed to building and opening up the datasets necessary to massively accelerate biomedical research. Some of our grants will go toward efforts to relieve known data bottlenecks, but others will be more speculative, made on the premise that artificial intelligence is currently reshaping how scientific discovery happens, and that this reshaping will surface fundamentally new data bottlenecks of its own. We have a long to-do list ahead of us, and I’m ecstatic to be joining @JacobTref on this effort! On writing: I’ve spent the last few years covering the intersection of AI with many, many subfields of the life sciences at owlposting.com, and it will continue + remain independent. Many exciting essays and podcasts are planned! Lastly, I remain extremely optimistic on Noetik and am very thankful for my time there. Consider following @Ronalfa and @recursus to stay updated on their efforts!
144
55
15
972
90,139
Matthew Nemeth retweeted
From an internal memo written in 2022
4
2
42
2,641
Matthew Nemeth retweeted
Introducing a new benchmark for long-horizon single cell biology: scBench-Long. Each of these evaluations took a small team of scientists and engineers working in concert around a week to construct, test and harden. 21 evaluations span melanoma CD8 T-cell reactivity, CD8 RNA+ATAC regulatory inference, human-monkey chimera development, KRAS-driven lung tumor aging, and lethal COVID-19 lung pathology. Across 1,068 completed trajectories, the strongest agent system - Opus 4.8 + CC - scored 25.4%. Long-horizon benches are significantly more difficult to build and interpret, requiring new research in interpretation tools, data infrastructure, grading methodology, etc. But the results are rewarding. These tasks are close to the end results you might see published in a paper or used to make a decision in an industrial research project, so watching agents make meaningful progress is very exciting. Sometimes they reach correct scientific conclusions in unanticipated and creative ways. We attempted to describe the motivation and main results in the quoted writeup. Full manuscript and interactive examples linked below. Long roadmap ahead and a lot more to come.
5
24
81
8,495
Matthew Nemeth retweeted
The analogy with Moore’s law is weak because the curve tracks progress in a measurement tool weakly coupled, and sometimes irrelevant, to most end goals in biotech. Even in the subset of cases where it’s directly used, such as prenatal screening, still don’t see broad adoption of the best sequencing technology (which you would expect immediately) because the rate-limiting step is elsewhere: generally regulation, reimbursement. The curve is often used to paint a misleading picture of where the bottlenecks are, and can be used to sell or build products around constraints that are not actually rate-limiting.
Hard to think of another chart in biology more misunderstood and misappropriated
6
5
3
53
11,070
Matthew Nemeth retweeted
I have long written and spoken about the many ways US immigration policy harms international students and scholars. This has been true for as long as I can remember, including when I first came to the US on a single-entry student visa more than 20 years ago under a process formerly known as muslim registry program. But the current administration has gone much further, through arbitrary policy changes, travel bans, and broad visa processing pauses that leave folks unable to work, travel, or train. These policies affect a minority of scientists, and in the current state of the world they can be easy to overlook. But we should not let that happen. I wrote about the quiet loss of Iranian scientific talent in US labs for @TheScientistLLC: the-scientist.com/the-quiet-…
320
107
33
482
281,179
Matthew Nemeth retweeted
Biology is the next agentic frontier after coding. Anthropic is aggressively improving their models on routine data analysis with careful attention to nuances of different assay types. Opus 4.8 is noticeably better at single cell / spatial analysis. We have already rolled it out to customers across pharma and academia. Cool to see our benchmarks on the system card.
Introducing Claude Opus 4.8: it builds on Opus 4.7 with sharper judgment, more honesty about its own progress, and the ability to work independently for longer than its predecessors. Available today at the same price.
8
38
5
283
22,421
Matthew Nemeth retweeted
Introducing SpatialBench-Long, a benchmark for long-horizon spatial biology. Agents must recover biological claims from raw data and realistic experimental context without prescribed methods. 24 evals span primary tumors, organoids, xenograft models, lineage-tracing systems, and aging/intervention biology. The best agents score 11.1%.
1
16
2
67
8,753
Matthew Nemeth retweeted
Figuring out how to benchmark agents on realistic biology research has quickly become one of my favorite types of engineering work. You work with scientists to get to the core of some biological claim, precisely assembling raw data/prior literature/experimental context in a little 'world' for an agent to stumble around in - only for its behavior to challenge what you thought to be true and to force you to deeply introspect empirical human behavior. Doing this well gets considerably more difficult as we better approximate and climb long horizon scientific work: the type one might publish in the results section of a paper or build a drug program around. Been working on a project in this direction for the past few months and excited to drop tomorrow.
2
12
90
4,619
Matthew Nemeth retweeted
Thrilled to share I've started at @Stanford's Department of Pathology (@StanfordPath) in addition to @ArcInstitute. Looking forward to a shorter commute after 5 years at @BerkeleyBioE and embarking on daring new projects We're recruiting multiple postdocs and technical staffšŸ‘‡
28
32
3
640
54,273