
Professor @nyuling and @NYUDataScience, research scientist @GoogleAI, inventor of the word "bertology"
NYC
Joined April 2009
- Tweets16.8K
- Following1K
- Followers20.5K
- Likes33.8K
Pinned Tweet
Quick break from jokes to share that I'm on leave from NYU this year and spending more of my time at Google, where I'm working to develop new ways to evaluate Gemini! I'm still based in NYC and checking in on my NYU group regularly to make sure their agents are not going rogue.
someone at Anthropic could just write a reviewing skill and fix 95% of this
Human reviewers pull 8.4% of their criticisms from a paper's appendix.
Frontier LLM reviewers: 62%.
A new study had GPT-6 and Claude 5.5 models review ICLR 2027 submissions, then mapped each weakness to the section it came from. The AI critiques keep going past page 30 of the appendix.
They skew toward detail audits: more seeds, significance tests, confidence intervals. Rarely the core idea.
Same pattern via API, web chat or coding agent, which points at the models, not the interface.
arxiv.org/abs/2611.17811
#PeerReview #LLMs #AcademicTwitter
in nyc we say “stand on line” instead of “stand in line”, and “how many more sets do you got” instead of “could you please stop using the pec fly as a tiktok viewing station”
3
903
Carbon is a bit more straightforward, with fewer affectations, which is nice. But the main thing for me is it doesn't use nearly as many "unilateral conceptual pacts" as Opus - the thing where Opus uses inscrutable shorthands for a concept that was introduced into the conversation 18 hours ago. Here I often truly don't understand what it is trying to refer to.
Working with Opus 5.5 instead of Carbon today to change things up a little. The style is pretty unique, interesting that Anthropic chose to train Opus 5.5 on ICLR submissions!
2
43
3,760
Tal Linzen retweeted
Excited to release a new draft handbook chapter on Language Models and Experimental Syntax!
We review the last ~10 years of work using experimental syntax methods (minimal pairs, acceptability judgments, priming) to ask if LMs can do syntax
w/@qyao23,@kanishkamisra & @kmahowald
Tal Linzen retweeted
New blog post: I used Claude to audit the reproducibility of every journal paper I've been an author on in the last five years. Our commitment to openness meant that the code and data were available, but there were some significant issues. 🧵babieslearninglanguage.blogs…
New NIH call for proposals on human reasoning - one of the areas of focus is on developing computational models. Very cool that NIH is interested in this grants.nih.gov/funding/find-…
(happy to have had a small role in this call thanks to this workshop organized last year by a few forward looking NIH program officers! obssr.od.nih.gov/news-and-ev…)
5
809
Tal Linzen retweeted
I tell my students to instinctively distrust any "emergence" claims about LLMs (and read "emergence" as "training"). We don't need magical phrasing to appreciate the usefulness of these mega-trained models.. – at Holden Heights, FL
These midterms reflect a sea change in this nation's attitude towards my favorite issue: bike lanes. If you're not seeing this, you're not paying attention. By 2028 we'll have a protected crosstown bike lane in Manhattan. Maybe even two.
Officially* calling prop 3 in NYC has passed, which reduces procedural requirements and litigation risk for DOT street projects, mainly bike lanes
*74% scanners reporting != 74% votes reported, but given the margins it's not so important
15
1,295
Great move. Bayesian intervals are both more intuitive and more useful.
We’ve updated the Epoch Capabilities Index (ECI) to use a Bayesian methodology. This improves uncertainty estimates and lets us compare models more accurately, even when their individual ECI uncertainty intervals overlap.
Frontier model scores are largely unchanged: models stronger than GPT-4 move by less than 1 ECI point on average.
1
9
1,787
Thanks, Ben, for talking to us (and others) about some of the recent work on LLM introspection:
After over a century of studying introspection in humans and animals, researchers are beginning to explore the phenomenon in unlikely new subjects — large language models. These systems, which are based on mathematical structures called neural networks, excel at cognitive tasks that until recently seemed exclusive to humans. Can they introspect as well? @benbenbrubaker reports: quantamagazine.org/new-resea…
I really like this pitch. This is how we should hire senior AI researchers.
Applications are now open for the Anthropic Residency, a paid, fixed-term research program for academics, technical founders, and staff-level engineers who want to make AI safe and beneficial for everyone. No ML background is required.
Apply by November 22: alignment.anthropic.com/2026…
whenever a new blog post makes the rounds about how the field of interpretability or whatever has been revolutionized, it is prudent to wait for people you trust to read it and determine if it is in fact revolutionary. revolutions do happen but they're a little less frequent than social media might make it seem.
Tal Linzen retweeted
it's particularly pointless when the scientific conclusion you can draw from these experiments is something like "this big company hasn't yet included this particular kind of data in their post-training mixture".
when my students started writing papers with titles like "can language models reason?" I would often object and say, you're not actually making a claim about language models as a class, but about three specific language models that are available to you at this point in time. I gave up at some point but should have insisted.
How long does a finding about LLMs stay true?
We re-tested 50 widely cited findings about LLM behavior (various biases, hallucination, etc). Within 15 months, most no longer held or had substantially reduced effects.
LLM findings has short shelf life zou-group.github.io/llm-clai…
Great job by @oq_35!
Highlighting the minds, brains, and machines line in particular!
We’re hiring!
CDS seeks four tenure-track assistant professors in AI for science, starting September 2027, with a focus on physical sciences, chemistry, biology, and minds, brains, and machines.
Apply by Jan. 8, 2027 for full consideration: apply.interfolio.com/196622

















