📄 arXiv:2610.21006 You ask a language model whether top-model API prices halve by 2027. It writes a ten-page rationale for its conclusion. What are the cruxes of that forecast? Problem: only by contrasting forecasts can you validate a crux matters for the conclusion! 🧵

ALT Flowchart. One forecasting question, whether top-model API prices halve by 2027, fans out into 30 forecasts from Claude Opus 5.5, ranging 7-15% yes. Pairs of forecasts are contrasted, each pair yielding crux questions. Paraphrases are merged into a column of 86 cruxes. Headline: cruxes are properties of the distribution of forecasts, not of any one forecast.
Slightly better: have the model make one forecast, then reread its reasoning and report its cruxes. Also cheap, and it works a lot better. But one forecast is one sample, and it misses many of the cruxes that move the model's forecasts from sample to sample.
Paper: arxiv.org/abs/2610.21006
Code: github.com/davidreber/foreca…
Data: github.com/davidreber/foreca…
With James Reber, @universeinanegg and @victorveitch.
David Reber retweeted
Encoding semantic similarity into vector similarity is a basic building block of AI systems. When studying this, we usually just think of a single fixed notion of vector similarity (e.g., cossim). However, semantic similarity is fundamentally *contextual*---a fixed similarity notion shouldn't suffice! Turns out the information geometric lens both greatly clarifies this apparent contradiction and also yields a simple and effective technique for getting contextual semantics with vector search.
@_AkashKannan_ @KihoPark_
arxiv.org/abs/2610.06919
[Hot take] Your Causal Variables Are Irreducibly Subjective
Mech interp keeps "finding the bug" in earlier interventions, but the real problem is upstream: your variable definitions are subjective choices no formalism can validate.
open.substack.com/pub/cichic…
4
112
David Reber retweeted
Democracy depends on an informed electorate. But political issues and ballot measures can be confusing, obscuring the effects of one outcome versus another. Moreover, politics is personal. Once we make an initial decision about an issue, it can be hard to change our mind or see things from “the other side.” And talking about issues with those with whom we disagree can be challenging, especially when the conversation feels more like a debate than a discussion.
Technology offers ways to alleviate these difficulties, but not without introducing problems of its own. The Internet and social media promised new ways for people to connect, discuss issues, and learn from each other. But in practice, both often inflame passions, solidify echo chambers, and spread misinformation. More recently, LLM chat interfaces may help people stay informed through personalized access to information, but mainstream chatbots tend to match user beliefs rather than clarifying or challenging them.12 Without the kind of pushback you’d encounter in a discussion between disagreeing friends, chatbots are ill-suited for helping people think through political issues in a balanced way.
The goal of CivicChats is to address these shortcomings. Starting with ballot measures, CivicChats helps people better understand political issues through three different modes of discussion: a Q&A mode for understanding what a measure does and what’s at stake, an argumentative mode that presents competing views to your own, and a reflective mode that helps you examine and develop your own thinking.
David Reber retweeted
Most mech interp work relies on activation patching, but patching activations destroys previous computation.
What if we want to use a different mechanism on the same residual stream? We propose dynamic weight grafting to interpret finetuned model weights.
🧵 1/n
David Reber retweeted
Predictive Interpretability > Mechanistic Interpretability
Prompting is the best method of scientific inquiry we have to study LLMs
It's socially devalued because it doesn't include much d/dx,O(),etc.
come to poster #3503 to talk about this or anything re: the science of LLMs
Excited to present our work on LLM-assisted explainability at #ICML2025!
🖼️ Poster: Wednesday, 11:00am–1:30pm (#E-2902)
📄 arxiv.org/abs/2410.11348
w/ @seanrson @toddknife @ggarbacea @victorveitch
If you're using LLMs to generate counterfactual pairs, rewrite twice—not once!
David Reber retweeted
Semantics in language is naturally hierarchical, but attempts to interpret LLMs often ignore this.
Turns out: baking semantic hierarchy into sparse autoencoders can give big jumps in interpretability and efficiency.
Thread + bonus musings on the value of SAEs:
David Reber retweeted
1/n You may know that large language models (LLMs) can be biased in their decision-making, but ever wondered how those biases are encoded internally and whether we can surgically remove them?
🧵 RATE: Score Reward Models with Imperfect Rewrites of Rewrites
1/ How do you measure whether a reward model incentivizes helpfulness without accidentally measuring length, complexity, etc?
Rewrites of rewrites give good counterfactuals, without needing to list all confounders!
4/ But how can we know we’re actually getting counterfactuals? We use a novel synthetic experiment to test how much our rewrite method is affecting known off-target correlates: induce a distributional shift, and see if the reported ATE changes! (It shouldn’t, and RATE passes ✅)
1
1
113
5/ Shout out to my amazing collaborators @seanrson @toddknife @ggarbacea @victorveitch give them a follow!
Dive in to the full paper at arxiv.org/abs/2410.11348
3
110
David Reber retweeted
Reward models are a key ingredient to understanding LLMs. But what they *actually* reward can be a mystery.
Turns out we can measure this directly! The trick is to use LLMs to produce "rewrites-of-rewrites" datasets to measure attributes in isolation.
arxiv.org/abs/2410.11348
David Reber retweeted
Are LLMs just doing next token predictions? It is believed that if an LLM can accurately predict the next tokens in a Wikipedia entry, it essentially "learns" the information.
But do pre-trained LLMs actually need to understand context sentences to solve this task? The answer is no!

















