
PhD student @StanfordAILab @StanfordNLP @Stanford advised by Tatsunori Hashimoto and Tengyu Ma. Prev: @OATML_Oxford @CompSciOxford
Stanford, CA
Joined September 2020
- Tweets170
- Following884
- Followers1.2K
- Likes489
Pinned Tweet
When LLMs are unsure, they either hallucinate or abstain.
Ideally, they should clearly express truthful confidence levels.
Our #ICML2024 work designs an alignment objective to achieve this notion of linguistic calibration in *long-form generations*.
arxiv.org/abs/2404.00474
🧵
Neil Band retweeted
How can an LLM self-improve at theorem proving? Inspired by STP (Dong & Ma, 2025), we propose a theoretical framework for understanding prover-conjecturer self-play algorithms. arxiv.org/abs/2606.01861. NeurIPS 2026, Sydney Poster Session 2, Tue 5-8pm @ Hall 1-4. 1/8
Neil Band retweeted
Anthropic and OpenAI will watermark their responses.
Yet this violates user privacy and incentivizes users to paraphrase away the watermark. A lose for both sides!
Instead, we rethink the incentive structure, and propose to shift the AI provenance goal from individual tracking to estimating aggregate AI usage.
We develop locally differentially private watermarks which provably limit the evidence that any individual text can contribute to detection, while still enabling accurate estimation of population prevalence of watermarked texts.
Neil Band retweeted
AI is using more and more of the world’s energy. But how much economic value are we getting from AI?
We’re introducing Joules-to-GDP (J2G), a framework that measures how well AI turns energy into economic value and shows which tasks, models, and hardware give the best return.
Across 44 occupations in finance, engineering, law, administrative work, and more in @OpenAI's GDPval, we arrive at three findings:
1) The tasks with the highest economic value are not the tasks with the highest economic value per joule. For example, CAD design earns ~10% more per task than building websites, but the latter earns ~27% more value per joule.
2) The most powerful model isn’t always the best use of energy. Gemma 4 31B delivers 56% of GLM 5.2’s net value with 23% of the energy, for 2.5x the value per joule.
3) More chips don’t guarantee proportionate returns. Moving the same model from 1 to 8 NVIDIA H100s uses 3.2x the energy but cuts completion time by only 22%, with about the same value.
With @SnorkelAI, we’re also releasing GDPWorks, a benchmark for economically valuable tasks across 36 additional occupations.
Joint work co-led by @amirzeinaalii and @Avanika15, with @Mersad_Abbasi, @Jon_Hartley_, John Hennessy, @HazyResearch, and @Azaliamirh.
More details below 🧵
Neil Band retweeted
Scaling laws depend on a metric that improves gradually with scale in order to extrapolate to a large target scale. We are interested in the accuracy metric, but you might get zero accuracy for small models. You can't extrapolate from a bunch of zeros.
Key idea: introduce a *hint* level, and fit a joint scaling law Accuracy(scale, hint) = ... More hinting makes the problem easier in a controlled way, so accuracy is no longer zero, allowing for extrapolation! Learn more below.
Can we forecast benchmark accuracy from models that score zero?
We (co-leads @LukeBailey181 @rckpudi) turn a benchmark into a series of easier hinted versions. Jointly modeling scale C and hint level H enables forecasting GSM8K acc (H=0) within 3 pts from 5600x less compute 🧵
Hinting is a really cool new primitive for building downstream task scaling laws and more
Awesome work by @suze_va @LukeBailey181 @rckpudi and team!
Can we forecast benchmark accuracy from models that score zero?
We (co-leads @LukeBailey181 @rckpudi) turn a benchmark into a series of easier hinted versions. Jointly modeling scale C and hint level H enables forecasting GSM8K acc (H=0) within 3 pts from 5600x less compute 🧵
Neil Band retweeted
arxiv.org/abs/2611.02270
Task scaling law. As small models are not able to show meaningful scores in benchmarks, give them privileged information, and formulate scaling law with strength of information.
Neil Band retweeted
I improved NanoGPT Speedrun from 39.9s → 6.7s (-83.2%), more than every record since 2025 combined (-81.4%)!
The web is full of rare, repeated content that the LM can’t memorize in time. I solve this with retrieval-augmented n-gram hints🧵
Neil Band retweeted
Do you really need to finish every rollout in RL for LLM? Not if you make your critic better—and trust it!
We propose Actor-Critic with Action Chunking (AC2): use a learned critic to score chunks of tokens => only need partial rollouts ⇒ train faster than GRPO!
Neil Band retweeted
Can an LM, starting from random init (!!), learn to generate all of its pretraining data?
Introducing Self-Play Pretraining with Zero Data. Two models start from random initialization: a generator proposes programs for a universal Turing machine and a learner trains on their outputs. We never train on any real data, but see predictable scaling on natural datasets: zero-shot val loss on images, text, audio, and melodies decreases predictably with self-play compute. And the learner develops in-context learning capabilities.
A fun proof-of-concept, co-led with @AdityaCowsik and @KfirDolev and co-authors @gbruno_dl, @ANourya @noahdgoodman, and @YoavLevine.
We're benchmarking frontier LLMs on philosophy with a wonderful interdisciplinary team led by philosopher / lawyer / CS PhD @michaelcheng76
If you're a philosophy student or faculty interested in joining the project, check out philosophybench.org!
1/ Introducing PhilosophyBench from @StanfordAILab @StanfordHCI, the first independent, large-scale benchmark for evaluating AI’s philosophical capabilities.
philosophybench.org
Neil Band retweeted
Curriculum learning changes the data. Progressive stacking changes the model. Human development changes both at once.
What happens when language model pretraining does the same?
We explore this in Curriculum-Guided Layer Scaling (CGLS) 🧵
Neil Band retweeted
Introducing the world's fastest tokenizer implementation, Gigatoken!
Gigatoken is ~500-1000x faster than HuggingFace, and ~100x faster than OpenAI's tiktoken for most tokenizer definitions on most machines.
These baselines are already multithreaded Rust implementations! 🧵
Neil Band retweeted
You're wasting FLOPs when scaling inference compute: by independently sampling parallel attempts, you burn compute rediscovering the same solutions.
Introducing QuasiMoTTo: we scale parallel sampling with correlated samples instead! These samples have higher coverage, are marginally exact draws from the LLM, and can be generated in parallel.
Result: same performance with 25-47% fewer samples in test-time scaling + 50% fewer training steps in RL!
In our new paper, we explore the design space of correlated samplers. Work with co-authors @probablynotaz9 (co-lead), @gandhikanishk, @noahdgoodman, and Emily Fox!
Neil Band retweeted
We open-sourced the code for this project!
You can use it to make synthetic LLM training data for any downstream target.
The code also gives you a minimal example for computing data-weight metagradients through LLM training + evaluation.
Check it out: github.com/TristanThrush/dat…
Neil Band retweeted
What if diffusion models could think ahead instead of being greedy at every step?🤔 We introduce:
Learned Relay Representations for Forward-Thinking Discrete Diffusion Models
Neil Band retweeted
Can a language model learn, end-to-end, what to keep in its own KV cache and what to throw away? Can it learn to forget while it learns to reason?
Deep learning's central lesson: capability emerges from end-to-end optimization, not heuristics/strong inductive biases. But for efficiency, we rely heavily on hand-designed approaches.
🗑️ Introducing Neural Garbage Collection (NGC): we train a language model to jointly reason and manage its own KV cache, using reinforcement learning with outcome-based task reward alone. No SFT, no proxy objectives, no summarization in natural language.
New paper with @jubayer_hamid, Emily Fox, and @noahdgoodman!
Neil Band retweeted
A synthetic data generation method that, when a model is trained on the generated data, it maximizes a certain differentiable objective. e.g. it is possible to make data that engraves a QR code in the weights of an LM head. (Or, more conventional things like translating documents to improve target language loss.)
Neil Band retweeted
New paper!
Want to precisely optimize synthetic training data to do practical or even wacky things?
Dataset Policy Gradients get you there, letting you target any differentiable training or post-training metric. We embedded a QR code in GPT-2’s weights using only training data!
Neil Band retweeted
Our new paper, “End-to-End Test-Time Training for Long Context,” is a step towards continual learning in language models.
We introduce a new method that blurs the boundary between training and inference. At test-time, our model continues learning from given context using the same next-token prediction objective as training.
With this end-to-end objective, our model can efficiently compress substantial context into its weights and still use it effectively, unlocking extremely long context windows for complex reasoning and applications in agents and robotics.
Paper: test-time-training.github.io…
Code: github.com/test-time-trainin…
Tim's an amazing researcher and mentor, go work with him!
I'm so happy to share that I’ll be joining @UofT as an Assistant Professor of Statistical Sciences and Computer Science, with an appointment at the @VectorInst, in 2026!
I'm recruiting postdocs and PhD students: timrudner.com!
Please help me spread the word!
🧵(1/5)
1
474


































