Can an LLM act as a selective model of a GPU during evolutionary search, by reasoning + forecasting a kernel’s runtime but deferring to a GPU when unsure? We produced 12k kernels + runtimes from evolutionary search, costing 400M reasoning tokens + 600 GPU-hours to answer this. In our work GPU Forecasters, we study language models as selective surrogates for GPU kernel optimization. 1️⃣ Off-the-shelf LLMs can forecast how a GPU responds to a candidate kernel with non-trivial accuracy. If we rank candidates by these predictions and measure only the top 10% on a GPU, the fastest kernel we find is within 20% of the best in the pool. 2️⃣ We want LLMs to not just be accurate but also calibrated, so that we can use their uncertainty for selective prediction: during search, we should trust only confident forecasts and verify less confident forecasts by sending them to the GPU. 3️⃣ We train an open-weights surrogate (GPT-OSS-20B) with RL to improve both accuracy and calibration. Calibration-shaped rewards improve both confidence reliability and ranking ability, while correctness rewards alone do not. 4️⃣ Inside a real kernel search, the surrogate finds faster kernels than an equal-GPU-budget baseline by considering more candidates per measurement. 5️⃣ We release 12,388 LLM-generated GPU kernels with measured runtimes spanning 118 operations, CUDA and Triton backends, 3 GPU types, taking 400M tokens + 600 GPU-hours to produce. This dataset can be used for analyzing LLM-driven evolutionary program search dynamics, post-training LLMs for kernel code generation, and things we didn’t get a chance to explore, like training reward models! Thread 🧵👇

Error-targeted learning is a direction I'm excited about for agentic post-training. The idea is to find mistakes an agent is making, and focus compute on training away those mistakes. It's efficient + tells you how training is changing your agent. We show how to make it work for a strong, recent terminal agent policy (Qwen3.5-based), and show training away repeated misbehaviors is a viable path to capability improvement. One of the reasons I like this paradigm so much is that it seems important for democratizing agentic post-training. Doing thousands of rollouts on long-horizon tasks in hundreds of environments is expensive, and it's hard to predict what the outcome of that training will be. In contrast, error-targeted learning lets you decide what errors you want to fix, and then you can watch those errors go away during training. It's very legible and accessible. 🧵👇

Strongly agreed. Reminds me of this perennially useful post from @jsuarez on how games aren’t toy envs that should be dismissed as being unrealistic: puffer.ai/blog/game-rl/

it's so bad it's good. Tempted to keep on reading to see what juicy insanities are going to turn up next in the slop

Beyond desk rejection at the AC's discretion, I don't think there are any permanent consequences

i had to sleep with some lights on after seeing this at midnight

the world needed us to stop the tide, thank you for your service

@adilsoubki @jakedineenasu

yeah there has to be some sort of rate limit for this. i don't see how this can continue

so the rumors are true... there are in fact human-written papers at iclr this year

i'm not sure what their endgame is tbh… i would really like to understand. It seems they think the "published at iclr" tag is some magic label that will give credibility even if the paper itself is very obviously low quality

did you remember to write "make no mistakes"? very important

incredible. the next step is placeholders in the experimental results as well