We are releasing AutoResearchExam, a benchmark on open-ended machine learning and engineering tasks. Our benchmark covers seven research areas including model training, data curation, AI safety and interpretability. benchmarks.bespokelabs.ai/au… In each task, we give agents 24 hours with a CPU or GPU machine to develop and improve their solutions through experiments and feedback. We measure both speed and quality with a combined score. Our benchmark has a unique feature: testing if agents create improvements that hold up on data they never see. We find that AI research agents often overfit as they try to improve. We see an interesting head-to-head comparison at the frontier: Astra starts the strongest and holds the lead for up to 19 hours but Fable 5.1 catches up and gets the top performing spot in the final hours. Qwen3.8 Max, Gemini 3.8 Flash and Grok 4.6 all sit on the cost-performance Pareto frontier, giving strong options at lower API budgets. Anthropic's Opus and Fable retain nearly all their validation performance on hidden tests, with gaps of 1.1% and 2.9%. Astra's improvement over Sol extends to generalization too, with that gap falling from 6.9% to 1.7%. (1/n)

Congratulations to the team for releasing Frontier CS2.

Glm 5.3 benefits come from post-training alone. Orders of magnitude cheaper than pretraining and can use proprietary data if they are transformed into RL environments.

As agents perform AutoResearch they get fixated on their old ideas and do only incremental innovations. We introduce a simple context management trick that forces the agents to forget everything they have done except the best solution they have found so far. This simple modification in context management allows the agents to get out of their idea basins and come up with more novel solutions. Closely related to the work 'Fork and Flush' and 'Idea-basins' that @DimitrisPapail posted a few days ago.

We are excited to announce that we post-trained NVIDIA Nemotron-3.5-Super-VL to be an autonomous ML researcher. We show that our post-trained model goes from 28% to 35% on AutoResearchExam, matching Qwen 3.8 Max, a model with 20× the total parameters. The tasks are long horizon, open ended AI and ML research: building a faster executor for a transformer compute graph, training a RL control policy, creating new uncertainty quantification methods, etc. The model gets to submit once, and it doesn't see its score until the end. We curated SFT data using Self-context distillation. This means that we let the Nemotron model to solve a problem with hints, then we SFT on these trajectories but without the hints. This gave us +11% on AutoResearchExam. As a next step, we added reflection: We ask the Nemotron model to review its own past attempts and writes concrete lessons learned. Then we SFT on these reflections. Surprisingly, training on these lessons improved the performance to +25% compared to the base model. The hardest part was choosing the right data. We selected traces based on task scores, completion checks, and runtime, then refined the sampling across training rounds to address failures in the base and fine-tuned models. We removed hints and filtered out steps that relied on information available only in those hints. The goal was to teach decisions supported by the task and history the model would actually see at inference. Full write-up: bespokelabs.ai/blog/teaching… Thanks to @NVIDIAAI for early access and collaborating on this.

We are excited to announce Bespoke Nimble-v4, our decision models. Scoring 68.4 on the Decision index, ranking at no1 , +10.50 points over Jev 1.13.

Very interesting ideas that show the importance of context management. Sometimes you just need to forget.

Our Bespoke Nimble system-1 model is now in Ollama, playing OllamaRacer. 🏎️

Dimitris has been proving you can do important AI basic research with a few GPUs and some agent max subscriptions. Some examples: ECHO paper, looped transformers, open mementos for context management come in mind.

Congratulations Qiuyang and to the whole team for getting FrontierSmith accepted to NeurIPS!

benchmarks.bespokelabs.ai/au… Grok 4.7 added to our live leaderboard for AutoResearchExam. It jumps to position no3 at 30Min research, drops to 4, then again at no3 at 4h, landing at position no4 after 24h of AutoResearch. Pretty good performance and cost tradeoff overall!

Thanks Emad for highlighting AutoResearchExam and the role of different harnesses

We lost our CEO for 2 days and then he came back with this awesome 9B model.