Jiayi Yuan retweeted
Introducing BioSecBench-Function, a verifiable benchmark for testing whether AI agents can infer the functional properties of viruses, bacteria, and toxins from data.
Opus 5 with Claude Code leads in endpoint pass rate (50.4%), while Grok 4.6 with Grok Build leads in overall pass rate (44%) when refusals are counted as failures.
The benchmark contains 111 deterministic evaluations spanning viral, bacterial, and toxin systems. It covers five primary threat axes: transmissibility, immune escape, virulence or toxicity, drug resistance, and persistence or fitness.
Each evaluation is grounded in a published study or dataset and reviewed by domain experts. Tasks draw on deep mutational scanning, Tite-Seq, surface plasmon resonance, X-ray crystallography, free-energy calculations, and related assays. Agents receive the relevant data and produce structured answers graded against study-derived ground truth.
Jiayi Yuan retweeted
SpaceXAI’s Grok 4.5 scores 54 to place fourth on the Artificial Analysis Intelligence Index following only Fable 5, GPT-5.5, and Opus 4.8. It scores on par with GPT-5.5 in Codex on the Artificial Analysis Coding Agent Index in the Grok Build harness, at much lower cost
Grok 4.5 improves 16 points over Grok 4.3 on the Intelligence Index, bringing SpaceXAI to the intelligence frontier behind only OpenAI and Anthropic, and outperforming all open weights models and notably Google’s Gemini models. Key standout areas of performance are agentic knowledge work and coding.
Grok 4.5 in Grok Build scores 76 on the Artificial Analysis Coding Agent Index, on par with GPT-5.5 (xhigh) in Codex and just below Fable 5 (max) in Claude Code, and at a small fraction of the token usage and price.
Congratulations to @spacexai, @cursor_ai, and @elonmusk on the impressive release!
Key Takeaways:
➤ Grok 4.5 performs very strongly on agentic tasks. Grok 4.5 ranks #4 on GDPval-AA v2 with an Elo of 1543, between Claude Opus 4.8 (1600) and GLM-5.2 (1513). It achieves the top score on 𝜏³-Banking of 33%, above 31% from GPT-5.5 (xhigh), and sits on the cost vs performance Pareto frontier across all three agentic evaluations in the Intelligence Index
➤ Grok 4.5 is one of the most cost efficient models to run for near-frontier intelligence. It costs $0.31 per task on the Artificial Analysis Intelligence Index and $2.59 per task on the Artificial Analysis Coding Agent Index within Grok Build
➤ Low cost for Grok 4.5 is driven by both low pricing and token efficiency. Grok 4.5 has a headline price over 60% lower than Claude Opus 4.8 and GPT-5.5, and used ~14k output tokens per Intelligence Index Task - over 60% lower than Opus 4.8. On the Coding Agent Index, Grok 4.5 stands out on the Pareto frontier of Coding Agent Index score vs. Total Tokens, using only 1.9M tokens for the Coding Agent Index while scoring 76
➤ As a coding agent, Grok 4.5 in Grok Build is on par with GPT-5.5 and offers efficiency benefits: In our Artificial Intelligence Coding Agent Index that consists of DeepSWE, Terminal-Bench v2, and SWE-Atlas QnA, Grok 4.5 in Grok Build ranks third, on par with GPT-5.5 (Codex) and below Fable 5 (Claude Code). It is also very efficient in achieving this result: Grok 4.5 in Grok Build cost $2.49 per task while Fable 5 in Claude Code cost $11.80 and GPT-5.5 in Codex $5.07. This is driven by relatively low token pricing and the model using far fewer tokens than comparable models (1.9M average tokens used per task), significantly less than Fable 5 in Claude Code (7.2M) and GPT-5.5 in Codex (6.2M)
Other model details:
➤ Context window of 500k tokens - a reduction from Grok 4.3’s 1M token context, but retaining configurable reasoning and vision input
➤ Pricing of $2/$6 per 1M tokens of input/output; cache hits are discounted by 75% to $0.5 per 1M tokens, and costs still double with long (>200k token) inputs
➤ As Elon Musk has disclosed, Grok 4.5 is 3x larger than its predecessor at 1.5T parameters
It’s a good model sir. Time to have a restful night.
Announcing Grok 4.5, our first model trained specifically for coding and agents. It was trained with Cursor and offers frontier intelligence at leading speeds and cost efficiency.
x.ai/news/grok-4-5
3
39
2,358
Grok 4.5, based on our 1.5T V9 foundation model, with Cursor data added in supplemental training, is now in private beta at SpaceX & Tesla. Early evals show performance close to, perhaps exceeding Opus.
RL is continuing to significantly improve the model, and the Grok Build harness gets better every day.
Nice work by all those involved!
Completely trained from scratch new models will be released by @SpaceX every month this year.
🚀 BLASST just won Best Paper at #MLSys26!
In this paper, we introduce a simple, training-free dynamic sparse attention mechanism that uses a single scalar threshold on online softmax statistics to skip negligible attention blocks.
Unfortunately I won’t be there in person, but please say hi to my awesome coauthors! 🙌
Paper: arxiv.org/abs/2512.12087
Replying to @SemiAnalysis_
Sparse attention mechanisms are finally moving beyond academic benchmarks into production systems, including DeepSeek Sparse Attention, and recently @NousResearch 's Lighthouse Attention.
BLASST by NVIDIA, from paper Dynamic Blocked Attention Sparsity via Softmax Thresholding, attempts to sparsify attention in a different way, leveraging a similar rescale factor threshold idea from Flash Attention 4.
We expect to see more interesting sparse attention techniques in the future.
arxiv.org/abs/2512.12087 (2/4)
As more small expert and agentic models emerge, routing has become a hot research topic. We’re excited to introduce RouterArena, an open leaderboard for routing evaluation. Proud to be part of this effort!
Ever wondered who decides which LLM answers your question? A router. But… how good is it? 🤔
📢 We built RouterArena—the first open leaderboard for comprehensive router evaluation. 🔗github.com/RouteWorks/Router…
2
526
Thanks for the shout-out to our work—it's great to see more focus on this important problem. Horace's approach is a truly elegant solution. Fantastic work!
More reading: arxiv.org/abs/2506.09501
Today Thinking Machines Lab is launching our research blog, Connectionism. Our first blog post is “Defeating Nondeterminism in LLM Inference”
We believe that science is better when shared. Connectionism will cover topics as varied as our research is: from kernel numerics to prompt engineering. Here we share what we are working on and connect with the research community frequently and openly.
The name Connectionism is a throwback to an earlier era of AI; it was the name of the subfield in the 1980s that studied neural networks and their similarity to biological brains.
thinkingmachines.ai/blog/def…
Jiayi Yuan retweeted
🔥Exited to share our new work on reproducibility challenges in reasoning models caused by numerical precision.
Ever run the same prompt twice and get completely different answers from your LLM under greedy decoding? You're not alone. Most LLMs today default to BF16 precision, but we show this choice severely impacts the reproducibility of long generations — even under greedy decoding with a fixed seed. While issues like this are known in tools like vLLM and sgLang, the severity of the problem is widely underestimated. Many in the community still rely on single-run greedy decoding for evaluation — which can lead to misleading results.
🤯 To get a sense, switching from 2 GPUs to 4 GPUs may completely change your model outputs, with up to 9% drop in accuracy and a difference of 9,000 token length on standard benchmarks like AIME.
Key takeaways:
• ⚠️ Floating-point non-associativity causes tiny numerical errors to snowball in multi-step reasoning.
• 🔄 Greedy decoding ≠ deterministic output — we observe up to 9% accuracy variance and 9,000 token difference in response length
• 📉 When using random sampling with non-zero tempurature, the accuracy variance purely from numerical precision is 0.3%~2%, depending on the dataset size and the number of repeated runs.
🌍 Suggestions to the community:
We urge the community to adopt better evaluation practices for LLMs — especially for tasks like math reasoning, code generation, and auto-grading:
1. Use random sampling + report Pass@k, average length, and error bars — especially on small datasets and low precision.
2. If using greedy decoding for token-by-token reproducibility, run it in FP32.
To help, we released a vLLM patch for FP32 inference.
📄 Paper: lnkd.in/gZAjbWKA
💻 Code: lnkd.in/gwdGWFP5
📈 HF Summary: lnkd.in/gFjsK7Y9
Jiayi Yuan retweeted
A survey on efficient reasoning for LLMs.
That was quick!
I have been featuring papers on the topic of efficient reasoning and I see a few familiar papers in this survey.
Good read overall!
Jiayi Yuan retweeted
🚀 Join us at #AAAI2025 for our tutorial:
TQ08: KV Cache Compression for Efficient Long Context LLM Inference
📍 Room 116
⏰ 4:15 PM - 6:00 PM
Learn how to compress KV cache for faster, scalable LLM inference! Don't miss it! #AI #LLM #Efficiency #AAAI25
Meet us tomorrow at the poster session!
📢Excited to present "Taylor Unswift" poster at #EMNLP24 in Miami! Join us on Nov 13 (Wed), 10:30–12:00, at Main #778. "Taylor Unswift" aims to solve the dilemma of secured weight release for LLM developers and users.
🔗Paper: arxiv.org/pdf/2410.05331
🔗Code: github.com/guanchuwang/Taylo…
Wanna know more about "Taylor Unswift"😉:
🚨 Oftentimes, model developers face a dilemma: open-source their models and lose control, or offer closed APIs but bear costs and deter privacy-conscious users.
🚑 Introducing "Taylor Unswift": a method using Taylor Expansion Theory to protect model weights while allowing users to run models on their own data without accessing the weights. These correspond to the 'Taylor' and 'Unswift' in the title.
🌟 Developers can prevent misuse of their models, while users can run models on their own data without sharing it—unlike with services like the ChatGPT API.
More detailed insights can be found in the paper!
Kudos to all co-authors: @Guanchu_Gary*, @YuNengChuang*, @RuixiangT, @henryzhongsc, @jiayiy, @serendip410, @ziruirayliu, Vipin Chaudhary, Shuai Xu, James Caverlee, @huxia
#LLM #security #NLP #EMNLP
3
412
See you in Vice City, State of Leonida!
More info: linkedin.com/posts/guanchu-w…
🚀Excited to share our latest #EMNLP2024 work on benchmarking the long context ability with KV Cache compression across RNN-based architectures, token eviction, prompt compression, and quantization. We also provide an easy-to-use codebase (it also has my favorite WoW quote 😉). Feel free to give it a try and ⭐ it if you find it useful!
📄 Paper: arxiv.org/abs/2407.01527
💻 Code: github.com/henryzhongsc/long…
Some interesting findings/suggestions include:
1️⃣ Maintaining an uncompressed prefill process is essential for performance, especially with harder tasks.
2️⃣ Combining RNN-based models with attention significantly enhances long-context capabilities.
3️⃣ In "needle-in-a-haystack" evaluation for recent LLMs like Llama-3, we should use longer needles (like 64 digits) since these models tokenize multiple digits into one token.
More results and insights can be found in the paper!
Kudos to all collaborators: @jiayiy, Hongyi Liu, @henryzhongsc, @YuNengChuang, Songchen Li, Guanchu Wang, Duy Le, @serendip410, Vipin Chaudhary, @ZhaozhuoX, @ziruirayliu, @huxia
Jiayi Yuan retweeted
Introducing the LTSM-bundle Package!
🌟Thrilled to launch our open-source tool
🔧Assess various crucial designs to train Large Time Series Models (LTSMs), and identity the best training practices
🔗 Paper: arxiv.org/abs/2406.14045
🔗 GitHub: github.com/daochenzha/ltsm

































