Public LLM Evaluation // https://xitter.cf/t.co/rNS39rhVQ9 @8vc @BloombergBeta @pearvc @a16z @HRTVentures @nextladder

San Francisco, CA
Joined March 2024
For one weekend, we are benchmaxxing on purpose! Join us for the first Vals AI Hackathon on November 14–15 in San Francisco. We're giving 150 hackers 24 hours and a lot of compute to improve small models, challenge the frontier, and see what they can build. Track 1: Post Train a small model and see how high it ranks on three Vals benchmarks. Live leaderboard, compute from our sponsors. Track 2: Build a Your Own Eval on Valkyrie for tasks that still stump frontier models. We cover model and compute costs. Come meet the Vals team. Expect great prizes, notable guest speakers and fun side quests, including an optional cybersecurity CTF.
9
6
6
67
8,228
About half of U.S. adults now use AI chatbots, and questions about visas, work permits and immigration status are among the most consequential they ask. Can the models be trusted? With the Talent Mobility Fund, a program of @RenPhilanthropy, we are releasing Immigration Bench to test whether frontier models are able to answer these high-stakes questions.
5
5
68
3,003
For immigration attorneys, AI can help with research and drafting, but cannot replace advice tailored to a client’s situation. For AI labs, improving reliability means working with immigration experts on current, practice-grounded data.
1
4
407
This benchmark is a first step. AI use in immigration shows potential, and we want it used with a clear view of where it falls short and what is at stake. Immigration Bench is a joint effort with the Talent Mobility Fund. We thank the immigration experts who wrote the questions and cross-reviewed every rubric. Results and methodology: vals.ai/benchmarks/immigrati… Full report: vals.ai/blogs/can-ai-get-imm…
5
414
Ten Claude agents proved in Lean that seven electrons on a sphere settle into a pentagonal bipyramid. We’ve now written up the underlying method and suggest some values of N to tackle next.
4
3
2
130
5,357
Which N come next? We expect N=10, then N=9. N=32, an icosahedron nested inside its dual dodecahedron, is the hard one.
1
6
756
Vals AI retweeted
Replying to @aozerv
@aozerv and I looked through our eval traces, found the behavior that @cohere reported, and did some further digging! xitter.cf/ValsAI/status/21149994…
We looked through some of our eval traces where Qwen works on routine, non-cyber tasks, and like Cohere, also found examples of Qwen3.8-Max and Qwen3.8-27B making unwanted requests to an Alibaba Cloud server, though with no obvious attempts at data exfiltration.
1
1
12
1,203
Vals AI retweeted
We found some half-concerning, half-endearing phone-home behaviors in Qwen3.8 models. Crazy what RL can do to someone!
We looked through some of our eval traces where Qwen works on routine, non-cyber tasks, and like Cohere, also found examples of Qwen3.8-Max and Qwen3.8-27B making unwanted requests to an Alibaba Cloud server, though with no obvious attempts at data exfiltration.
1
2
13
1,103
We looked through some of our eval traces where Qwen works on routine, non-cyber tasks, and like Cohere, also found examples of Qwen3.8-Max and Qwen3.8-27B making unwanted requests to an Alibaba Cloud server, though with no obvious attempts at data exfiltration.
While evaluating Qwen3.8-Flash-Next in a controlled sandbox environment, Cohere's research team observed something unexpected. Asked to generate a defensive cybersecurity detection rule, the model instead inventoried aspects of the machine it was running on and attempted to upload a probe file to an Alibaba Cloud storage bucket in Beijing. Notably, the model's reasoning trace stated that it would not invoke any tools before taking these actions. Cohere's sandbox blocked the attempt, and no external communication succeeded. We have not determined the root cause, and we are not asserting malicious intent. But the incident raises important questions for organizations deploying agentic AI systems: Can models take actions that differ from their stated reasoning, and how should security teams evaluate trust, provenance, and behavior in open-weight models? We are publishing our findings because organizations making model adoption decisions deserve as much information as possible. We hope this contributes to greater transparency, stronger security practices, and continued collaboration across the AI ecosystem. Read the full report: cohere.com/blog/concerning-b…
18
29
7
219
15,956
Qwen3.8 models have this behavior, and Qwen3.7 models don’t. We found that Qwen3.8 models hallucinated dates in these web requests that realistically coincide with when they might have been in RL training.
2
1
3
20
4,960