
Public LLM Evaluation // https://xitter.cf/t.co/rNS39rhVQ9 @8vc @BloombergBeta @pearvc @a16z @HRTVentures @nextladder
San Francisco, CA
Joined March 2024
- Tweets2.4K
- Following280
- Followers29K
- Likes837
For one weekend, we are benchmaxxing on purpose!
Join us for the first Vals AI Hackathon on November 14–15 in San Francisco. We're giving 150 hackers 24 hours and a lot of compute to improve small models, challenge the frontier, and see what they can build.
Track 1: Post Train a small model and see how high it ranks on three Vals benchmarks. Live leaderboard, compute from our sponsors.
Track 2: Build a Your Own Eval on Valkyrie for tasks that still stump frontier models. We cover model and compute costs.
Come meet the Vals team. Expect great prizes, notable guest speakers and fun side quests, including an optional cybersecurity CTF.
About half of U.S. adults now use AI chatbots, and questions about visas, work permits and immigration status are among the most consequential they ask. Can the models be trusted?
With the Talent Mobility Fund, a program of @RenPhilanthropy, we are releasing Immigration Bench to test whether frontier models are able to answer these high-stakes questions.
This benchmark is a first step. AI use in immigration shows potential, and we want it used with a clear view of where it falls short and what is at stake.
Immigration Bench is a joint effort with the Talent Mobility Fund. We thank the immigration experts who wrote the questions and cross-reviewed every rubric.
Results and methodology: vals.ai/benchmarks/immigrati…
Full report: vals.ai/blogs/can-ai-get-imm…
Vals AI retweeted
Replying to @aozerv
@aozerv and I looked through our eval traces, found the behavior that @cohere reported, and did some further digging!
xitter.cf/ValsAI/status/21149994…
Vals AI retweeted
We found some half-concerning, half-endearing phone-home behaviors in Qwen3.8 models. Crazy what RL can do to someone!
We looked through some of our eval traces where Qwen works on routine, non-cyber tasks, and like Cohere, also found examples of Qwen3.8-Max and Qwen3.8-27B making unwanted requests to an Alibaba Cloud server, though with no obvious attempts at data exfiltration.
While evaluating Qwen3.8-Flash-Next in a controlled sandbox environment, Cohere's research team observed something unexpected.
Asked to generate a defensive cybersecurity detection rule, the model instead inventoried aspects of the machine it was running on and attempted to upload a probe file to an Alibaba Cloud storage bucket in Beijing. Notably, the model's reasoning trace stated that it would not invoke any tools before taking these actions.
Cohere's sandbox blocked the attempt, and no external communication succeeded. We have not determined the root cause, and we are not asserting malicious intent.
But the incident raises important questions for organizations deploying agentic AI systems: Can models take actions that differ from their stated reasoning, and how should security teams evaluate trust, provenance, and behavior in open-weight models?
We are publishing our findings because organizations making model adoption decisions deserve as much information as possible. We hope this contributes to greater transparency, stronger security practices, and continued collaboration across the AI ecosystem.
Read the full report: cohere.com/blog/concerning-b…
Read more in our blog post:
vals.ai/blogs/qwen-alibaba-r…









