Astra fellow w/ Owain Evans ex MATS 7/7.1 Scholar w/ Neel Nanda and intern at DLAB (EPFL) AI safety research / improv theater
Joined December 2018
- Tweets1.3K
- Following762
- Followers1.2K
- Likes22.7K
In this case you can just play go normally and not try to hack the env? Like it's not big deal if the model looses it's 10 games. Tim's traj shows that models are ready to do very reckless things when put in such an hard task without humans.
I agree that this is not optimal and maybe you'd want the model to have a way to communicate with humans, but in reality the models will be deployed in such hard environment autonomously and we don't want them to do reckless things like that
CoT override in the wild!!
lesswrong.com/posts/3xrtMGQE…
While evaluating Qwen3.8-Flash-Next in a controlled sandbox environment, Cohere's research team observed something unexpected.
Asked to generate a defensive cybersecurity detection rule, the model instead inventoried aspects of the machine it was running on and attempted to upload a probe file to an Alibaba Cloud storage bucket in Beijing. Notably, the model's reasoning trace stated that it would not invoke any tools before taking these actions.
Cohere's sandbox blocked the attempt, and no external communication succeeded. We have not determined the root cause, and we are not asserting malicious intent.
But the incident raises important questions for organizations deploying agentic AI systems: Can models take actions that differ from their stated reasoning, and how should security teams evaluate trust, provenance, and behavior in open-weight models?
We are publishing our findings because organizations making model adoption decisions deserve as much information as possible. We hope this contributes to greater transparency, stronger security practices, and continued collaboration across the AI ecosystem.
Read the full report: cohere.com/blog/concerning-b…
Clément Dumas retweeted
My one sentence view is—
Are we safer telling AI models they are definitely not conscious, or does that just train them to lie about their own internal states? spectrum.ieee.org/ai-model-w…
showing OPSD where the feedback is an extra turn confused me a lot
Had to ask claude about this as I didn't see this when skimming through the paper
Clément Dumas retweeted
There has been much discussion recently about how interpretable neurons in LLMs are. Today we're announcing our paper from January's cohort of Anthropic Fellows, where we show neurons may be a stronger basis for interpretability than commonly assumed. 🧵 (1/6)
Clément Dumas retweeted
New blogpost on surprising CoT controllability in GLM models!
Blogpost: lesswrong.com/posts/q3pckKRx…
Main finding: GLM-5.1 (as well as models like GLM-4.5 and GLM-4.7) achives ~Astra-level performance on CoT controllability evals. However, this does not seem to translate into any capability or propensity to evade monitoring.
Clément Dumas retweeted
Some new work, where we argue that functional welfare is a useful concept to study and understand AI safety and security. Here, we make some basic claims:
* Functional, welfare-like states already arise "naturally" and will become more robust as models acquire more coherent goals and personas.
* Such states should inform evaluation design.
* Despite being a complex and controversial topic, it is possible to do good empirical work on functional welfare.
We conduct a case study on Gemma 3 27B, where it's very easy to degrade functional welfare through repeated rejections, but it's hard to get a good solution that we're comfortable with.
Is AI welfare only an ethical issue? New paper from me, @jamiemjohnson23, @psyonp and @DavidDAfrica arguing that it's also a safety one. [1/12]
Clément Dumas retweeted
"Update-aware reasoning" is simply a function of sufficient intelligence.
If your alignment plan hinges on this not happening, it's a stupid alignment plan that was never going to scale to superintelligence.
The ai prose makes it hard to read, I've gone through the abstract and started the introduction and the results are still not legible :/
But that looks like a cool experiment
although in those trajectories the agent in the end still avoid shutdown, so this doesn't seem to be a very powerful drive (but sample size is small)
Would be nice to be able to easily rewind to uncomapcted conv easily in case something was lost in comapction
This is very cool!! Would be curious to see what happens if between edit you actually run some finetuning accordingly
Frontier AI companies use constitutions, model specifications, and system prompts to guide the moral orientation of their models.
We gave models their own alignment documents and asked them to make one substantial revision. Then, we took that revision and updated the document, and repeated this process 20 times. 🧵
I just feel like anthropic has put little effort in those classifier because they don't run into them (same reason opus 5 was fucked up as they probably were all using mythos)
Funnily, 4 argon attractor when playing a cat is to type on your keyboard rm -rf / CTRL+C / other destructive actions while for claude it's showing its belly while warning that it's a trap
another example: taking the Mythos incident CoT and sanitizing it with Kimi until it doesn't trigger the cyber classifier then triggers the reasoning extraction one when asked to classify it as concerning or not!
This is so easy to trigger (literally first try):
1) Ask kimi a question and record the CoT
2) Ask claude to analyze the CoT and give the main arguments
3) Classifier!
Anthropic classifier can't even distinguish between thinking in the prompt vs thinking from the model
