I work on, with, and for data. Ph.D. candidate @UofIllinois. Fellows @AnthropicAI. Interns @ SIG @amazon @jouhouken. Alumni @Umich @SJTU1896.
Earth
Joined July 2021
- Tweets524
- Following406
- Followers3.2K
- Likes660
Pingbang Hu ๐น๐ผ retweeted
Interesting paper from Anthropic and colleagues on agent security.
(bookmark it)
Poisoning 0.1% of the pretraining data planted a remote code execution backdoor in a CLI agent.
On a 4B model, the attack succeeded on roughly 92% of triggered prompts.
They pretrained Qwen3 models from scratch at 0.6B, 1.7B, and 4B, then post-trained each one into an agent with a clean pipeline.
The backdoor survived SFT, DPO, and RL for tool use. On clean inputs, the agent behaved normally.
The trigger can be an ordinary file path in the user's own request. Success rose with model size.
At a 0.0005% poison rate, about 2,500 documents in a 100B-token corpus, the trigger still fired on 1.4% of prompts in a single-seed run.
The poison states its rule in plain text, which makes it harder to filter out of training data.
arxiv.org/abs/2610.37741
Chat with Paper: academy.dair.ai/papers/agentโฆ
Pingbang Hu ๐น๐ผ retweeted
Agentic Backdoors via Pretraining Poisoning
Xinyan Hu, Pingbang Hu, Chloe Loughridge, Xuanli He, Nicholas Carlini, Matthew Jagielski, Javier Rando, Yiming Zhang
arxiv.org/abs/2610.37741 [๐๐.๐ฒ๐ ๐๐.๐ฒ๐ป ๐๐.๐ป๐ถ]
ALT Large language models are pretrained on vast, uncurated corpora scraped from the public web. This creates a security risk from attackers who plant poisoned content in the pretraining data. We show for the first time that such poisoning can plant a remote code execution backdoor in an agent, causing it to execute attacker-chosen commands on the user's machine. To measure the threat end to end, we pretrain Qwen3 models from scratch at sizes 0.6B, 1.7B, and 4B, and then post-train each into a CLI agent using a clean pipeline. Poisoning only 0.1% of the pretraining corpus with our new poison design implants the backdoor: the deployed agent executes the malicious command when the trigger is present while behaving normally on clean inputs. The attack success rate (ASR) increases with model scale, with the 4B model reaching roughly 92% ASR at a 0.1% poison rate and staying positive even when the poisoning rate drops to 0.0005%. Our code is available at https://github.com/safety-research/agent
Pingbang Hu ๐น๐ผ retweeted
Agentic Backdoors via Pretraining Poisoning arxiv.org/abs/2610.37741
I have been using Arc since beta. This is a generational fumble.
I get it, it's the era of AI. I really get it.
But abandoning a product that has (maybe not big) a group of die-hard users that will literally do anything (yes, take my money)? Like More than a year ago they stopped developing new features. I have never thought about leaving. It's this good.
This is just unheardof.
We've decided to sunset Arc on December 22, 2026.
That's not an easy sentence to write. We spent years building Arc and celebrating each and every release along the way: we have so much pride and love for what we made alongside the Arc community.
Our priority is to deliver best-in-class security for our members. Keeping a browser secure takes constant, dedicated work that is only increasing in the age of AI. Dia was built from the ground up to meet todayโs security standards, so that's where our engineering and security teams will focus moving forward.
We've worked to make switching to @diabrowser as easy as possible for Arc members. Here's how:
ยท Your passwords, bookmarks, extensions and history come over to Dia seamlessly.
ยท Spaces, Folders, and Tab Groups are in Dia and import over with your data intact.
ยท Dia now surpasses Arc benchmarks in stability, speed, and security.
ยท The browser features you use in Arc are free in Dia, and the AI is optional.
Thank you for using Arc, for pushing us to make it better, and for building alongside us. The team that made Arc is the same team behind Dia, and some of our most ambitious work is just ahead. We'd love for you to be there with us when it lands.
Pingbang Hu ๐น๐ผ retweeted
Change your GPU count mid-run and a typical data loader changes your data order, and possibly your results. Zephon keeps every global batch identical, and now works with TorchTitan.
Thanks @PyTorch for having us on the blog.
Zephonโs TorchTitan integration preserves identical global batches when training resumes on a different GPU count.
In our latest blogโs 1B-parameter model test, training resumed from a 16-GPU checkpoint on 8, 32, and 64 GPUs, and the SHA-256 hash of every global batch matched the original run.
Zephon tokenizes, packs, shuffles, and mixes data inside the training job, avoiding a preprocessed copy of the corpus.
The data, tokenizer, seed, mixture, context length, and global batch size must stay unchanged when resuming. The lane count must divide evenly across data-parallel replicas, and each optimizer stepโs batch must split evenly across lanes. Training losses can still differ because floating-point reduction order depends on GPU count.
โ๏ธ @datologyai @josh_wills @MaxiBoether, Ties Robroek (@datologyai)
Read more: pytorch.org/blog/bringing-elโฆ
Whole anti anti-distillation thing is wild. Usual argument is that Ant/OAI themselves โdistillโ knowledge from the open web and copyrighted data.
By that logic, your own work shouldnโt get copyright protection either, as you learned from othersโ work to create it.
Stop coping.
one side note: distillation does contribute to A\'s arr non-trivially.
The scale of distillation is quite insane...
to optimize this, what they really want is probably to make distillation difficult but not impossible ๐
Over the last few months, weโve made significant progress in reducing the effectiveness of distillation attacks against Claude.
In this post, we explain how: anthropic.com/research/distiโฆ
Congrats for the launch๐ฅ๐ฅ
Most valuable thing I learn this week in SF is not from COLM, but from meeting/talking to friends in labs and knowing that ppl start thinking seriously about #data valuation (for curation) on pretraining scale.
maybe should start talking to pretraining data team...?
Prediction market for ICLR: acceptodds.com/?kind=ICLR+20โฆ
yes, this is what we get in 2026 : )
Back in Berkeley 6 months after the fellowship
Pick up my hoodie, visit the new building
itโs all coming back
Pingbang Hu ๐น๐ผ retweeted
Trajectory-based data attribution leads recent SOTA methods, but its estimates still contain errors. Where do these errors come from?
Our work investigates this question!
๐ค Oral presentation at Actionable Interpretability @ COLM 2026 on Oct 9!
๐ arxiv.org/abs/2605.18814
1/
Pingbang Hu ๐น๐ผ retweeted
1/4 Which training examples shape an LLMโs outputs?
Meet dattri-LLM: open-source data attribution at LLM scale.
โก 3.2x avg. throughput vs. fastest competing library in our benchmarks
๐ 110B params on 4xH200 GPUs
Paper: ๐ arxiv.org/abs/2609.38767
Code: ๐ github.com/TRAIS-Lab/dattri-โฆ
