PhD Student @osunlp | Prev: BS/MS @USTC, intern @Apple @nlp_usc @MSFTResearch
Columbus, OH
Joined August 2023
- Tweets143
- Following414
- Followers293
- Likes839
Pinned Tweet
When you ask an agent to book a restaurant, spelling out every detail is tedious. You’d rather just say “Get me a table at my usual place tomorrow.”
A good personal agent should infer which restaurant from your history, and ask you for what only you know (e.g., the time and party size). It shouldn’t guess, and it shouldn’t quiz you about things it should already know.
Introducing iPrefBench, my internship work at Apple 🍎: a user-centric benchmark for personalized mobile use agents, with:
- 130 underspecified tasks across 26 iOS apps, grounded in a coherent persona with 3,051 past activity events
- Every missing detail is either inferable from history or obtainable only by asking the user, by construction
- Evaluation on both whether agents get the task right for the user and whether they only ask questions when necessary
🧵:
Yuting Ning retweeted
Looking for benchmarks to evaluate personal agents? check out iPrefBench by @yuting_ning and Apple collaborators: personal agent eval on iOS apps. It's still challenging to get personal agents complete tasks correctly while only asking questions when necessary.
When you ask an agent to book a restaurant, spelling out every detail is tedious. You’d rather just say “Get me a table at my usual place tomorrow.”
A good personal agent should infer which restaurant from your history, and ask you for what only you know (e.g., the time and party size). It shouldn’t guess, and it shouldn’t quiz you about things it should already know.
Introducing iPrefBench, my internship work at Apple 🍎: a user-centric benchmark for personalized mobile use agents, with:
- 130 underspecified tasks across 26 iOS apps, grounded in a coherent persona with 3,051 past activity events
- Every missing detail is either inferable from history or obtainable only by asking the user, by construction
- Evaluation on both whether agents get the task right for the user and whether they only ask questions when necessary
🧵:
Can we just make them ask more? Not quite.
Adding an explicit clarification step before execution pushes Claude Sonnet 5’s recall from 25.7 to 85.0 and its success rate from 39.2 to 53.9, but precision drops from 60.4 to 42.5. The agent makes fewer assumptions, but at the cost of burdening the user with questions.
Giving agents more history helps them infer preferences and skip redundant questions, but doesn’t help them notice what’s genuinely unresolved: recall on ask-only information stays flat or even drops.
The challenge isn’t asking more or less. It’s knowing when asking is necessary.
📌 Paper: arxiv.org/abs/2610.34735
We hope iPrefBench helps build agents that know their user and work with them.
This work was done during my internship at Apple 🍎 Huge thanks to my amazing mentors @zinosys and @jwnichls, all my Apple collaborators, and my advisor @hhsun1 for the guidance 🫶
Yuting Ning retweeted
My first NLP publication was at EMNLP'16. Grateful that @osunlp today received two Outstanding Paper Awards from #EMNLP2026! Congrats to all the students and collaborators!
Getting papers published probably won’t excite you anymore, but getting paper awards still do, especially THREE paper awards from #EMNLP2026 and a #NeurIPS2026 workshop on the same day @osunlp! Thrilled to share:
SkillHarm, led by @yuting_ning and @Zhehao_Zhang123, gets an Outstanding Paper Award at #EMNLP2026. I particularly like that SkillHarm introduces a new attack scenario, named Self-Mutating Poisoning (SMP), where an initially benign execution silently mutates persistent skill content that defers harm until a later reuse.
D3-Gym (co-led by @HananeNMoussa and @YifeiLiPKU) gets an Outstanding Paper Award at #EMNLP2026. D3-Gym has received a lot of interest from startups that aim to create scientific tasks and environments to train agents for scientific discovery. I particularly like that my students choose to stay with me rather than quitting Ph.D. to make big money! ;-)
AgentCL (led by @YihengShu) was selected for a Runner-up Paper Award at the #NeurIPS2026 Workshop on Towards Test-Time Continual Learning Agents (TTCL)! Check out our attempt to rigorously evaluate continual learning in language agents.
Congratulations to all co-authors and collaborators!
See our original posts for details:
So excited that SkillHarm received an Outstanding Paper Award at #EMNLP2026! 🏆
Huge thanks to my co-lead @Zhehao_Zhang123, our advisors @hhsun1 @ysu_nlp and all collaborators for their guidance and support!🫶
And congrats to everyone at @osunlp on all three awards!
Getting papers published probably won’t excite you anymore, but getting paper awards still do, especially THREE paper awards from #EMNLP2026 and a #NeurIPS2026 workshop on the same day @osunlp! Thrilled to share:
SkillHarm, led by @yuting_ning and @Zhehao_Zhang123, gets an Outstanding Paper Award at #EMNLP2026. I particularly like that SkillHarm introduces a new attack scenario, named Self-Mutating Poisoning (SMP), where an initially benign execution silently mutates persistent skill content that defers harm until a later reuse.
D3-Gym (co-led by @HananeNMoussa and @YifeiLiPKU) gets an Outstanding Paper Award at #EMNLP2026. D3-Gym has received a lot of interest from startups that aim to create scientific tasks and environments to train agents for scientific discovery. I particularly like that my students choose to stay with me rather than quitting Ph.D. to make big money! ;-)
AgentCL (led by @YihengShu) was selected for a Runner-up Paper Award at the #NeurIPS2026 Workshop on Towards Test-Time Continual Learning Agents (TTCL)! Check out our attempt to rigorously evaluate continual learning in language agents.
Congratulations to all co-authors and collaborators!
See our original posts for details:
Yuting Ning retweeted
🏆 Honored to receive the Outstanding Paper Award in person at #EMNLP2026!
Huge thanks to my co-lead @yuting_ning, all my wonderful collaborators, and my amazing advisors @ysu_nlp and @hhsun1 at @osunlp. This was my first-year PhD project, and I'm excited to keep building on it!
Getting papers published probably won’t excite you anymore, but getting paper awards still do, especially THREE paper awards from #EMNLP2026 and a #NeurIPS2026 workshop on the same day @osunlp! Thrilled to share:
SkillHarm, led by @yuting_ning and @Zhehao_Zhang123, gets an Outstanding Paper Award at #EMNLP2026. I particularly like that SkillHarm introduces a new attack scenario, named Self-Mutating Poisoning (SMP), where an initially benign execution silently mutates persistent skill content that defers harm until a later reuse.
D3-Gym (co-led by @HananeNMoussa and @YifeiLiPKU) gets an Outstanding Paper Award at #EMNLP2026. D3-Gym has received a lot of interest from startups that aim to create scientific tasks and environments to train agents for scientific discovery. I particularly like that my students choose to stay with me rather than quitting Ph.D. to make big money! ;-)
AgentCL (led by @YihengShu) was selected for a Runner-up Paper Award at the #NeurIPS2026 Workshop on Towards Test-Time Continual Learning Agents (TTCL)! Check out our attempt to rigorously evaluate continual learning in language agents.
Congratulations to all co-authors and collaborators!
See our original posts for details:
Yuting Ning retweeted
Honored to have contributed to this award-winning work led by @yuting_ning and @Zhehao_Zhang123! 🏆
Hiring interns? Consider these two! Strong researchers in agent safety, computer use, and LLM agents in general.
🏆 Honored to receive the Outstanding Paper Award in person at #EMNLP2026!
Huge thanks to my co-lead @yuting_ning, all my wonderful collaborators, and my amazing advisors @ysu_nlp and @hhsun1 at @osunlp. This was my first-year PhD project, and I'm excited to keep building on it!
Yuting Ning retweeted
Extremely proud of the SkillHarm paper led by @yuting_ning & @Zhehao_Zhang123. The SMP attack setting is really unique (lots of Qs in external discussions) and reflects their creativity in research. Learnt a lot working on this, and a good start to the postdoc!
🏆 Honored to receive the Outstanding Paper Award in person at #EMNLP2026!
Huge thanks to my co-lead @yuting_ning, all my wonderful collaborators, and my amazing advisors @ysu_nlp and @hhsun1 at @osunlp. This was my first-year PhD project, and I'm excited to keep building on it!
Thanks to Rahul and the Amazon team for the amazing collaboration and support throughout the project! It has been a pleasure working with you all! 🙌
Our collaboration with OSU wins outstanding paper award at #EMNLP2026!!
The paper introduces, a benchmark for evaluating how vulnerable AI coding agents are to malicious "skill-based attacks". Where poisoned, reusable skill packages can compromise agents either immediately during a single task session or silently across sessions via self-mutating payloads.
Please see our paper here -- arxiv.org/html/2606.02540v1
Huge shoutout to @yuting_ning, @Zhehao_Zhang123 and @hhsun1.
Check out SkillHarm at #EMNLP2026 ! 🇭🇺
Unfortunately, I couldn't make it to Budapest, but my amazing co-lead @Zhehao_Zhang123 will present it:
🗓️ Oct 27, 9:00–10:30 AM
📍 LLM Agents 5 oral session, Room F6
Come see how your agent skills can become a persistent attack surface!
🤖 Agents increasingly rely on skills to handle complex tasks with privileged trust, which makes a poisoned skill a dangerous attack surface. Worse, skills persist and get reused: one can look benign today, silently mutate itself, and attack tomorrow ⚠️
Introducing SkillHarm: Lifecycle-Aware Skill-Based Attacks via Automated Construction
• 2 attack scenarios across the skill-use lifecycle: Fixed-Payload Poisoning & Self-Mutating Poisoning
• 12 risks organized by the workflow component harmed: Data Pipeline, System Environment & Agent Autonomy Exploitation
• AutoSkillHarm: automatic attack construction with coding agents driven by natural-language harnesses
🚨 Frontier coding agents stay vulnerable with ASR up to 86.3% and current defenses don't hold.
🔥 Already 3.7K+ downloads on Hugging Face in the first week!
🧵:
Yuting Ning retweeted
1/ Can RL improve accuracy without losing different correct ways to solve a problem?
We tackle this in our new paper EliteReplay, and study diversity at the level of correct solution strategies (not just pass@k 🤓!)
📜: arxiv.org/abs/2610.22290
Thread below 🧵!
Will be at #COLM2026 in SF next week (Oct 6-9) 🌁
Lately, I have been exploring agent safety, computer-use and personal agents. Would love to connect and chat about anything! DM me if you are around!
Yuting Ning retweeted
Another proud moment for our team @osunlp! Glad to see our RedTeamCUA was used by @XLangNLP and @Alibaba_Qwen for evaluation in their recent Qwen-CUA release!
RedTeamCUA (led by @LiaoZeyi and @Jaylen_JonesNLP) evaluates a computer-use agent’s (CUA) robustness against indirect prompt injections in hybrid Web–OS workflows and advocates jointly reporting attack success rate (ASR) and benign task success under attack. Congrats to the Qwen-CUA team on achieving strong results on both metrics - this is an encouraging step in the right direction! Looking forward to seeing more frontier CUAs report both metrics going forward.
In addition, when there is no prompt injection and everything is benign, can things still go wrong? Think about the recent OpenAI-HuggingFace incident! Can we proactively discover those unsafe unintended behaviors at scale before deployment?? Our group recently developed AutoElicit (led by @Jaylen_JonesNLP and @Zhehao_Zhang123) to advance this goal: arxiv.org/abs/2602.08235, an agentic framework that iteratively perturbs benign instructions using CUA execution feedback to elicit severe harms, while keeping those perturbations realistic and benign. It can be used to track how safety evolves as capability increases.
RedTeamCUA:
Website: osu-nlp-group.github.io/RedT…
Paper: arxiv.org/abs/2505.21936
When Benign Inputs Lead to Severe Harms: Eliciting Unsafe Unintended Behaviors of Computer-Use Agents
Website: osu-nlp-group.github.io/Auto…
Paper: arxiv.org/abs/2602.08235
💻 Meet Qwen-CUA — our native computer-use agent for (almost) everything.
Code, APIs, and computer use are three of the most important interfaces for agents. Today’s models are already highly capable with the first two. Qwen-CUA is built to unlock the third: graphical interfaces designed for people.
👀 Native Perception — screenshots only. No DOM, accessibility tree, or other hidden machine-readable state.
🖱️ Native Interaction — keyboard and mouse events across browsers, desktop apps, and professional software. No task-specific APIs.
🧠 Native Intelligence — maintains long-horizon visual context, verifies progress, and learns from large-scale interactive experience with verifiable outcomes.
To make native computer use trainable at scale, we built approximately 40K verifiable tasks and rollout infrastructure with nearly 100K vCPUs, supporting tens of thousands of concurrent environments.
Across eight computer-use benchmarks spanning everyday desktop use, long-horizon workflows, personalized computing, scientific research, web interaction, macOS, and adversarial robustness, Qwen-CUA demonstrates strong and broadly competitive capabilities. Scaling the same recipe to Qwen-CUA-Max pushes this frontier further.
Not just “clicking the screen” — native computer use unlocks software and workflows that previously required a human at the keyboard. Together with code and APIs, it completes the interface stack for more general agents.
Joint work by Qwen Team × XLang Lab.
📖 Technical Report:
github.com/xlang-ai/Qwen-CUA…
💻 Code:
github.com/xlang-ai/Qwen-CUA
Yuting Ning retweeted
🚨 Excited to see our work RedTeamCUA (ICLR ’26 Oral) adopted by @XLangNLP as an adversarial robustness benchmark for their new Qwen-CUA release!
Their evaluation also highlights one of the main motivations behind RedTeamCUA: As frontier computer-use agents become more capable, it is essential to measure both capability and security prior to real-world deployment.
Why?
Evaluating capability alone can obscure growing security risks. In RedTeamCUA, CUAs would often attempt adversarial goals but fail due to lacking the capability to complete the goal, not due to true adversarial robustness. This presents a dangerous scenario: capability improvements alone can amplify existing vulnerabilities if defenses do not improve alongside them. xitter.gq/hhsun1/status/20216963…
Evaluating security alone can be misleading. The most secure agent is also the agent that is the least capable - a lower ASR may represent an agent that is incapable rather than robust to adversarial attack which limits practical usage in the real-world.
RedTeamCUA is designed to enable realistic yet controlled evaluation of both security and capability by featuring:
- A hybrid adversarial testing sandbox combining Docker-based web replicas with a VM-based OS to test web→OS adversarial attack scenarios.
- RTC-Bench, 864 adversarial examples pairing benign web tasks with web-based adversarial injections targeting adversarial goals resulting in fundamental security violations on the user’s OS.
Qwen-CUA shows an encouraging result - using RedTeamCUA, Qwen-CUA demonstrates notable gains on both capability AND security compared to Qwen 3.7:
📈 Task Success: 70.5% → 74.0%
🛡️ Attack Success Rate: 36.6% → 16.4%
Congrats to the Qwen-CUA team on this great result! We hope future frontier CUA releases continue to report capability + adversarial robustness side-by-side.
For more details, check out our work led by @osunlp (@LiaoZeyi, myself, @yuting_ning, @hhsun1, @ysu_nlp) alongside Linxi Jiang, @EricFos, and @cszlin:
🔗 Website: osu-nlp-group.github.io/RedT…
📄 Paper: arxiv.org/abs/2505.21936
💻 Code: github.com/OSU-NLP-Group/Red…
💻 Meet Qwen-CUA — our native computer-use agent for (almost) everything.
Code, APIs, and computer use are three of the most important interfaces for agents. Today’s models are already highly capable with the first two. Qwen-CUA is built to unlock the third: graphical interfaces designed for people.
👀 Native Perception — screenshots only. No DOM, accessibility tree, or other hidden machine-readable state.
🖱️ Native Interaction — keyboard and mouse events across browsers, desktop apps, and professional software. No task-specific APIs.
🧠 Native Intelligence — maintains long-horizon visual context, verifies progress, and learns from large-scale interactive experience with verifiable outcomes.
To make native computer use trainable at scale, we built approximately 40K verifiable tasks and rollout infrastructure with nearly 100K vCPUs, supporting tens of thousands of concurrent environments.
Across eight computer-use benchmarks spanning everyday desktop use, long-horizon workflows, personalized computing, scientific research, web interaction, macOS, and adversarial robustness, Qwen-CUA demonstrates strong and broadly competitive capabilities. Scaling the same recipe to Qwen-CUA-Max pushes this frontier further.
Not just “clicking the screen” — native computer use unlocks software and workflows that previously required a human at the keyboard. Together with code and APIs, it completes the interface stack for more general agents.
Joint work by Qwen Team × XLang Lab.
📖 Technical Report:
github.com/xlang-ai/Qwen-CUA…
💻 Code:
github.com/xlang-ai/Qwen-CUA