Utkarsh Tyagi (@utkarsh4430) profile picture

Audio and Multimodal Research @ScaleAILabs | CS @umdcs

San Francisco
Joined July 2018
Photo by Utkarsh Tyagi (@utkarsh4430)Photo by Utkarsh Tyagi (@utkarsh4430)Photo by Utkarsh Tyagi (@utkarsh4430)Photo by Utkarsh Tyagi (@utkarsh4430)Photo by Utkarsh Tyagi (@utkarsh4430)Photo by Utkarsh Tyagi (@utkarsh4430)Photo by Utkarsh Tyagi (@utkarsh4430)Photo by Utkarsh Tyagi (@utkarsh4430)Photo by Utkarsh Tyagi (@utkarsh4430)
HUGE! Curious to try this for audio-native rubric rewards in speech-to-speech RL, especially for tone, delivery, and turn-taking.
Introducing the Hume Audio Decision API. Ask anything: How many speakers? Is anyone laughing? Is the caller frustrated? What used to take 10+ models is now one call. Trained on 10M+ human judgments, so it answers the way human listeners would. Realtime, or across hours of recordings. Request access below.
2
10
944
Utkarsh Tyagi retweeted
A few months ago I left @Apple Siri AI to build with the most talented systems for AI team I ever worked with. Happy to bring this to fruition this soon. We are just getting started 🚀
Introducing Avartha. We're building the infrastructure to bring frontier-grade intelligence to real-time applications. Our first release, Ava Zero, is a voice agent entirely running open models using our inference infrastructure and ranks #1 on Sierra's tau-voice benchmark with highest task completion rate and generation quality and just costs $2.5 per hour. Open models are now at the frontier of voice AI.
1
4
4
81
19,381
sooo many voice AI startups coming up I didn’t know the full cascaded pipeline was ASR → LLM → TTS → seed round
1
7
228
Asking to be on a paper, doing zero work, and then missing the author checklist is impressive. Congrats, you found a way to contribute negatively!
2
1
113
15,932
Been working on this benchmark for the past few months, and the gap is striking! 🚀 AI can solve incredibly complex problems, yet still lacks the everyday intuition of how the world works. With Humanity’s Sixth Sense, we’re putting that missing intuition to the test. 🧠👀
We’re introducing Humanity’s Sixth Sense, a new benchmark testing the intuitive visual reasoning people rely on every day. Across 522 open-ended tasks spanning images and video, HSS tests everything from spatial and causal reasoning to social understanding. The gap is significant. Humans score 93.1%, while the strongest model, GPT-6-astra, reaches 53.6%. The median model scores just 30.9%.
3
4
38
1,608
Utkarsh Tyagi retweeted
Audio LLMs are good at what they hear, but mostly not where it's coming from. From my @Google internship: SEA-LM, egocentric spatial audio understanding for smart glasses. One model → what, where, when, who (you vs. bystanders), and what was said. arxiv.org/abs/2610.05610
1
2
1
61
3,193
Utkarsh Tyagi retweeted
Glad to share that 3 papers I contributed to at @ScaleAILabs have been accepted to NeurIPS 2026, spanning post-training and agent evaluation 🎉 🔍 Reward Hacking in Rubric-Based Reinforcement Learning We conducted a systematic study of reward hacking in rubric-based RL, distinguishing failures caused by weak verifiers from failures in rubric design, and identified an interesting stopping criterion. arxiv.org/abs/2605.12474 🧠 Not Every Rubric Teaches Equally We introduce a policy-aware curriculum for rubric-based RL that emphasizes the criteria most useful for learning at each stage of training. arxiv.org/abs/2605.20164 🛠️ SWE Atlas We introduce a benchmark for evaluating coding agents beyond issue resolution, covering codebase Q&A, test writing, and refactoring. arxiv.org/abs/2605.08366 Really proud of the team and the work behind these. Looking forward to sharing more and catching up with everyone at NeurIPS in Sydney!
10
8
54
2,064
Utkarsh Tyagi retweeted
"MCP-Atlas: A Large-Scale Benchmark for Tool-Use Competency with Real MCP Servers" has been accepted to NeurIPS 2026!! I had the privilege to be co-first author on the paper and to work with virtually all of the frontier labs to enable them to report numbers on our benchmark!
6
7
3
32
3,230
Utkarsh Tyagi retweeted
Excited to share that TAC has been accepted at NeurIPS 2026. TAC enabled timestamped audio captioning and speaker diarization in overlapping settings. TAC captions when combined with text-only reasoner achieves SOTA scores on most of the audio(-visual) benchmarks. Work done at @Adobe with @pseetharaman, @urinieto, @justin_salamon, @ZeyuJin and others. Super exciting work. More here: sonalkum.github.io/tacmodel/
2
5
31
1,411
Utkarsh Tyagi retweeted
🎉 Excited to share that “Reward Hacking in Rubric-Based Reinforcement Learning” has been accepted to #NeurIPS2026! Reward hacking in post-training is a particularly interesting and important problem to work on. We study how models can learn to game rubric-based rewards during post-training, satisfying the rubric without necessarily improving the broader response quality. We characterize these failures at the criterion level and show that stronger verifiers help, but don’t fully eliminate reward hacking. Link: arxiv.org/abs/2605.12474 Check out some of our recent post-training research projects at @ScaleAILabs: RGSD: arxiv.org/html/2606.12507 POW3R: arxiv.org/abs/2605.20164
4
8
29
1,069
🎉 Excited that “Not Every Rubric Teaches Equally” is accepted to NeurIPS 2026! 🎉 POW3R turns static rubric rewards into policy-aware training signals, emphasizing the criteria that actually separate good and bad rollouts for the current state of the model. Link - arxiv.org/pdf/2605.20164 Our team at @ScaleAILabs has been cooking on some really exciting RL/post-training research lately. Check it out 👀 RGSD - arxiv.org/abs/2606.12507 Reward Hacking with Rubrics - arxiv.org/abs/2605.12474
4
7
1
52
2,552
Career advice I did not expect to give: go work for the “data vendor” 😭 You may not be building the frontier model, but you get exposed to problems across a bunch of frontier labs: audio, multimodal, coding, multilingual, safety, physical AI, RL environments, and whatever new thing appeared this week. As data becomes half the game and RL-env startups casually raise millions, this has been an absurdly good place to learn. Getting to publish open research at the same time = 🫶
8
5
4
156
11,437
Really interesting direction by @kyutai_labs. We’ve been actively experimenting with similar ideas around SteerDuplex (arxiv.org/pdf/2609.12623v1) Another axis I find especially exciting is *thinking while listening* - not just while speaking. Humans don’t stop reasoning when the other person has the floor; we continuously form and revise hypotheses as information arrives. The hard part is reasoning under incomplete evidence: intermediate thoughts need to be speculative and revisable, rather than committing too early to a single CoT. A full-duplex model should ideally be able to update, branch, or discard its internal reasoning as the utterance unfolds.
We're releasing Voice of Reason, a speech-native model that does math out loud. Give it a spoken problem without transcription nor text LLM in the loop, and it reasons and answers in speech. GSM8K goes from 27.3% for GLM-4-Voice to 77.1%. Link in 🧵
1
36
4,251
Utkarsh Tyagi retweeted
excited to release our latest @ScaleAILabs on post-training full-duplex speech models! 1/ we fine-tune Moshi (by @kyutai_labs) on in-house conversation data to enable spoken steerability, i.e. the model’s ability to respond in a prompted tone, pace and style 2/ next, we use RLVR to improve Moshi’s turn-taking, backchannels and interruption recovery while simultaneously enhancing its general multi-turn intelligence 3/ during RL, we identify cases of reward hacking (such as empty responses gaming individual rewards) and mitigate them with robust, composite objectives SteerDuplex, our resulting model, demonstrates a post-training recipe that simultaneously improves steerability, interactivity and intelligence for full-duplex speech architectures
Full-duplex voice models can listen and speak at the same time, but can they change how they speak when you ask them to? Introducing SteerDuplex + SteerBench: the first benchmark that measures steerable full-duplex dialogue. 65.1% audio-steering APR vs 20.6% for the best open baseline. 🧵
4
3
37
3,198
Full-duplex voice models can listen and speak at the same time, but can they change how they speak when you ask them to? Introducing SteerDuplex + SteerBench: the first benchmark that measures steerable full-duplex dialogue. 65.1% audio-steering APR vs 20.6% for the best open baseline. 🧵
3
15
3
118
15,496
Steering is only half the problem. The other half is timing: when to speak, when to wait, when to keep going. RL improved it. Responding correctly after you interrupt went from 72.5% to 82.5%. Speaking over your pauses dropped from 26.5% to 9%. Then we found the trap: reward a model for going quiet when interrupted, and it learns to never speak at all. Timing only counts next to a judge on what was actually said, a continuity term that wants about 4 seconds of real speech, and a waveform check that throws out empty runs. Silent runs dropped from 25 of 96 to 4.
1
1
384
The full recipe: SFT teaches what to say and how to sound. Two-stage RL teaches when to speak, when to wait, and when to keep going. Content, delivery and timing have to be trained together. Optimize one alone and you quietly damage the other two. SteerDuplex is @ScaleAILabs and @gammaumd Paper: arxiv.org/abs/2609.12623v1 Benchmark, code and checkpoints: coming soon 🔊
6
357