Surajit Dasgupta (@zephyrzilla) header
Surajit Dasgupta (@zephyrzilla) profile picture

Staff Research Scientist @servicenow Previously ML, NLP @FactSet | Opinions are my own.

Hyderabad, India
Joined January 2012
Photo by Surajit Dasgupta (@zephyrzilla)Photo by Surajit Dasgupta (@zephyrzilla)Photo by Surajit Dasgupta (@zephyrzilla)Photo by Surajit Dasgupta (@zephyrzilla)Photo by Surajit Dasgupta (@zephyrzilla)Photo by Surajit Dasgupta (@zephyrzilla)Photo by Surajit Dasgupta (@zephyrzilla)
Surajit Dasgupta retweeted
We asked GPT-6 Astra to edit Qwen3.5-0.8B weights to improve math. It directly changed only 0.016% of weights, fixing 0.8B's reasoning loops & improving math up to 6x and even instruction following 3x, without gradients or training. We call this Weight-Space Programming: frontier LLMs can now natively read/write neural network weights! 🧵 Blog: rikiyat.github.io/blog/weigh…
Readers added context they thought people might want to know
LLM was given weights explicitly as text in prompt and used code/forward passes (per authors' logs) to ablate Qwen looping direction, a known thinking-mode issue per model card. Primarily fixed repetition in that mode, not general 'natively' improving weights/math. rikiyat.github.io/blog/weight-sp… github.com/RikiyaT/lm_und… huggingface.co/Qwen/Qwen3.5-0… huggingface.co/RikiyaT/Qwen3.…
2,366
7,257
5,753
55,946
10,069,232
Surajit Dasgupta retweeted
🥳 Excited to share that this work is now accepted at NeurIPS 2026 main conference and CascadeBench is now open source! We gave a frontier model every business rule in an enterprise system, perfectly retrieved. It still couldn't predict what a single action would do. Fetching the rules isn't the bottleneck; composing them is. 📦 𝗖𝗮𝘀𝗰𝗮𝗱𝗲𝗕𝗲𝗻𝗰𝗵 is out, with every field change attributed to the rule that caused it: 🤗 huggingface.co/datasets/Serv… 📄 arxiv.org/abs/2605.12178 Congrats to @TheJishnuNair @patricebechard @rmahesh__ @zephyrzilla @shiva_malay, Sravan, Aakash, @johanobandoc @sagardavasam , @chrisjpal @siris_krishna @ServiceNowRSRCH @Mila_Quebec See you at NeurIPS!
1
8
3
24
2,464
Surajit Dasgupta retweeted
We are excited to release 📊 SynthDocBench (🎉 Accepted at COLM 2026)!   Frontier VLMs score 95%+ on DocVQA. So why do they fall apart on long documents?   SynthDocBench isolates the reason—the bottleneck isn't reasoning, it's perception 🧵.   #COLM2026
9
6
2
16
2,246
Surajit Dasgupta retweeted
Announcing EnterpriseOps-Gym-AA, our independent leaderboard for ServiceNow's EnterpriseOps-Gym, testing whether AI agents can run real enterprise operations while following business rules and policy We've partnered with @ServiceNow to launch EnterpriseOps-Gym-AA on Artificial Analysis, adapting the original EnterpriseOps-Gym benchmark (github.com/ServiceNow/Enterp…) to run on our Stirrup agent harness. The benchmark drops agents into live enterprise systems and asks them to complete multi-step, stateful work across 8 business domains, with models graded on the final state of the underlying databases rather than the steps taken. As we launch the eval, @AnthropicAI's Claude Fable 5 (max) leads at 51%, ahead of @GoogleDeepMind's Gemini 3.5 Flash (high, 50%), @OpenAI's GPT-5.5 (xhigh, 47%), with @Zai_org's GLM-5.2 (max, 43%) as the highest-scoring open weights model. Key elements of EnterpriseOps-Gym-AA: ➤ Stateful, multi-step enterprise work: Agents operate live enterprise systems where actions are often irreversible. A wrong write can corrupt the environment the way it would in production, so agents cannot brute-force their way to a solution. ➤ Live tool use across 8 domains: Models act through live Model Context Protocol (MCP) servers over 164 database tables and 512 tools, spanning core business systems (Customer Service, HR, IT Service Management), collaboration tools (Email, Calendar, Teams, Drive), and cross-domain Hybrid tasks. ➤ Outcome-based SQL verification: Each task is graded by executable SQL checks on the final environment state, testing goal completion, state integrity, policy compliance, and unintended side effects. This rewards outcomes over rigid action sequences and allows alternative valid solution paths. Key results: ➤ The frontier just clears 50%: Claude Fable 5 (max) leads at 51%, the only model clearly above half, with Gemini 3.5 Flash (high) close behind at 50% - a fast, low-cost Flash-tier model edging out larger frontier systems. Even the strongest models complete only about half of the 1,117 oracle-mode tasks. ➤ Models must pass all verification checks to succeed at a task: Models must satisfy all verification checks to succeed at a task, because a single missed constraint often means failure in the real world. We see that models satisfy most individual verifier checks (79% for the leader, Claude Fable 5) but complete far fewer whole tasks (51%). ➤ Open weights are within ~8 points: GLM-5.2 (max, 43%) is the strongest open weights model, about eight points behind the overall leader, with DeepSeek V4 Pro (max, 40%) and Kimi K2.7 (40%) close behind. We would like to thank @ServiceNowRSRCH , @Mila_Quebec, @UMontreal and the benchmark authors for their excellent work building EnterpriseOps-Gym, and appreciate their collaboration in launching it on Artificial Analysis!
18
16
9
238
27,096
🚀 Grok Voice Think Fast 1.0 (@xAI) lands on the Pareto frontier on EVA-Bench — no system in the eval beats it on accuracy without sacrificing experience, or vice versa. 📊 Leaderboard: servicenow.github.io/eva/#re… @elonmusk #VoiceAgents #ServiceNowResearch #EVABench #GrokVoice #xAI
4
28
8
87
129,062
Surajit Dasgupta retweeted
🚨The World Model Myth in Enterprise Systems🚨 Everyone’s obsessed with world models for agents. But in enterprise systems? The entire "world" (rules, workflows, transitions) is readable at runtime. No hidden dynamics. We asked the dangerous question: Do enterprise systems need learned world models?
1
6
4
14
2,282
Surajit Dasgupta retweeted
Do Enterprise Systems Need Learned World Models? The Importance of Context to Infer Dynamics
9
11
2
52
9,737
Surajit Dasgupta retweeted
📢 EnterpriseOps-Gym is now accepted to ICML 2026 🇰🇷✨ website: enterpriseops-gym.github.io/ 🧩 1,150 expert-curated tasks 🏢 8 enterprise domains 🧰 512 tools ✅ Deterministic verifiers (Outcome + Integrity + Compliance) 📦 Fully containerized, no enterprise instance required 📊 𝗙𝗿𝗲𝘀𝗵 𝘂𝗽𝗱𝗮𝘁𝗲: GPT-5.5 numbers are out (alongside the strongest open and closed baselines). We will keep updating as new models drop because long-horizon reliability is moving fast, and we want to stay current. One exciting (and humbling) signal: we’ve already seen frontier lab teams experimenting with EnterpriseOps-Gym to stress-test and improve their agents: including folks at OpenAI, Mistral AI, and NVIDIA AI. 🙏 📈 𝗘𝗮𝗿𝗹𝘆 𝗿𝗲𝘀𝘂𝗹𝘁𝘀 𝗮𝗿𝗲 𝗽𝗿𝗼𝗺𝗶𝘀𝗶𝗻𝗴: Top open models  including NVIDIA Nemotron Super  are showing strong performance, in some cases competing with frontier models. @shiva_malay @sagardavasam @PShravannayak @turingcom @jonsid @ServiceNowRSRCH @Mila_Quebec
2
15
1
45
3,552
Surajit Dasgupta retweeted
Terminal Agents Suffice for Enterprise Automation ServiceNow research shows terminal-based coding agents with direct API access match or outperform complex MCP and GUI agents, proving strong foundation models need only simple programmatic interfaces for enterprise automation.
6
14
1
54
10,504
Surajit Dasgupta retweeted
What if we didn't need MCP servers after all? What if we didn't need browser-use agents either? What if... Claude Code was enough? In our latest paper, we test exactly this: Can simple terminal agents outperform web agents and tool-based agents on real enterprise tasks?
2
17
3
37
7,614
Surajit Dasgupta retweeted
🧵 Introducing 𝐄𝐧𝐭𝐞𝐫𝐩𝐫𝐢𝐬𝐞𝐎𝐩𝐬-𝐆𝐲𝐦🚀 : a rigorous new benchmark for stateful agentic planning and tool use in real enterprise environments. 1,150 expert-curated tasks · 512 tools · 164 DB tables · 8 domains. Every task verified by hand-written SQL, checking goal completion, state integrity and policy compliance🔥 𝐓𝐡𝐞 𝐡𝐞𝐚𝐝𝐥𝐢𝐧𝐞: Claude Opus 4.5 — our best-performing model succeeds on just 37.4% of tasks. With oracle tool access. No tool discovery required. 📄 arxiv.org/abs/2603.13594 (trending #4 on daily-papers) 🌐 enterpriseops-gym.github.io 🤗 huggingface.co/datasets/Serv… 💻 github.com/ServiceNow/Enterp…
3
28
6
60
9,474
Surajit Dasgupta retweeted
Mercury 2 is live 🚀🚀 The world’s first reasoning diffusion LLM, delivering 5x faster performance than leading speed-optimized LLMs. Watching the team turn years of research into a real product never gets old, and I’m incredibly proud of what we’ve built. We’re just getting started on what diffusion can do for language.
314
562
267
4,188
1,052,907
Surajit Dasgupta retweeted
合成データのためのワークフローをDifyみたいなUIで作れるツール 適当なFAQ作って、クエリの生成を試してみたのだが、サッと立ち上げられてLocalデータともスムーズに接続できて体験よかった 進行状況のモニタリングとか、Model / Promptの比較とかもやりやすそう
1
1
5
473
🚀 Introducing AprielGuard — an 8B parameter state-of-the-art guardian model. Built to catch policy violations and jailbreaks alike, fully post-trained on synthetic data using SyGra. 🧠 Model Weights: huggingface.co/ServiceNow-AI… 📄 Blog: huggingface.co/blog/ServiceN… @JayKasundra09 @SeganBoss @SathwikTejaswi @ServiceNow @ServiceNowNews @NVIDIAAI @nvidianewsroom Why it matters 👇
1
18
1
34
3,812
Apriel 1.6 15B Thinker 🚀 is trending top-5 image-text-to-text model on @huggingface ! Try it out and let us know what you think. Chat with it on @togethercompute We've also hosted on @huggingface in a HF space called Apriel chat. Thanks to all of our collaborators and partners. @ServiceNow @ServiceNowRSRCH @NVIDIAAI @turingcom @tscholak
12
30
1,332
Surajit Dasgupta retweeted
Okay okay, due to reasonable feedback we added: @cohere for their non commercial models @ServiceNow with Apriel, I like folks there (and pipeline rl) Motif @tngtech as a shout out for awesome hacks and merges of big MoEs This is DEFINITELY right, no take backs
5
16
5
91
26,778
1/5 🚀Apriel-1.6-15B-Thinker: a 15B multimodal reasoner scoring 57 on the Artificial Analysis Intelligence Index - approaching the performance of ~200B-scale frontier models while remaining an order of magnitude smaller. 🧠Model weights: huggingface.co/ServiceNow-AI… 📄Blog: huggingface.co/blog/ServiceN… 💬Chat demo: huggingface.co/spaces/Servic… @SathwikTejaswi @sagardavasam @tscholak @NVIDIAAI @nvidianewsroom @togethercompute @turingcom @ArtificialAnlys
9
54
9
219
24,611
Surajit Dasgupta retweeted
Don't sleep on PipelineRL -- this is one of the biggest jumps in compute efficiency of RL setups that we found in the ScaleRL paper (also validated by Magistral & others before)! What's the problem PipelineRL solves? In RL for LLMs, we need to send weight updates from trainer to generator (to generate data from our latest policy being trained). (Conventional PPO-off-policy) A naive approach would be to "start generators on a batch, wait for all sequences to complete, update the model weights for both trainers and generators, and repeat. Unfortunately, this approach leads to idle generators and low pipeline efficiency due to heterogeneous completion times. (Pipeline-RL) Instead, we simply let the generators continue generating tokens without discarding or finishing ongoing generations in-flight whenever we need to do a weight update -- doing an "in-flight" weight update. As such our KV caches for these generations would be stale, as they would come from LLM with earlier copy(ies) of the weights) but this is ok (see below).
In-flight weight updates have gone from a “weird trick” to a must to train LLMs with RL in the last few weeks. If you want to understand the on-policy and throughput benefits here’s the CoLM talk @DBahdanau and I gave: youtu.be/Z1uEuRKACRs
12
56
8
471
131,915
Surajit Dasgupta retweeted
to continue the PipelineRL glazing, @finbarrtimbers implemented PipelineRL for open-instruct a little bit ago and it ended up being probably the single biggest speedup to our overall pipeline. We went from 2-week long RL runs to 5-day runs, without sacrificing performance (combined with some other threading etc. updates). Here's IFEval perf for an internal model (same data, same starting model, same bsz). Same number of training steps, same end perf, but PipelineRL is much faster.
Don't sleep on PipelineRL -- this is one of the biggest jumps in compute efficiency of RL setups that we found in the ScaleRL paper (also validated by Magistral & others before)! What's the problem PipelineRL solves? In RL for LLMs, we need to send weight updates from trainer to generator (to generate data from our latest policy being trained). (Conventional PPO-off-policy) A naive approach would be to "start generators on a batch, wait for all sequences to complete, update the model weights for both trainers and generators, and repeat. Unfortunately, this approach leads to idle generators and low pipeline efficiency due to heterogeneous completion times. (Pipeline-RL) Instead, we simply let the generators continue generating tokens without discarding or finishing ongoing generations in-flight whenever we need to do a weight update -- doing an "in-flight" weight update. As such our KV caches for these generations would be stale, as they would come from LLM with earlier copy(ies) of the weights) but this is ok (see below).
5
29
5
217
54,749
Not All Bits Are Equal: What We Learned From 1700 Experiments on Memory-Optimal Reasoning Given a fixed memory budget, how should you allocate across model weights, KV cache, and test-time compute to maximize accuracy in reasoning models? For example: would you choose a 32B, 4-bit model with a 14k token budget, or an 8B, 16-bit model with 30k tokens? We ran 1,700 experiments on the Qwen3 family to find out. We varied: - Model size (0.6B-32B), - Weight precision (4/8/16-bit via GPTQ), - Serial test-time compute (token budgets 2k→30k via budget forcing), - Parallel test-time compute (Maj@K, up to K=16), - KV cache compression (eviction: R-KV, StreamingLLM; quantization: HQQ at 2/4/8-bit). This is great work led by my Krafton/UW collaborators @jhyuckkim (Krafton), @ethan_ewer (undergrad!! at UW-Madison), @taehong_moon (Krafton), @jon_ghoh (UC Berkeley) Here is a summary of the main findings: At the 4B+8-bit threshold optimal mem strategy flips For models effectively smaller than 8-bit 4B, spend memory on more (or higher-precision) weights, not on longer generations. For larger models, do the opposite: allocate memory to longer generations until performance saturates. This threshold isn't arbitrary, i.e., it is right at the point where weights dominate KV cache/token count. The reasoning task matters (eg your mileage may vary, no universal recipe!) Math reasoning (eg AIME25): 4-bit quantization is almost always a bad idea. An 8B model at 16-bit outperforms a 14B model at 4-bit with similar memory spent. The numerical precision in weights seems to matter for "reasoning heavy" tasks. Almost as if the model’s capacity to utilize test time compute is decimated by quantization... Knowledge heavier tasks (GPQA-D): 4-bit is broadly memory-optimal. Here, parameter count matters more than precision. The interpretation is that here you want more effective weights to store things, and raw parameter count dominates test time compute. How does parallel test time compute (fancy for majority voting) factor in? Majority voting (Maj@K) increases KV cache linearly with K. It improves the mem–acc trade-off only when the model is >= 8-bit 4B effective size; the optimal K grows with the memory budget. Below that scale, serial test time compute should be preferred. Weight quantization alone isn't enough! Both KV cache eviction and KV quantization push the mem optimal Pareto frontier higher across all model sizes we tested. Should you prefer KV evict or quant? - Small models (<8-bit 4B): KV cache eviction wins - Large models (≥8-bit 4B): Both strategies competitive Latency/throughput note: End-to-end latency is dominated by generation length. When you care a lot about latency, 8-bit often sits at a better speed–accuracy point than 4-bit. Note on batching: When model weights, i.e., params, are amortized across concurrent generations (that is, batched inference), the tradeoff shifts, as you’d expected. At a batch size of 16 the 0.6B model never appears on the Pareto. The 4B-8bit model always appears no matter the batchsize in the ~1-2GB memory region (good model setting for mobile devices!) What This Means ***There is no universal memory-optimal strategy for reasoning models!*** The right choice depends on almost every parameter involved, but here is one way to choose: If effective size < 8-bit 4B --- Spend your bits on model capacity/precision over longer token budgets. --- Prefer 8-bit for math-heavy tasks. --- Use KV eviction over KV quantization. --- Stick to serial scaling; Maj@K is memory-inefficient here. If effective size ≥ 8-bit 4B ---Increase token budget till gains saturate. ---Maj@K always helps, so grow K with available memory. ---KV quantization is competitive with eviction; choose based on implementation and maybe taste 😊 Important Caveat: These findings are specific to the Qwen3 family on AIME25 and GPQA-D. The thresholds and strategies will vary with different architectures, training methods, and task distributions.
12
71
11
457
101,335