voice inference
San Francisco
Joined August 2026
- Tweets25
- Following16
- Followers119
- Likes19
640 tok/s on Gemma 4 with a megakernel
Gemma 4 26B-A4B runs at 173 tok/s using vLLM on H100 and a megakernel improved this 3.7x to 640 tok/s that too using a older TPU v5e-8 without any spec decoding.
We wrote Gemma-specific kernels for both the dense and MoE models and reworked expert sharding on top of Inferact's megakernel to reach these numbers. Full write-up linked in the comments.
here is the article : withhopper.com/blog/gemma-tp…
the architecture of deepseek's new 4.1 flash is a step change for long context tasks. we are seeing 2x more conc. sessions maintaining tps on a single node, compared to GLM 5.3 Flash.
while there are many diff agentic use cases, at hopper, we are seeing demand mainly for low latency and batched inference. For the later, we are currently using deepseek v4 flash(at 51M tok/min) and 4.1 flash gives us a new angle to get more economical at long-context tasks like code reviews, test generation, repo wide Q&A, oncall & log analysis etc
Combined, the previous ~900 ms latency spikes dropped to ~400 ms - with far fewer spikes overall.
We’re now seeing:
• >200 tok/s on general prompts
• >800 tok/s on predictable in-domain prompts
Hopper retweeted
chinese labs are proving open weights and revenue aren't in tension.
zhipu did $142m in revenue in all of h1, while its august exit run-rate was $1.6b. in other words, one month of revenue is now roughly the entire first half.
api went from 15.2% to 86.5% of revenue in a year, while token volume grew ~40x and inference cost per token fell ~80% all on models they gave away.
deepseek is playing a similar game for somewhat different reasons. their approach seems to be to price inference so the hardware pays itself back in ~10 months, even while saying demand is inelastic enough that they could probably charge much more.
the emerging strategy is pretty simple: give away the weights to maximize distribution and token volume, then win on serving economics.
the profit pool may be moving from access to intelligence toward serving intelligence, which would make open weights less of a sacrifice and more of a customer acquisition strategy.
and this is before you get to moonshot or alibaba, who are certainly not struggling w/revenue growth either.
We're proud to be the fastest inference provider for PhoneLLM.
All served from the US only, starting on day 0, with ZDR by default.
Introducing PhoneLLM, an open model for voice agents.
GPT 5.6 Terra performance on typical voice agent tasks at 1/3 the latency and 1/18 the cost.
For voice agents, we need models that are both very low latency and very good at tool calling and instruction following.
There's a trade-off here, and we often have to compromise on either latency or capability when building voice agents. With PhoneLLM (and the training and data stack that made this model possible) we're fixing this problem.
For the last couple of years, most of the effort in frontier model development has gone towards leveraging test-time compute. Which is awesome! Models of all shapes and sizes are available that perform really, really well ... if you have "thinking" turned on for your model.
But if you need your agent to respond at voice conversation speed, you can't use thinking models.
PhoneLLM is a full-weights fine-tune of NVIDIA Nemotron Nano 30B. We trained on a wide range of real-world telephone and customer support use cases. The training focused on taking the excellent Nano 30B base capabilities and teaching the model to do typical voice agent tasks with thinking disabled.
The results are really good: accurate tool calling and concise, on-topic responses in long conversations.
And fast: TTFAT measured server-side is <100ms if you run PhoneLLM on a lightly loaded B200. :-)
But seriously, when we characterize model latency, we do it with full, end-to-end, batched request simulations using real Pipecat voice agent pipelines.
You can serve more than 80 concurrent agents on a single B200 with P95 end-to-end TTFAT <600ms. Including network overhead. That's an LLM cost-per-minute around $0.0025. (1/4 of a cent.) At a latency lower than any third-party API offers today.
More details about this model, including weights on @huggingface, how to spin it up with one click on Modal, and a starter project repo you can clone, are in the thread ...
Breeze TTS 2 is live day 0 on Hopper
Try Here: withhopper.com/console/tts
From our tests, it's exceeds 11Labs and matches sonic 3.5 in naturalness. Kudos to their team for pushing the open weights frontier.
Breeze TTS 2 is now the leading Open Weights TTS model in the Artificial Analysis Provider Voices Speech Arena, surpassing Fish Audio S2 Pro by 90 Elo points
Breeze TTS 2 is the latest TTS model from @BreezeBlueX, supporting 50 languages, voice generation from text prompts, and streaming generation. Its weights are openly available on Hugging Face.
Key takeaways:
➤ Provider Voices: Breeze TTS 2 ranks #1 among Open Weights TTS models, leading the next best Open Weights model, Fish Audio S2 Pro at 1,125, by 90 Elo points. It also ranks #6 overall out of 100+ models, with an Elo of 1,215.
➤ Controlled Voices: Breeze TTS 2 ranks #3 among Open Weights TTS models, with the same Elo as Fish Audio S2 Pro at 1,002. It trails the leading Open Weights model, Mistral's Voxtral TTS, at 1,010 by 8 Elo points, and ranks #16 out of 39 models overall with an Elo of 1,002.
➤ Speed: Breeze TTS 2 processes 45 characters per second, trailing the leading Open Weights model, Fish Audio S2 Pro, at 102 characters per second.
➤ Price: Breeze TTS 2 is priced at $34 per 1M characters on BreezeBlue's hosted endpoint, more expensive than competitor Open Weights model, Fish Audio S2 Pro, at $15 per 1M characters, though weights are also available for self-hosting both models.
See more details and listen to samples below ⬇️
Hopper retweeted
2026 feels very different from previous years.
Single-GPU optimization is becoming increasingly commoditized: profile the timeline, optimize hotspots, adopt TRT-LLM/FlashInfer kernels, have agents write kernels, or borrow ideas across engines. Spec(EAGLE-3/DSpark/DFlash2) and quant(NVFP4 with ModelOpt) are all becoming increasingly standardized.
The real differentiation is moving to system-level optimization: PD disaggregation and KV stores — maximizing TPM/GPU under TPS SLAs while pushing cache hit rates higher.
Engine optimization is becoming increasingly deterministic and programmatic.
But there are still a few months left in 2026. We’ll still see plenty of interesting work ahead — new models, new hardware, and new ideas on existing ones. Looking forward to it!
Models are increasingly trained to be token-maxxers.
We evaluated GLM-5.3 (low) vs GLM-5.2 (low) on IFBench for realtime workloads and it surprisingly regressed.
GLM-5.3 API is now live.
- Built for coding, defensive cybersecurity, and long-horizon agentic tasks
- Priced the same as GLM-5.2
- Available via the official API and partner model gateways
Get started: docs.z.ai/guides/llm/glm-5.3
