I'm having fun with this Voice AI Superheroes theme that Opus 5.5, Gemini Flash, and I cooked up. Next up in the superhero hot seat is Caroline di Vittorio, engineer at LangChain. Caroline leads voice observability and evals on the LangSmith platform. We talked about how you measure voice agent success, latency (of course), helping customers get started building voice agents, helping customers scale in production, and the explosion of new voice AI use cases happening right now. Caroline is talking about voice agent best practices next Thursday (October 29th) at Deepgram Speak in San Francisco. Come hang out with us at the conference. It's free, and a pretty big cross-section of the voice AI developer community will be there.

Our friends at @modal shipped a new Sticky Sessions feature yesterday that's extremely useful for building very low-latency voice agents and other realtime AI applications. Today, you can build (and serve at scale) voice agents that deliver 500ms voice-to-voice response times. But this is 3-4x faster than most of the voice agents we see in production. To be fast, you need a few things: fast models; good orchestration of turn detection and data streaming between models in your agent pipeline; fast WebRTC or telephony client-server network routing; and optimized placement/hosting/routing for your agent and inference containers. Sticky Sessions makes it easy to route a specific user's requests to the same container every time. If you're an infrastructure person, you might describe this as session-aware serverless routing. The Modal team did a lot of work here to get the details right, so things like load balancing and auto-scaling work nicely in conjunction with Sticky Sessions. The video is a demo clip showing a voice agent running on Modal with a 500ms voice-to-voice latency. Ben Shababo, who built this demo, used: - Modal's Qwen 3 ASR service for transcription - Modal's PhoneLLM Mini service as the agent brain - The brand new @GradiumAI Phonon text-to-speech model for the voice - Pipecat as the agent framework and WebRTC routing layer - Modal Sticky Sessions A few really cool things, here. All the models are running co-located on a mixture of GPUs and CPUs: L4, B200, and the Phonon model is running on CPU. Running Phonon on the CPU means that voice audio starts streaming in <50ms and you don't have to manage a GPU cluster for TTS. There are no extra network hops or network overhead in any of the routing. And the agent container uses Modal memory snapshots so a new agent (orchestration code, Phonon model, and the Smart Turn model) are all ready and responding to new requests in under 2 seconds. If you've ever built out voice agent cluster infrastructure on top of, say, Kubernetes, you know how impressive all of the above attributes are! A link to Ben's repo is in the thread below.

Deepgram is hosting a voice agents developer conference in San Francisco on October 29th. I'm speaking at the conference, we have some fun Pipecat announcements to make, and lots of friends are either attending, or speaking, or both. I figured this was an opportunity to record some conversations with people in the voice AI ecosystem about what they're interested in right now, what they're talking about at Speak, and what's new since we last caught up. @bnicholehopkins, founder of @covaldev, has been building simulation and testing tools for voice agents since 2024. Before Coval, Brooke worked on simulation tooling at Waymo. Brooke and I riffed about how deployments of enterprise voice AI are growing incredibly fast right now, how we think about building with open source components and models, applying systems engineering principles to agentic AI development, and aiming for "self-improving agent loops." I've been experimenting with prompting Opus to create motion graphics and video effects. A discerning 11-year old of my acquaintance called the result, here, "kind of cringe." But I kind of like it. See what you think. :-)

Fast is better than slow. We've got two new PhoneLLM models optimized for conversational voice AI. PhoneLLM Mini (30B) — When you need the lowest possible latency (sub-100ms TTFAT), smallest model size, and lowest cost. Good multi-turn conversation behavior and tool-calling accuracy in a model that can run >100 concurrent agents on a B200 and fits on an RTX 5090. PhoneLLM Max (120B) — A "microthinking" model for complex conversational tasks. Sub-200ms TTFAT. SOTA task completion performance at 1/10th the latency and cost of models with similar capabilities. Both of these models are full-weights fine-tunes of @NVIDIAAI Nemotron models. Mini is based on Nemotron 3 Nano. Max is based on the brand new Nemotron 3.5 Super VL. Both of these models are available as open weights on HuggingFace, and from four inference partners: AWS, Baseten, Cloudflare, and Modal.

I'll be at Deepgram Speak on October 29th with 500 or so new and old friends who build voice AI agents and applications. As mini embedded Pipecat and I were just discussing, I'm really excited about Deepgram Speak. We're working on some very fun Pipecat stuff for the conference. If you're interested in models for realtime AI, building production voice agents, or what the future holds for voice and AI-native software, please join us in San Francisco.

We have some fun voice AI stuff to talk about on the @NVIDIAAI Nemotron Labs livestream tomorrow: fine-tuning audio and LLM models for specific agents and use cases. Join us live if you want to chat and ask questions. 10:00am Pacific Time on Tuesday.