This is awesome. Multi-character interactions in Unreal Engine, completely LLM-driven and unscripted. You can spin your own version of this up pretty easily. This is another Pipecat + Jev experiment from @JonPTaylor. Watch Jon walk around the game world, talk to the characters, and ask them to do things. They interact with Jon, with each other, and with the world. Tech stack: - One Pipecat agent process syncing state with Unreal via the Pipecat RTVI protocol. - Three non-player characters. Each character is a Pipecat worker with its own LLM prompt and tools. Each character also has its own TTS service so its voice is unique. - When Jon (the human) talks, Jev assesses Jon's intent and which character(s) he is talking to. - Unreal 5.8 with Spatial Audio. (Watch the video with headphones on!) - Pipecat C++ client embedded in Unreal Engine.

Introducing PhoneLLM, an open model for voice agents. GPT 5.6 Terra performance on typical voice agent tasks at 1/3 the latency and 1/18 the cost. For voice agents, we need models that are both very low latency and very good at tool calling and instruction following. There's a trade-off here, and we often have to compromise on either latency or capability when building voice agents. With PhoneLLM (and the training and data stack that made this model possible) we're fixing this problem. For the last couple of years, most of the effort in frontier model development has gone towards leveraging test-time compute. Which is awesome! Models of all shapes and sizes are available that perform really, really well ... if you have "thinking" turned on for your model. But if you need your agent to respond at voice conversation speed, you can't use thinking models. PhoneLLM is a full-weights fine-tune of NVIDIA Nemotron Nano 30B. We trained on a wide range of real-world telephone and customer support use cases. The training focused on taking the excellent Nano 30B base capabilities and teaching the model to do typical voice agent tasks with thinking disabled. The results are really good: accurate tool calling and concise, on-topic responses in long conversations. And fast: TTFAT measured server-side is <100ms if you run PhoneLLM on a lightly loaded B200. :-) But seriously, when we characterize model latency, we do it with full, end-to-end, batched request simulations using real Pipecat voice agent pipelines. You can serve more than 80 concurrent agents on a single B200 with P95 end-to-end TTFAT <600ms. Including network overhead. That's an LLM cost-per-minute around $0.0025. (1/4 of a cent.) At a latency lower than any third-party API offers today. More details about this model, including weights on @huggingface, how to spin it up with one click on Modal, and a starter project repo you can clone, are in the thread ...