Come build a superhuman tutoring agent with me @speak

@speak, we've been working closely with OpenAI on GPT-Live-1 for the past few months, building with an entirely new class of voice model model that continuously listens and speaks at the same time. Here's some of what we learned.

Congrats to the @indent team and @sthupukari on launching! We've been using Indent for well over a year now @speak and it's organically become one of our few internal agents adopted by almost the entire team, and it helps us make more data-driven decisions daily. I've been really impressed with the significant progress they've made on a lot of the nitty gritty hard parts of enterprise agent uptake including auth/IAM and team memory and context. Onwards!

Amazing technical writeup worth reading in full - kudos to the gpt-live team on this work! It's clear that full duplex/bidi is the future of voice and it's really cool to peek under the hood at the complex systems engineering that makes it all "just work". Some observations/notes for voice agent builders: - The whole post is a huge flex, but especially turning the “improve startup latency” ticket into a new webrtc handshake protocol. Generational scope creep. @juberti what was the original estimate on that issue? :) - The voice must flow. Nothing matters but continuous realtime audio flow and smoothness. Be extremely clear in your voice agent’s system design about division of responsibilities between the realtime band and async channels/sidecars, and treat the async RPC boundary as a first-class conversational & UX design principle. - Many other details are direct consequences of optimizing for voice fluidity: prompt caching and avoiding audio-blocking prefills, live context compaction with blue-green-style cutovers, smoothly handling long-running voice sessions (that are much longer than gpt-live’s context window) - Future models and versions of gpt-live will continue to become more intelligent and handle more tasks natively in the realtime band. As inference speed improves, the model will also be able to spend more tokens thinking and delegating within the realtime response latency budget. This means both the actual delegation and “perceived delegation” boundary will push out. - Interesting details on the turn-transcript heuristics, e.g. “we prioritize coherence in the displayed assistant responses even when the user speaks in the middle.” The full bidirectional/duplex model arch breaks free of modeling the convo as a traditional ordered list of chat messages, but heuristic/projected semantic turn messages are still critical for most voice agent harness systems you’d want to build around the core live model. - Realtime full-duplex sessions load your voice agent application server’s CPU (stream handlers, queues, network paths) more heavily than almost any other ordinary application type. Think about the fact that each full-duplex session processes ~100 audio frames per second across both directions. This is a very nontrivial driver of higher infra/capacity costs when scaling realtime voice experiences. - The bar has been set extremely high for long-running voice agent infra providers, with a lot of additional complexity to support very smooth realtime voice model inference with an explicit temporal dimension. It’ll be interesting to see the industry patterns that emerge here.