
mix of engineering and research | building voice @rumik_ai
Joined August 2023
- Tweets154
- Following203
- Followers126
- Likes796
Pinned Tweet
Is the speaker token really what keeps a voice consistent throughout an utterance?
The answer turned out to be far more interesting than a speaker tag.
nullHawk retweeted
the main problem with current full duplex models is that almost everyone is iterating on the same moshi architecture which imo has some real bottlenecks.
in moshi every text token is locked 1:1 to an audio frame that gets spoken right away (apart from padding and special tokens). because of this rigid lockstep the native architecture cant generate a burst of text without advancing audio time. if the model generates 20 text tokens time has to move forward 20 frames so exactly 1.6 seconds of audio gets synthesized alongside. you cant pause the audio clock to let the LLM write out a block of text.
within base moshi you could in theory train it to output reasoning text or JSON tool calls while forcing the audio stream to output silence tokens. but since the clock cant be decoupled that "silent thinking" still runs at 12.5 tokens per second. a 50 token thought chain or tool call means the user sits through 4 solid seconds of awkward dead air.
the other option is synthetic data where it keeps talking while producing the JSON or the thinking. but there is no data in the wild for that so you generate a lot of it and now your data is the bottleneck. the voice will only ever be as natural as the most natural TTS you can use it will always carry the quirks of the harness you built to create it and if whoever is making that data cant think through the scenarios of dyadic conversations well, you are cooked.
so people came up with delegation system which is a pain in itself. you train the frontend model to be smart enough to hold a decent conversation and to hand off tool calling or thinking to a much larger, more capable model. this is engineering and its the only thing that has made the 2 or 3 full duplex models we have like openai gpt live possible at all.
thinking cant be done in speech. the signal to noise ratio is low and there isnt enough speech data to give the model a good enough world model in its weights. how much data is there of people doing maths out loud? even we humans dont think in speech. yes there is "thinking out loud" but speech is not the medium humans think in. thinking or reasoning needs a dense representation like text or maybe some latent but pure speech is a fuck all dream.
i saw dessa a few days ago. it seems good and sounds good but god knows how much data they trained it on and even it needs text, or some good engineering like ASR over its thinking speech, to actually interact with the real world or be even remotely useful. train on all the call centre data u want then ask it a basic question.
this also reminds me of a question from @juliarturc something like: if you converted every text available into speech and trained on that, what would happen? idk it might be smart or it might not. it depends on a lot of things like whether the speech encoder you use preserves everything needed for speech understanding. codecs are good enough these days but i believe they are still a bottleneck.
even if you solve that,context window is a problem and so is context degradation in speech compared to text LLMs current speech models have the memory of a goldfish.
there are just too many bottlenecks but at the rate we are going we might get there next year only if a lot of people come out in the open and open source the data they are sitting on and stop giving fake demos by training moshi on their privately acquired data.
nullHawk retweeted
People have yet to realise just how powerful this is and the possibilities it unlocks for solving real consumer problems. We’ve invested deeply in voice technology, building our own speech models in-house, so this use case is a natural fit for us. @iradotso ftw
Replying to @iradotso
@iradotso
I asked IRA to find me a paramedics for injection near my place. It searched and called three places and then sent me transcript of the call made and made the booking at Sanocare which were the cheapest and quickest. Crazy!! @lets_dig_deeper
trust me guys, this one is the coolest thing i have ever tested. as a self proclaimed introvert, who doesn't likes to talk to people on call. she called the PG owners and brokers on my behalf and talked to them and negotiated the deal.... and hunted PG for me end to end (which seems like a nightmare task for me).
it reminds me my tasks lined up for the day, or any important mails or msges for me that i need to see etc. just as a personal assitent. we had LLMs and agents before this but nothing felt so real.
obv there are alternates in market rn, i have tested them too, but this somehow feels soo comfortable to use. and does the tasks properly.
@iradotso FTW
i was one of the early members and can i just say @iradotso is awesome . she helps me with numerous things like every morning she goes to arxiv and find papers for me and gives me a good read .
one time my furlenco carpenter didnt show up and she chased him to make sure he came the next day all by herself. she can call people on behalf of u and email and handle everything.
nullHawk retweeted
(1/n) i'm putting out sneak peeks of a new neural audio codec that i've been working on recently.
it matches @kyutai_labs mimi on ~1200x less data.
5,770 hours of english audio. no chinese, hindi, or japanese in training. yet it matches mimi's STOI on all four languages.
nullHawk retweeted
I built a port of Microsoft’s X++ windows only compiler for Apple Silicon
While building at Lab0.ai. I couldn’t find an open-source X++ compiler port for macOS, so I built one of my own🔥
Still expanding support. Contributions welcome: github.com/lab0-ai/xppc-maco…
do checkout the mlx ports!!!
huggingface.co/rumik-ai/rumi…
huggingface.co/rumik-ai/rumi…
12
1,590
nullHawk retweeted
if you value intelligence above all other human qualities, you’re gonna have a bad time
we also used our in-house voice conversions models and ECAPA-TDNN for speaker separation and introducing new speakers... weights are out, go check it out! : huggingface.co/rumik-ai/rumi…
publishing tech report soon 👀
nullHawk retweeted
You just cloned a 50,000-line repo. Where do you even start?
I built kenly for exactly that moment — it walks you through any codebase like a senior dev giving you KT. Opens the files, highlights the exact lines, explains the flow, answers your questions. Live in your terminal
jensen huang shared an open letter on why open models matter.
we made it brief, then gave it to the voice model we’re opening next.
throughout the brief, it switches between 8 languages without losing its voice.
india has never lacked voices.
it’s time the models carrying them are built here, and built in the open
nullHawk retweeted
Today, we are releasing verifiers v1 — an overhaul of our environment stack for the modern era of agentic RL and evals.
We decompose environments into a taskset, a harness, and a runtime.
Run complex agentic tasks like coding and computer use at scale, in any harness.
























