nullHawk (@null_hawk) header
nullHawk (@null_hawk) profile picture

mix of engineering and research | building voice @rumik_ai

Joined August 2023
Photo by nullHawk (@null_hawk)Photo by nullHawk (@null_hawk)Photo by nullHawk (@null_hawk)Photo by nullHawk (@null_hawk)Photo by nullHawk (@null_hawk)Photo by nullHawk (@null_hawk)Photo by nullHawk (@null_hawk)Photo by nullHawk (@null_hawk)Photo by nullHawk (@null_hawk)Photo by nullHawk (@null_hawk)
Is the speaker token really what keeps a voice consistent throughout an utterance? The answer turned out to be far more interesting than a speaker tag.
2
6
15
1,790
the main problem with current full duplex models is that almost everyone is iterating on the same moshi architecture which imo has some real bottlenecks. in moshi every text token is locked 1:1 to an audio frame that gets spoken right away (apart from padding and special tokens). because of this rigid lockstep the native architecture cant generate a burst of text without advancing audio time. if the model generates 20 text tokens time has to move forward 20 frames so exactly 1.6 seconds of audio gets synthesized alongside. you cant pause the audio clock to let the LLM write out a block of text. within base moshi you could in theory train it to output reasoning text or JSON tool calls while forcing the audio stream to output silence tokens. but since the clock cant be decoupled that "silent thinking" still runs at 12.5 tokens per second. a 50 token thought chain or tool call means the user sits through 4 solid seconds of awkward dead air. the other option is synthetic data where it keeps talking while producing the JSON or the thinking. but there is no data in the wild for that so you generate a lot of it and now your data is the bottleneck. the voice will only ever be as natural as the most natural TTS you can use it will always carry the quirks of the harness you built to create it and if whoever is making that data cant think through the scenarios of dyadic conversations well, you are cooked. so people came up with delegation system which is a pain in itself. you train the frontend model to be smart enough to hold a decent conversation and to hand off tool calling or thinking to a much larger, more capable model. this is engineering and its the only thing that has made the 2 or 3 full duplex models we have like openai gpt live possible at all. thinking cant be done in speech. the signal to noise ratio is low and there isnt enough speech data to give the model a good enough world model in its weights. how much data is there of people doing maths out loud? even we humans dont think in speech. yes there is "thinking out loud" but speech is not the medium humans think in. thinking or reasoning needs a dense representation like text or maybe some latent but pure speech is a fuck all dream. i saw dessa a few days ago. it seems good and sounds good but god knows how much data they trained it on and even it needs text, or some good engineering like ASR over its thinking speech, to actually interact with the real world or be even remotely useful. train on all the call centre data u want then ask it a basic question. this also reminds me of a question from @juliarturc something like: if you converted every text available into speech and trained on that, what would happen? idk it might be smart or it might not. it depends on a lot of things like whether the speech encoder you use preserves everything needed for speech understanding. codecs are good enough these days but i believe they are still a bottleneck. even if you solve that,context window is a problem and so is context degradation in speech compared to text LLMs current speech models have the memory of a goldfish. there are just too many bottlenecks but at the rate we are going we might get there next year only if a lot of people come out in the open and open source the data they are sitting on and stop giving fake demos by training moshi on their privately acquired data.
Full duplex models are having a moment now, are people using it in production? Why not? What is your top reason for not using full duplex models?
21%Cost
5%Latency
40%Control
34%Reasoning / tool call
209 votes • Final results
20
33
8
202
22,652
People have yet to realise just how powerful this is and the possibilities it unlocks for solving real consumer problems. We’ve invested deeply in voice technology, building our own speech models in-house, so this use case is a natural fit for us. @iradotso ftw
Replying to @iradotso
@iradotso I asked IRA to find me a paramedics for injection near my place. It searched and called three places and then sent me transcript of the call made and made the booking at Sanocare which were the cheapest and quickest. Crazy!! @lets_dig_deeper
2
5
1
10
991
trust me guys, this one is the coolest thing i have ever tested. as a self proclaimed introvert, who doesn't likes to talk to people on call. she called the PG owners and brokers on my behalf and talked to them and negotiated the deal.... and hunted PG for me end to end (which seems like a nightmare task for me). it reminds me my tasks lined up for the day, or any important mails or msges for me that i need to see etc. just as a personal assitent. we had LLMs and agents before this but nothing felt so real. obv there are alternates in market rn, i have tested them too, but this somehow feels soo comfortable to use. and does the tasks properly. @iradotso FTW
i was one of the early members and can i just say @iradotso is awesome . she helps me with numerous things like every morning she goes to arxiv and find papers for me and gives me a good read . one time my furlenco carpenter didnt show up and she chased him to make sure he came the next day all by herself. she can call people on behalf of u and email and handle everything.
2
6
188
nullHawk retweeted
(1/n) i'm putting out sneak peeks of a new neural audio codec that i've been working on recently. it matches @kyutai_labs mimi on ~1200x less data. 5,770 hours of english audio. no chinese, hindi, or japanese in training. yet it matches mimi's STOI on all four languages.
11
19
1
163
8,090
nullHawk retweeted
I built a port of Microsoft’s X++ windows only compiler for Apple Silicon While building at Lab0.ai. I couldn’t find an open-source X++ compiler port for macOS, so I built one of my own🔥 Still expanding support. Contributions welcome: github.com/lab0-ai/xppc-maco…
3
5
19
639
do checkout out guys!
rumik oss 1 is now trending no.4 on @huggingface ❤️
1
9
374
if you value intelligence above all other human qualities, you’re gonna have a bad time
898
3,522
1,180
24,370
9,971,103
we also used our in-house voice conversions models and ECAPA-TDNN for speaker separation and introducing new speakers... weights are out, go check it out! : huggingface.co/rumik-ai/rumi… publishing tech report soon 👀
introducing rumik oss 1 india's first state of the art open source, fully pre-trained voice model capable of speaking 20+ languages trained on only 66k hours of data, it beats even the top closed models on multiple benchmarks including emotion realism weights are now live 👇🏻
1
3
45
4,099
You just cloned a 50,000-line repo. Where do you even start? I built kenly for exactly that moment — it walks you through any codebase like a senior dev giving you KT. Opens the files, highlights the exact lines, explains the flow, answers your questions. Live in your terminal
2
2
3
110
nullHawk retweeted
jensen huang shared an open letter on why open models matter. we made it brief, then gave it to the voice model we’re opening next. throughout the brief, it switches between 8 languages without losing its voice. india has never lacked voices. it’s time the models carrying them are built here, and built in the open
6
8
56
4,814
hahaha, and this will be the anthem of this event 😂
wait is this a real basecamp bangalore event???
4
2
23
3,901
nullHawk retweeted
we’re open sourcing a voice model built for how people actually speak. 20+ languages. native accents. mid-sentence language switching. Expressive yours soon.
36
44
9
610
29,574
nullHawk retweeted
less latency. more presence.
2
25
1,003
nullHawk retweeted
silk mulberry just unlocked 8 new voices. who are you picking?
1
2
1
19
1,122
nullHawk retweeted
meet ira. the ai that remembers what mattered, asks about your day, and gives the small things somewhere to go.
3
4
31
1,886
Today, we are releasing verifiers v1 — an overhaul of our environment stack for the modern era of agentic RL and evals. We decompose environments into a taskset, a harness, and a runtime. Run complex agentic tasks like coding and computer use at scale, in any harness.
34
87
39
838
275,650
spent a day optimizing seed-vc. assumed batching was the free lunch. It was 1.4x. the real cost: the reference prompt is prepended to every chunk, diffused through all 15 steps, then thrown away. you can't cache it, because the DiT is bidirectional. you can only make it shorter.
1
11
237