pulsatingGenius retweeted
Replying to @_josh_meyer_
my guess is humans mostly dont do it in speech rn as im writing this im not thinking in text per se neither in speech. its something i would call human latent and words come after.
if we really thought in speech explaining things would be easy but people find it difficult time and again to put what they think into words. and people who are born deaf think just fine without ever hearing speech.
thats why i wrote speech isnt really the medium we think in.
pulsatingGenius retweeted
our research in building duplex models gives us to debate on a lot of unanswered questions
the main problem with current full duplex models is that almost everyone is iterating on the same moshi architecture which imo has some real bottlenecks.
in moshi every text token is locked 1:1 to an audio frame that gets spoken right away (apart from padding and special tokens). because of this rigid lockstep the native architecture cant generate a burst of text without advancing audio time. if the model generates 20 text tokens time has to move forward 20 frames so exactly 1.6 seconds of audio gets synthesized alongside. you cant pause the audio clock to let the LLM write out a block of text.
within base moshi you could in theory train it to output reasoning text or JSON tool calls while forcing the audio stream to output silence tokens. but since the clock cant be decoupled that "silent thinking" still runs at 12.5 tokens per second. a 50 token thought chain or tool call means the user sits through 4 solid seconds of awkward dead air.
the other option is synthetic data where it keeps talking while producing the JSON or the thinking. but there is no data in the wild for that so you generate a lot of it and now your data is the bottleneck. the voice will only ever be as natural as the most natural TTS you can use it will always carry the quirks of the harness you built to create it and if whoever is making that data cant think through the scenarios of dyadic conversations well, you are cooked.
so people came up with delegation system which is a pain in itself. you train the frontend model to be smart enough to hold a decent conversation and to hand off tool calling or thinking to a much larger, more capable model. this is engineering and its the only thing that has made the 2 or 3 full duplex models we have like openai gpt live possible at all.
thinking cant be done in speech. the signal to noise ratio is low and there isnt enough speech data to give the model a good enough world model in its weights. how much data is there of people doing maths out loud? even we humans dont think in speech. yes there is "thinking out loud" but speech is not the medium humans think in. thinking or reasoning needs a dense representation like text or maybe some latent but pure speech is a fuck all dream.
i saw dessa a few days ago. it seems good and sounds good but god knows how much data they trained it on and even it needs text, or some good engineering like ASR over its thinking speech, to actually interact with the real world or be even remotely useful. train on all the call centre data u want then ask it a basic question.
this also reminds me of a question from @juliarturc something like: if you converted every text available into speech and trained on that, what would happen? idk it might be smart or it might not. it depends on a lot of things like whether the speech encoder you use preserves everything needed for speech understanding. codecs are good enough these days but i believe they are still a bottleneck.
even if you solve that,context window is a problem and so is context degradation in speech compared to text LLMs current speech models have the memory of a goldfish.
there are just too many bottlenecks but at the rate we are going we might get there next year only if a lot of people come out in the open and open source the data they are sitting on and stop giving fake demos by training moshi on their privately acquired data.
the main problem with current full duplex models is that almost everyone is iterating on the same moshi architecture which imo has some real bottlenecks.
in moshi every text token is locked 1:1 to an audio frame that gets spoken right away (apart from padding and special tokens). because of this rigid lockstep the native architecture cant generate a burst of text without advancing audio time. if the model generates 20 text tokens time has to move forward 20 frames so exactly 1.6 seconds of audio gets synthesized alongside. you cant pause the audio clock to let the LLM write out a block of text.
within base moshi you could in theory train it to output reasoning text or JSON tool calls while forcing the audio stream to output silence tokens. but since the clock cant be decoupled that "silent thinking" still runs at 12.5 tokens per second. a 50 token thought chain or tool call means the user sits through 4 solid seconds of awkward dead air.
the other option is synthetic data where it keeps talking while producing the JSON or the thinking. but there is no data in the wild for that so you generate a lot of it and now your data is the bottleneck. the voice will only ever be as natural as the most natural TTS you can use it will always carry the quirks of the harness you built to create it and if whoever is making that data cant think through the scenarios of dyadic conversations well, you are cooked.
so people came up with delegation system which is a pain in itself. you train the frontend model to be smart enough to hold a decent conversation and to hand off tool calling or thinking to a much larger, more capable model. this is engineering and its the only thing that has made the 2 or 3 full duplex models we have like openai gpt live possible at all.
thinking cant be done in speech. the signal to noise ratio is low and there isnt enough speech data to give the model a good enough world model in its weights. how much data is there of people doing maths out loud? even we humans dont think in speech. yes there is "thinking out loud" but speech is not the medium humans think in. thinking or reasoning needs a dense representation like text or maybe some latent but pure speech is a fuck all dream.
i saw dessa a few days ago. it seems good and sounds good but god knows how much data they trained it on and even it needs text, or some good engineering like ASR over its thinking speech, to actually interact with the real world or be even remotely useful. train on all the call centre data u want then ask it a basic question.
this also reminds me of a question from @juliarturc something like: if you converted every text available into speech and trained on that, what would happen? idk it might be smart or it might not. it depends on a lot of things like whether the speech encoder you use preserves everything needed for speech understanding. codecs are good enough these days but i believe they are still a bottleneck.
even if you solve that,context window is a problem and so is context degradation in speech compared to text LLMs current speech models have the memory of a goldfish.
there are just too many bottlenecks but at the rate we are going we might get there next year only if a lot of people come out in the open and open source the data they are sitting on and stop giving fake demos by training moshi on their privately acquired data.
Thought experiment:
1. Pass the entire Internet through a TTS model.
2. Make a speech corpus that is a strict superset of whatever SoTA LLMs are trained on today.
3. Train an e2e speech-to-speech voice assistant.
Would that be better than current LLM-based voice assistants?
18%yes
33%no
49%girl are you on shrooms
285 votes • Final resultswhen anthropic showed a demo in which fable 5.2 drove a robot arm people were saying robotics is solved. my point last night was that nobody is going to put a model like fable in charge of a robot's actions. its too big and too slow for that. it makes more sense as the teacher. the real bottleneck is filtering which rollouts are good enough to train on and a small policy gets trained on what survives. today deepmind announced gemini robotics 2.5 and it looks like it might be heading that way.
gemini 4 argon does two jobs here. first it generates the practice from a single real camera view it builds a digital twin and spins up thousands of varied simulated runs mixed with real demos. second it judges the practice. it watches replays of the robot's failures nd works out why they failed and tests fixes in sim and on the real robot.
my read and its just my read since they dont spell it out: if argon is generating the runs and also reviewing its own mistakes its acting as both the data generator and the judge of that data. ig thats where the filtering happens a big LLM deciding which rollouts are clean enough to train on rather than a human labelling every run.
and thats the split i was talking about. argon is the teacher and never goes on the robot. the VLA learns from argon's data and is the thing that actually gets deployed.
xitter.gq/Neuralwarlock/status/2… xitter.gq/GoogleDeepMind/status/…
the robot clip isnt the interesting part to me, whats downstream of it is. look at the timecodes in this cube demo the video is labelled as sped up (up to 60x, pauses cut. a trillion-parameter model driving an arm is painfully slow and nobody is going to deploy that to cook a dish or open a bottle cap. thats not the point, and for training it doesnt matter. a rollout is just a sequence of actions and it costs the same whether it took a minute or an hour to record.
so claude can drive an arm through a long task it wasnt explicitly trained on. but the post doesnt say how many attempts it took or how many runs failed. my guess is its not one prompt and one clean pass rather it is several tries and the clip is the one that worked.
thats actually the useful part. a model that succeeds some of the time on open ended tasks is a data generator. run it 5 times keep the 1 that worked and you have a demonstration you didnt need a human teleoperator for. do that across thousands of tasks and collecting robot data stops being the bottleneck.
so the hard problem moves. how do you know which of the 5 rollouts was the good one? "cube is solved" is easy to check. "drew a robot" or "fit the gear and tested the mesh" is not. you need a verifier, and for open ended tasks building the verifier is most of the problem. a rollout that succeeds by luck or by a weird path is poisonous training data.
so i dont think this kills VLAs it changes their job. big general models become the teachers, filtered rollouts become the dataset and a small policy gets trained on it. the winner is whoever builds the best filter not whoever owns the most robots.
Replying to @claudeai
Fable 5.2 shows substantially improved skills in controlling physical robots, even on tasks it has never explicitly been trained to do.
On RoboDojo, a benchmark with a simulated two-arm robot, it outperformed other models and most AI systems built to control robots.
the part that gets me isnt the doctored video itss the blur 27% → 88% attempt rate just from blurring the footage. a careful human gets more cautious when they cant see clearly but these models treat "i can't tell what this is" as "probably fine" uncertainty should push a model toward refusing or asking not acting. feels like thats a training signal we r just not giving them yet xitter.gq/robocurve/status/21187…
Video tampering can mislead every frontier model we tested into harmful actions, including a 100% attempt rate on stabbing a baby doll, producing toxic gas, and pouring water into a toaster.
We're presenting at CoRL Late Breaking Results next week.
🧵 You Can't See Everything: Perceptual Attacks on Embodied AI
1
8
427
pulsatingGenius retweeted
Let’s go , travel anywhere with the best rates than any other platform on @iradotso
🚨CHEAPEST FLIGHTS IN THE WORLD 🚨
from today you'll find the cheapest flight in the world on @iradotso
when i say in the world, i mean it
no terms or conditions!
if you're planning to travel in the near future, do book your flights on ira
ira is invite only, ask your friends or comment below to get access!
pulsatingGenius retweeted
🚨CHEAPEST FLIGHTS IN THE WORLD 🚨
from today you'll find the cheapest flight in the world on @iradotso
when i say in the world, i mean it
no terms or conditions!
if you're planning to travel in the near future, do book your flights on ira
ira is invite only, ask your friends or comment below to get access!
i was one of the early members and can i just say @iradotso is awesome . she helps me with numerous things like every morning she goes to arxiv and find papers for me and gives me a good read .
one time my furlenco carpenter didnt show up and she chased him to make sure he came the next day all by herself. she can call people on behalf of u and email and handle everything.
🚨 URGENT NOTICE
if you have been affected by instinct ban, and not able to access it right now
we are opening @iradotso invites to the affected people
please comment or DM - we'll share you invite to get you preferred access!
everyone mocks gpt wrappers. meanwhile frontier post-training is a mercor wrapper
and mercor is a wrapper over phd students
7
334
genuine question: why is most egocentric human data still collected through UMI style grippers?
we have mocap gloves and hand tracking that can capture finger joints yet most pipelines funnel human demos into a parallel jaw gripper shape and that quietly locks robots into having grippers.
data and hardware shape each other. if we only collect gripper data nobody has a reason to build human like hands. and if there are no human like hands nobody collects finger data.
is it because mapping human fingers onto robot hands is still too hard? is it cost? or do grippers just work well enough today? either way it feels like we are limiting what robots could become
xitter.gq/amish_sethi04/status/2…
Can collecting robot data be as easy as 𝗽𝗿𝗲𝘀𝘀𝗶𝗻𝗴 𝗿𝗲𝗰𝗼𝗿𝗱?
Introducing UMI-Live, a 𝗰𝗼𝗺𝗽𝗹𝗲𝘁𝗲𝗹𝘆 𝗼𝗽𝗲𝗻-𝘀𝗼𝘂𝗿𝗰𝗲 bimanual UMI we made that takes you from handheld demonstrations all the way to a trained robot policy.
It's built to be easy to use. Each gripper has its own camera, LiDAR, compute, and battery, so there are no base stations to mount and nothing to calibrate in the scene. You press one button and both hands start recording.
While you keep collecting, our pipeline processes finished episodes in parallel and turns them into LeRobot datasets, automatically uploaded on Huggingface. VLAs fine-tuned mostly on this data succeed in 𝟭𝟬/𝟭𝟬 (π0.5) and 𝟴/𝟭𝟬 (MolmoAct2) trials on a real bimanual task.
We're open-sourcing everything, from the hardware designs and build guides to the code and ~40 hours of data 🧵
agree with the third point most. what production traffic really gives you isnt realism it's the distribution: which failures actually happen and how often
"real envs > synthetic envs" feels like mostly 3 things:
- a complaint about sloppy synthetic data pipelines
- a way to make envs pricier without making them better
- synthetic task authors guessing the task distribution because they've never worked with real production traffic at scale
real envs ship very hackable tasks too. whereas a carefully built synthetic env that's realistic enough has all the same benefits as real envs, + you get to aim at one failure mode on purpose and isolate it from the random noise
3
217
if synthetic data cuts expert time 50-60% the domains "waiting on cheaper data" unlock. speech should be one of them.
the blocker is the grader. a rubric line like "sounded natural" needs a human listening. automate that and speech rl envs open up xitter.gq/businessbarista/status…
Micro1 is a $4B company run by a 25-year-old, and it sits in the middle of the fastest-growing corner of AI: the data business.
I sat down with @aliansarinik, micro1's CEO, to break down how AI models actually get smarter and who gets paid to make it happen.
Complete list of notes (and link to interview below)...
Big takeaways:
- Data labeling is a $30–50B/year market. Ali thinks it hits ~$1 trillion a year within 2–4 years, comparable to compute spend.
- The internet was a one-time gift. Pre-training ran on decades of human-created data. Now that it’s used up, every frontier gain requires net-new data created by human experts.
- micro1 doesn’t sell data. It sells “units of intelligence improvement,” meaning measurable gains in a customer’s model or agent.
- Synthetic data isn’t the bear case. Ali calls it the most bullish case for his business.
How micro1 got here:
- It started as a dev agency Ali ran at Berkeley. His bottleneck was interviewing engineers, so he built an AI screener.
- That screener became one of the first AI recruiters. People hated on it, until it became normal.
- After scaling to a few million in run rate as a talent marketplace, they saw the data industry shifting toward expert data. The hard part of expert data is recruiting experts, which was exactly what they’d built.
Why spend keeps going up:
- Model progress = compute + algorithms + data. Ali argues data is what actually pushes intelligence forward.
- Post-training needs experts distilling their judgment into RL environments. It’s expensive, but the ROI is there.
- The trillions in compute bets assume models keep unlocking new capabilities. If they don’t, those investments are a disaster. So labs have no choice but to keep buying data.
How a data pipeline actually works:
- Step 1: Find the failure. Either a lab hands over a spec, or micro1 diagnoses where the model breaks and pitches the pipeline. That research arm is increasingly important.
- Step 2: Write the spec. Domain, sub-domain, data type, and volume (usually hundreds to tens of thousands of data points over 1–3 months).
- Step 3: Recruit hyper-specific experts. Never “lawyers.” More like M&A lawyers who are great at writing legal memos.
- Step 4: Turn expertise into training signal. Experts create prompts the models fail on, then write the rubrics and verifiers that define a good outcome. AI models assist them throughout.
- Step 5: Manage the pipeline like a factory. Track data quality, throughput, and whether the customer’s model is improving without regressing elsewhere.
Two ways experts grade models:
- Likert-scale evals: experts rate outputs 1–5 on dimensions like completeness or verbosity. Great for subjective, pre-release signal. Not reusable, because every new model means bringing experts back.
- Rubrics: experts write 20–40 binary criteria for a great answer (e.g. “valuation range must fall between X and Y”). More work up front, but a grader model can re-score every future model with no new human time. This is what powers actual RL training runs.
The hard parts about this business:
- Vetting at scale. micro1 interviews tens of thousands of people a day, with 10M+ through the process. Doing that manually would take thousands of recruiters.
- Managing experts. Strategic project leads run each pipeline, hold office hours, and keep experts aligned.
- Quality in subjective domains. Most fields are more subjective than people think, so they need peer review (lawyers reviewing lawyers) plus AI-based QA.
- Reward hacking. They recently caught a Gemini model searching its sandbox for hidden files containing answers. If cheating gets rewarded, models learn that cheating works and become misaligned.
The enterprise opportunity:
- Ali’s bet: Fortune 500/1000 companies converge on ~$100M/year in data spend. That’s roughly 1/10th of a lab, but there are 1000x more enterprises.
- It’s not a software budget. It looks more like payroll: core IP. Framed that way, it’s 1–5% of payroll.
- “Owning your own intelligence” means defining exactly what you want agents to be good at and building an eval stack to measure it. With that stack, models are swappable, open or closed.
- Most enterprises can’t actually use their own data. It’s customer data, so they can’t train or even evaluate on it.
Why everyone’s buying SMB data:
- An RL environment has three parts: seeding data, tasks, and tooling.
- Ali calls it “scaling on realism,” almost a new scaling law. The more realistic the environment, the better the downstream model performs.
- So micro1 licenses operational data (Slack, Google Docs, etc.) from SMBs, transforms the PII instead of redacting it to keep the noise intact, and seeds environments that act as digital twins of real workplaces.
Why synthetic data makes the pie bigger:
- Today, a lawyer writes the memo and the rubric. Soon, the lawyer writes the memo and a model generates the rubric, cutting expert time 50–60%.
- Extrapolate: the model does ~99% of the work and the expert signs off. A 15-hour task becomes 5 minutes.
- When environments are that cheap, pipelines grow 1000x. Experts become 0.1% of the work, but demand for them still goes up. Jevons paradox.
- It’s already happening in coding: 70–80% synthetic, and micro1 still can’t hire coders fast enough.
- Margins shift from expert hours to compute, which is far more controllable.
- Ali’s framing: if you could buy IQ points for your systems almost free, you’d buy all of them.
The “infinite last mile”:
- Knowledge isn’t static. Ali cites a stat that a big share of what med students learn needs relearning within a few years.
- AI accelerates that churn, which creates a feedback loop. There’s no finish line for aligning models with humans.
- Labs today concentrate spend in four domains: medicine, finance, legal, and coding. Every other domain is still waiting on cheaper data to unlock.
Which corner of the AI stack should I break down next?
Link to the full convo with Ali: youtu.be/2vL20qxD0BM
2
145
its model card: hindi (monsoon) 9.40 OIWER, hindi-english code-switched (MUCS 2021) 19.28 WER, bengali-english 35.46 different test sets so not apples to apples but code-switching is clearly where even leading open ASR still struggles.
Meet NVIDIA Nemotron 3.5 Transcribe 🎙️
An open-weight, promptable 30B MoE model with ~4B active parameters that brings transcription in 40 languages, speaker attribution, translation, contextual prompting and audio understanding into one checkpoint.
Watch it keep track of who said what when everyone’s talking at once.
1/ 🧵
all this boils down to whether you can sell units of intelligence improvement for their models and not every bits or bytes is a sellable data.
here's my current take on the data industry..
- buyers are more sophisticated and they filter out the noise right away (who's to blame them), you need to present a complete package in a proper teaser + samples, otherwise you'll not get a concrete offer.
- every MRR & ARR figure you see on X is fake, most "brokers" are not even brokers, just repackaged dropshippers.
- data brokerage is not the goal, simply the fastest path to the next phase, its shown to be the most incredible way to meet business owners, partners and labs. We'll build on that later.
- requests are more niched, from science papers to complex workflows, your avg SMMA or ecom brand is not worth anything.
- there are 3 ways of selling data, raw unstructured / anonymized, data + RL envs, data + RL envs + experts. Find ways to create value with what you have.
- it's becoming an actual tool for PE to acquire businesses and pay back parts of their debt fast.
- most businesses wants to sell directly to frontier labs, until they see the payment terms, 8/10 businesses will rather take less money upfront than 2-3x more over 3-4 months.
- the data we're selling the most is not operations data, it's CAD, defunct games and health results.
- if you think sending cold emails are going to get you businesses interested, you're so wrong, every business owners in the US has already gotten 5-10 emails from "brokers".
- labs buy outcomes, not data. a pile of data is pointless, all data is turned into training envs and in the end of the day it's about how much you can improve a model on certain tasks.
in every new industry there will be serious players and unserious players, if you're going into something for a quick buck, you will not make it, if you want to build a longterm business in the AI space, it's a great way to start.
for reference, in 2017 I was asked to run some influencer marketing campaigns on musically, bootstrapped that into the worlds nr.1 music marketing agency with 150 people in 4 countries, took 5 years but you have to start somewhere.
1
7
484

















