Qualia

Decision models used in robot data can be useful. Two things changed since our decision-model post last week. OpenAI’s Decision API got better on our robot recordings. Asked to keep or remove recording, its score went from 63% to 69% in nine days. It also catches more reports whose counts don’t add up, and finds more of the right curator notes. Not everything moved up. It caught fewer contradictions in a vision model’s report. And on fallen boxes it now flags more standing boxes too (more in the comments). And Jev from TypeSafe can look at pictures now with the update. On the same recordings its newest version scores 73%. Both answer in about half a second, ten times faster than a general vision model. Hand them the 16 frames one picture at a time and both get better still: Jev to 84%, OpenAI’s to 75%. Qualia scores 97% on the same recordings. On fallen boxes, the hardest call we check, OpenAI’s catches more falls than Jev: 72% against 55%. But it gets there by saying “fallen” much more often. When it says a box fell, it’s right 23% of the time. Jev is right 27% of the time. With either one, someone would check about three standing boxes for every real fall. In this eval, Qualia catches over 90% of the falls, and when it says a box fell, it’s right 96% of the time. Decision models are a great building block for curation. On their own, they don’t curate a dataset yet. Every chart, and how we ran it, in the updated post: qualiastudios.dev/blog/decis… Awesome to have OpenAII and TypeSafe AI pushing here.

Argon4 Robotics Data Performance metric: Awesome to have the new Gemini model released from @GoogleDeepMind. The model does well on understanding robotics actions (as google is advertising pretty hard). It's still beaten by Astra on counting the correct number of placements by a robot in a dataset. We will be releasing more results on this model in the coming days, and we are wiring it into our curation harness in certain places at @QualiaRobotics. But right now it matters where it is used, because it is beaten by Astra on the most overarching task. The place where we have found it to beat Astra is timing proximity of the robot placements to the ground truth.

Royal visit from @LeRobotHF at @QualiaRobotics today - Steven doesn’t have X because it’s too much noise - fair.

Robotics MiniFactory Hackathon is underway in Copenhagen! Demos tomorrow night🤖 Amazing talent from all over Europe. Here are some sneakpeaks @huggingface @LeRobotHF , @Atech_dev , @QualiaRobotics

Over 100 really strong applications all over the world in less than 24 hours for a 'niche' role like a robotics research engineer just goes to show how excited people are for this field of work, and how few opportunities exist to work on it...

Are reward models coming to @QualiaRobotics? Yes. Because the only real deployment grade foundation models improve over time.

One of the most respected AI evals teams in the world just started training robots on @QualiaRobotics . These are the people behind some of the most discussed benchmarks. When benchmark authors pick your platform, you're doing something right. (They also break everything, which is honestly more useful.)

@QualiaRobotics part of hosting the hackathon at @KTHAISociety 🤖

@QualiaRobotics team♥️

Thanks for delivering the robots safe n sound @JannikGrothusen Check them out at @RobotLearningCo 🤖