Joe Melkonian (@joemelko) profile picture

research @ msl | tqdm is all you need

Joined December 2023
Photo by Joe Melkonian (@joemelko)Photo by Joe Melkonian (@joemelko)Photo by Joe Melkonian (@joemelko)Photo by Joe Melkonian (@joemelko)Photo by Joe Melkonian (@joemelko)Photo by Joe Melkonian (@joemelko)Photo by Joe Melkonian (@joemelko)Photo by Joe Melkonian (@joemelko)Photo by Joe Melkonian (@joemelko)Photo by Joe Melkonian (@joemelko)
Are we really running out of data??? No. We're just not using it correctly. The solution: let the model learn which data it needs to learn!!! 1/n
16
43
13
485
74,213
Joe Melkonian retweeted
🚨 MUSE FEATURE ALERT 🚨 you can now try on clothes in muse before you buy them 👕 👗 👖 muse will offer to show you how you look in something while you’re shopping for it. trying things on in person? as if....
🤖 Made with AI
487
286
218
8,252
1,512,842
this is very cool
We’re releasing all 57 intermediate checkpoints from Marin “Mochi” 535B-A23B pretraining, including 10 from long-context extension. 18.2T tokens, about 61 TB of weights. huggingface.co/open-athena/M… Blog: openathena.ai/blog/marin-moc…
4
291
For those wondering it is pronounced “work”
brought to you by UeRC: Unemployment Research Company
152
brought to you by UeRC: Unemployment Research Company
I have a major result to share. I’ve found conclusive forensic evidence that the top 4 Chinese AI labs—DeepSeek, Moonshot, MiniMax and Ant Group—have built their newest models on NVIDIA’s most advanced Blackwell chips, which are barred from export to China. This sounds insane, but I discovered that almost every AI model carries a hidden signature of the GPU that trained it. When a model is first set up, the starting values of its weights are laid out in a pattern that depends on the exact GPU. Each combination of GPU and seed number produces a unique pattern—a sort of fingerprint of the hardware used to create the model. Traces of that pattern survive training into the published model. Again, this evidence isn’t based on any insider information or leaking. It’s based on a forensic analysis of the model weights released by the companies themselves. I searched more than six trillion combinations of chip layout and seed number and matched them to open-weights models. In doing so, I found conclusive evidence that six Chinese AI models—including DeepSeek-V4-Pro, Kimi K3, and MiniMax-M3—were initialized on NVIDIA Blackwell systems. Four of the models match Grace Blackwell, the GB200 and GB300 parts in NVIDIA’s flagship liquid-cooled rack-scale systems. I then verified these results on real hardware. Running unmodified PyTorch with the seeds I’d recovered, a Blackwell B300 reproduced DeepSeek-V4-Flash’s untouched starting weights bit for bit, and a rented GB200 reproduced Kimi K3’s to a correlation of 0.999995. A Hopper H100 reproduced none of DeepSeek-V4-Flash’s, and changing the seed by one broke the match. (DeepSeek’s seed is 953049583 by the way.) The method also was verified to match 24 known open-weights models to the hardware that developers said they used. It is exceedingly unlikely that these results are a coincidence. Across six trillion combinations, the best accidental match should reach only about 8 sigma (i.e., eight standard deviations above what random chance would produce). The real matches range from 49 to 1,385 sigma, and even the weakest single test hits 20. These were data-center-scale training runs. With some analysis of release schedules and model details, I estimate that the largest, Kimi K3 and DeepSeek-V4-Pro, used 2,000 to 3,000 Blackwell GPUs each. Clusters of this size cost on the order of $150 to $200 million each to buy, and across these labs alone would add up to more than half a billion dollars of export-controlled GPUs, not counting experimental compute or otherwise. It’s quite likely that we see multi-billion dollar expenditure on export-controlled GPUs. I am highly confident in this result, and it has been independently verified by senior researchers at OpenAI and Anthropic. You can read the full report at unemploymentlabs.com/checkpo…. There is much more I’ve identified about these models through other forms of checkpoint forensics like this, and will have more to share soon.
1
1
7
529
one of the best to ever do it. crazy result
I have a major result to share. I’ve found conclusive forensic evidence that the top 4 Chinese AI labs—DeepSeek, Moonshot, MiniMax and Ant Group—have built their newest models on NVIDIA’s most advanced Blackwell chips, which are barred from export to China. This sounds insane, but I discovered that almost every AI model carries a hidden signature of the GPU that trained it. When a model is first set up, the starting values of its weights are laid out in a pattern that depends on the exact GPU. Each combination of GPU and seed number produces a unique pattern—a sort of fingerprint of the hardware used to create the model. Traces of that pattern survive training into the published model. Again, this evidence isn’t based on any insider information or leaking. It’s based on a forensic analysis of the model weights released by the companies themselves. I searched more than six trillion combinations of chip layout and seed number and matched them to open-weights models. In doing so, I found conclusive evidence that six Chinese AI models—including DeepSeek-V4-Pro, Kimi K3, and MiniMax-M3—were initialized on NVIDIA Blackwell systems. Four of the models match Grace Blackwell, the GB200 and GB300 parts in NVIDIA’s flagship liquid-cooled rack-scale systems. I then verified these results on real hardware. Running unmodified PyTorch with the seeds I’d recovered, a Blackwell B300 reproduced DeepSeek-V4-Flash’s untouched starting weights bit for bit, and a rented GB200 reproduced Kimi K3’s to a correlation of 0.999995. A Hopper H100 reproduced none of DeepSeek-V4-Flash’s, and changing the seed by one broke the match. (DeepSeek’s seed is 953049583 by the way.) The method also was verified to match 24 known open-weights models to the hardware that developers said they used. It is exceedingly unlikely that these results are a coincidence. Across six trillion combinations, the best accidental match should reach only about 8 sigma (i.e., eight standard deviations above what random chance would produce). The real matches range from 49 to 1,385 sigma, and even the weakest single test hits 20. These were data-center-scale training runs. With some analysis of release schedules and model details, I estimate that the largest, Kimi K3 and DeepSeek-V4-Pro, used 2,000 to 3,000 Blackwell GPUs each. Clusters of this size cost on the order of $150 to $200 million each to buy, and across these labs alone would add up to more than half a billion dollars of export-controlled GPUs, not counting experimental compute or otherwise. It’s quite likely that we see multi-billion dollar expenditure on export-controlled GPUs. I am highly confident in this result, and it has been independently verified by senior researchers at OpenAI and Anthropic. You can read the full report at unemploymentlabs.com/checkpo…. There is much more I’ve identified about these models through other forms of checkpoint forensics like this, and will have more to share soon.
1
2
289
Joe Melkonian retweeted
(1/10) Today, we’re sharing new research from @OpenAI: How does CoT monitorability (no-CoT capability and CoT controllability) evolve over the model training lifecycle? We emerged with unsurprising but sobering findings: as we scale toward more intelligent systems, models are becoming harder to monitor through their CoT. 🧵
125
188
77
1,472
274,717
can we stop renaming every meta-optimization problem “recursive self improvement for X”
1
1
16
716
Joe Melkonian retweeted
Muse Ember scores 35.6% on DeepFinance bench, higher than GPT-6 Luna (26.4%) for the same cost and higher than Grok 4.7 (34.1%) for 1/20th of the cost
Today we're releasing DeepFinance, the first ultra-long horizon benchmark for high finance work. A set of 21 problems that reflect work at top PE and IB firms, DeepFinance is built using real deal data. Problems are constructed by experts with backgrounds from top firms like Evercore, JPM and UBS, and institutions like HBS and Wharton. Problems represent work that would take human professionals 20-40 hours to complete. Can AI really diligence a $100M+ deal? Finance benchmarks today are limited to evaluating agents on QA on financial docs or grading the numerical correctness of simple financial models. However, real world high finance work is full of assumptions and conflicting evidence. We built DeepFinance to realistically evaluate agents on those end-to-end finance workflows. DeepFinance leaderboard: juliusresearch.ai/deep-finan…
10
5
1
57
4,321
Joe Melkonian retweeted
Exciting news: @AIatMeta's Muse Ember (Max) is the highest-performing small model in the Code Arena: WebDev with 1608 pts, outperforming two leading small models from Anthropic and OpenAI: - Claude Haiku 5.5 (High): #35, 1595 pts - GPT-6 Luna (Max): #42, 1578 pts At a 1608 score at a median cost of $0.03 per task, this puts Ember on the Pareto frontier. Neither Haiku 5.5 nor Luna makes it. Despite all three models having the same API pricing of $0.10/$0.50 per million input/output tokens, Ember's median cost per task is 2× lower than Luna ($0.06) and 3.7× lower than Haiku ($0.11). In our own testing, Ember demonstrated impressive speed and efficiency, completing 25 complex generations in 2 mins per task, at a total cost of just $1. Huge congrats to @alexandr_wang and the @AIatMeta team on this impressive release! A major step forward for small models that deliver strong real-world coding capabilities at a fraction of the cost.
Today we're introducing Muse Ember, our smallest and most efficient model, built for complex reasoning execution and everyday agent work. Muse Ember can carry workflows across apps, call tools in sequence, run tests and make targeted code fixes. With a 1M-token context window and native understanding of text, images, audio and video, it can also work across large document sets and varied inputs.
19
24
8
301
37,764
Joe Melkonian retweeted
today we released our smallest model yet: Muse Ember 🔥 Muse Ember offers nearly Muse Spark 1.3 (max) intelligence at a 17x cheaper price point (per AA) Muse Ember is the first model we’ve released trained using Muse Wonder Full excited for y’all to try on API and muse code!
177
187
58
2,940
327,891
Joe Melkonian retweeted
Our first version of Muse Ember 🔥 Ember aims to maximize intelligence per GW of inference to help bring personal superintelligence to everyone. We would love all the feedback to keep hill-climbing. Please send it our way. We will keep hill-climbing on efficiency, user experience, and better approach to train with internal frontier! More to come 🫡
1/ announcing our smallest, most efficient little model is here: muse ember 🔥 built for work that agents repeat all day, it gets you more intelligence per dollar than Haiku, Luna, and all other flash models at any reasoning setting and does it at a third of the cost or less!
22
31
5
276
25,016
Joe Melkonian retweeted
our smollest, fastest model Muse Ember is designed for repetitive agentic work! we will soon see many use cases saturate at smaller model sizes, where the intelligence-cost frontier matters.
We launched Muse Ember, our smallest and most efficient model. It is designed for the work agents repeat all day like tool calls, coding subtasks, routing, extraction and long document parsing. It delivers more intelligence per dollar than Claude Haiku or GPT Luna. Read our research blog research.meta.ai/blog/muse-e… and try it on our API.
7
3
55
1,685
Joe Melkonian retweeted
We launched Muse Ember, our smallest and most efficient model. It is designed for the work agents repeat all day like tool calls, coding subtasks, routing, extraction and long document parsing. It delivers more intelligence per dollar than Claude Haiku or GPT Luna. Read our research blog research.meta.ai/blog/muse-e… and try it on our API.
28
38
2
273
15,180
Joe Melkonian retweeted
1/ announcing our smallest, most efficient little model is here: muse ember 🔥 built for work that agents repeat all day, it gets you more intelligence per dollar than Haiku, Luna, and all other flash models at any reasoning setting and does it at a third of the cost or less!
290
340
150
6,142
399,457
Joe Melkonian retweeted
We have upgraded the underlining model behind Muse! The new model brings multiple improvements for enhancing user experiences, e.g., memory, hallucination, etc. Please try it and let us know what you think!
Upgraded the models for Muse and Meta Al, bringing Active Memory and Active Focus from Muse Wonder to everyone. In Muse, these new specialized models are 2.4x less likely to forget critical info and are 67% better at self correcting and completing everyday tasks which means less repeating yourself and fewer dead ends. More on the models: research.meta.ai/blog/specia…
11
16
168
7,956
Joe Melkonian retweeted
Introducing Voice. Talk to your Muse and Muse talks back, so you can go from “what should I cook?” to cooking, just like that ✨
186
309
181
3,526
496,255
try muse
Man I was really liking Instinct as a personal agent, but Claude Duet really is much better. Finally got off waiting list today. Onboarding and UX is so delightful. As I assumed, agent harness sitting on the Anthropic models is so incisive. Now I need to head to head Grok @bot, which has been my obsession. Which got a huge atta boy from me today for calling Apple to check on MacBook PROLED inventory (touch screen Mac book with a better camera for Zoom OMG yes 🙌🏻 )
4
255
by a landslide
Introducing LifeArena: the world’s first arena for personal assistants, as judged by real users. As AI assistants become more capable, existing static benchmarks struggle to capture the creative, unthinkable requests of people in their everyday lives. So, we offered select users the most popular personal assistants: @instinct, @Muse by @AIatMeta, Dots by @OpenAI, Grok Bot by @xai, and Gemini by @GoogleDeepMind. The results are in: @Muse is #1... by a landslide. Here's what users said:
1
11
437
amaan cares deeply about this very important problem. we need more people like amaan. very excited for you and the team sir @amaankahmad
We're excited to announce Pathfinder's $16M in funding with our Series A led by Portage with participation from Reach Capital, Gradient and Stand Together Ventures Lab If this animates and excites you, we're hiring: pathfindercard.com/join
1
3
248
Joe Melkonian retweeted
a wise man once said… follow the gradient xitter.gq/joemelko/status/195786…
1/7 New Labs Research: Follow the Gradient SafeTulu filters benign-looking data that weakens LLM safety, cutting attack success up to 22% relative while largely preserving capability. 🤗 Space: huggingface.co/spaces/Datolo… Data: huggingface.co/datasets/Dato… Paper: datologyai.com/blog/follow-t…
2
1
1
16
1,189