muse wonder full on VulcanBench pretty competitive overall with Astra, Opus 5.5, and Sol 6.1, but definitely slower which is something we’re working on ⚡️

Okay so I managed to squeeze in getting model cards out today for my Muse Wonder Full benchmark with @VulcanBench - enjoy! This, plus all the data from every run will be on the VulcanBench site soon(ish), and all of the traces will be on Github as well. Everything I do with VulcanBench is free and open source. As a truly independent benchmark (it's just me!) I share it all, every single trace from every run, plus the source code for VulcanBench so you can even re-rerun specific tests, or the entire benchmark if you want to. What stands out looking at this is that Astra is fast, but you pay for it. GPT-6.1 Sol is fast and cheap. Opus wins in accuracy, and Muse Wonder is accurate, but expensive and slow. Right now, with the data I have, I don't think I would use Muse Wonder for coding, and I'm not sure that's the market they're going after, yet. That being said, looking at the accuracy numbers, it's clear the Muse team has built a very intelligent model. If they can figure out how to wrangle the time/cost per task, they could very easily become a model usable for daily coding tasks. Live long and benchmark 🖖

Okay my Muse Wonder Full benchmark is now complete with @VulcanBench - busy weekend so might not get to put all the data and model cards together until next week. But I wanted to share a quick peek at the combined score and how it compares to other models. The TL;DR is, with Muse Wonder Full, Medium effort seems to be the sweet spot, but, you need to take tokens-per-task and cost-per-task into account when deciding what model to use. In the two tables below, first is combined accuracy score, second is time per task. Right now Muse Wonder Full is quite a bit slower than Opus 5.5, much slower than GPT-6.1 Sol, and way slower than GPT-6 Astra. Overall, based on what I'm seeing, I would use Opus 5.5 or GPT-6.1 Sol over Muse Full at this point. You'll get the same or better accuracy faster and cheaper. More to come in model cards and deeper analysis on the site, but it's the weekend and I'm hanging with family more than doing benchmarks so might be a day or two. You should do the same! 🖖

Okay, my Muse Wonder Full benchmark with @VulcanBench continues on, but slowly. I upgraded to the $200/mo plan, but still couldn't get through two full effort levels. For comparison, on the $100/mo plan I was able to benchmark Astra on every effort level 👀 That being said, I wanted to thank the team at Meta, they have been jumping in to help, and have acknowledged that there seems to be some issues at lower effort levels that they are looking into. Right now I'm out of usage for a week, but I wanted to share the results I have so far. From an accuracy standpoint, Muse Wonder does pretty well, but it's the time per task where it really runs into issues. But like I said above, this could be an overthinking issue at lower effort levels. In a week or so I should get more usage and can test High and if I still have usage after that, continue up the effort levels. That being said, it is a bit disappointing that I can benchmark any other frontier model in a single sub in a few days without hitting usage issues, and Muse Wonder could take over a month to benchmark given how quickly it runs through usage. Updating my analysis section on the VulcanBench site as well so anyone that wants to dive into the details can. It is a great honor to be able to benchmark the frontier and discover results like this. As I've said many times before, I have incredible respect for the teams building these models, this is crazy hard stuff. I do think Meta is turning a corner here, excited to continue the benchmark when I can 🖖

Insane respect for @alexandr_wang, was not expecting this. Someone from the team already reached out, excited to figure out how to get my Muse Wonder benchmark back on track 🖖

Trying to determine how much it could cost me to benchmark Muse Wonder...every other model I have benchmarked has been able to do it no problem, in a sub. Muse Code's $50/mo tier and can't get through even all the tasks in my benchmark suite at low effort. Looks like I would need a budget between $310-$770 to benchmark Muse Wonder vs. $0 for models from companies like Anthropic and OpenAI since I can run those in my subs. So...trying to think of alternate paths. I've been told that potentially Muse Code just might not be a super optimal harness, and Pi might be a better harness to use it with. Might try this before calling it. Such a bummer because I really like the team at Meta, but I have no horse in this race and have to be independent and just let the data guide me. And right now, I can't possibly understand why I would code with a model that is so much more expensive than the latest models from Anthropic and OpenAI. But, not giving up yet, just can't put $500+ into benchmarking this if I already know I wouldn't recommend it for coding with the early cost-per-task data I'm seeing. Next I'm going to compare a few tasks at low effort in Pi to see if the cost-per-task is lower than in Pi...hoping it's just the harness and not the model 🤞

Out of the gate, I am finding Muse Wonder to be a very expensive model to use. At Low Effort, I hit my weekly usage limit on my Muse Code plan in less than 12 hours. With VulcanBench, I couldn't get through an entire effort level run without hitting my limits. Compare this to even something like Astra, which did every effort level, in under 12 hours, for less than $50. That makes Muse Wonder more expensive, and a lot slower, than Astra. Now I'm not sure what to do. I'll either needs to pay a small fortune to benchmark this model given how token hungry it is, or take a month or longer benchmarking it. Put together some preliminary results comparing Muse Wonder to Opus 5.5 and GPT 6.1 Sol w/respect to token use. While it's looking like an accurate model, I care a lot about cost-per-task, as we all should, and right now, this number is looking very high.

Okay, my Haiku 5.5 benchmark with @VulcanBench is now finished, and with it, a direct comparison to GPT-6.1 Sol. Bet you weren't expecting that comparison 👀 While I'm interested in accuracy, I was also very curious about cost-per-task. I know a lot of people immediately think about comparing this to Luna, I did too, but I thought Sol would be an even more interesting comparison, and it turned out to be! Haiku 5.5 actually beat GPT-6.1 Sol in accuracy on my Frontier v4 benchmark, which I was really not expecting. At the same time, rumors of Haiku 5.5 being an insanely cheap model don't seem to pan out. GPT-6.1 Sol is significantly cheaper, and beats Haiku 5.5 and Low, and Medium effort, and they tie at High effort. As I've said many times, but always need to state when I share my results. A score between 80 - 90 is more than enough for routine daily tasks. So based on the data I have right now, I would say GPT-6.1 Sol Medium really is pretty darn optimized when it comes to both accuracy and cost for regular daily coding tasks. Haiku 5.5 is a much stronger model from an accuracy standpoint that previous versions, but it's still an expensive model when you look at cost-per-task. As always, every benchmark I publish with VulcanBench is full open source, you can view the traces, look into anything you want to dive deeper into, or even run anything you want yourself. All traces from all 115 sessions, with final scores, can be found here on Github: github.com/morganlinton/vulc… And for anyone who wants the high-level overview, this will be on the VulcanBench site soon. Oh and if you like what I'm doing with VulcanBench, when you do pop over to Github, feel free to give me a star, it's the little things that matter to independent benchmarkers like me. Live long and benchmark 🖖

Alight, making progress on @VulcanBench Frontier v5. For those who missed my earlier posts, I'm doing two different eval suite builds in parallel. One mostly driven by Fable and Opus in Claude Code, the other in Factory. Just finished Wave 1 on the Frontier v5.1 Factory build, 52 hours, damn missions are awesome. Onto wave two. This will be, without a doubt, my most well thought out eval suite I have ever build. Let's challenge the next frontier 🖖

Okay, thought I'd share some early results from my Haiku 5.5 benchmark with @VulcanBench And I figured, comparing to GPT-6 Luna was probably the best comparison. Max is running now, but so far Haiku is doing much better than Luna on accuracy. Luna wins on cost but Luna actually failed every single task in Frontier v4 at Low and Medium efforr vs. Haiku which got 56% at lower effort levels. At Extra High Haiku almost aces Frontier v4 vs. Luna which got less than 50% at the same effort level. More to come, live long and benchmark 🖖

Okay, just got the results of my Sonnet 5.5 benchmark with @VulcanBench on my Frontier v4 eval suite. Super interesting, not what I was expecting. It's amazing how many times my own benchmarks surprise me. While both Opus 5.5 and Sonnet 5.5 do score higher than GPT-6.1 Sol, it's not by much, and Sol is just sooooo much more cost effective. At the same time, looking at Opus 5.5 specifically, it is definitely ahead enough, especially and Medium and High effort levels, that I do see a strong reason to give it your hardest tasks. But for easy and medium difficulty stuff, not really sure why you wouldn't use GPT-6.1 Sol, the cost-per-task is just insanely low. I test all Claude models in Claude Code and OpenAI models in Codex, so could also be that @thsottiaux and team are working some magic with Codex to help with token efficiency, because this is pretty wild. Adding to the VulcanBench site and Github soon so everyone can dig into the data. Everything I do is independent and open source, you can look at the data from every single run, even run it all yourself if you want to. Live long and benchmark 🖖

BREAKING: Grok 4.7 takes the #1 spot on VulcanBench Frontier v4, a benchmark for challenging coding tasks. 🥇 The top three positions all belong to Grok 4.7. Its winning run scored 93.15 and passed all 23 tasks.