Ramit Sharma

Okay, so I have decided not to benchmark Kimi K3.1 Pro with @VulcanBench after running some initial tests. This is one of the interesting and complex things about running these benchmarks. At the end of the day, as a CTO, I care about models that me and my team would use for daily coding tasks. And this means models need to have a good balance of accuracy and cost efficiency. I started testing Kimi K3.1 Pro on low effort, with four tasks, and it solved none of them, took 26 minutes per task, and cost $2.05. For comparison, GPT-6.1 Sol low, on the same four tasks solved three out of four, took 4.5 minutes per task, and cost $0.21. This is also a really good example of why you can't just look at input/output token cost to judge the cost of a model. Kimi K3.1Pro and GPT-6.1 Sol both cost $2/M input tokens and $10/M output tokens. What input/output cost doesn't take into account is how many tokens a model uses per task, and in this case, Kimi K3.1 Pro uses ~10x the tokens, so it's actually ~10x more expensive than GPT-6.1 Sol. So I'm going to pass on benchmarking this model, it would cost me between $200 - $300 to do it, and I already know this isn't a model I'd use with my team. Live long and benchmark, thoughtfully 🖖