Anthropic released Claude Haiku 5.5 today, so we ran the 5.5 family through our coding agent token usage benchmark.
Haiku, Sonnet, and Opus each did 5 tasks, 3 times each, in Claude Code at its default medium effort.
Haiku 5.5 spends a significant amount of time on reasoning and planning, which is typical for how smaller models increase intelligence.
early results on Luna vs Jev vs Decisions API on a few internal classification tasks
tldr; they are very comparable on performance and speed
We ran the new models through our coding agent token usage benchmark. @OpenAI's GPT-6.1 Sol vs @AnthropicAI's Claude Sonnet 5.5 and Claude Opus 5.5.
For every one of them, the biggest line item was context. Claude still does more formal planning than GPT.
GPT-6.1 Sol put 58% of its cost into context and only 3% into planning. It finished all 5 tasks for $0.45.
Sonnet 5.5 and Opus 5.5 spent 40% and 38% on context and about a quarter on planning.
Opus cost $1.58 (~3.5x Sol). Sonnet cost $0.72 (~1.6x Sol).
never tell an agent to "be thorough"
we found a "be thorough" in an internal agent system prompt, removing it decreased spend by 67% with no significant change to quality 🤯
one of the biggest challenges of agent knowledge MCPs is showing the impact of it to users
if done right, memory just works and makes your agent better, but its invisible to users.
we've been experimenting with allowing agents to give feedback at the end of their sessions on how @dosu_ai impacted their trajectory: "qualitative agent feedback"
the results are early, but really interesting!
Replying to @swyx
I ran out of 500GB of disk space for this same reason












