Fitz 🔳

Hi Friends! This time, instead of comparing different AI models, I compared the delivery performance of different AI Harness tools running the same model. We evaluated Pi, OpenCode, Claude Code, and Cursor using the same development prompt in the Port Crane Chess scenario. All tools used Sonnet 5.5 via OpenRouter, configured with medium effort. Task Overview: We asked the tools to develop a port simulation featuring autonomous crane control, physics-based cargo handling, dynamic task allocation, collision avoidance, and automatic task reassignment when a crane becomes unavailable. The evaluation focused on how quickly and cost-effectively each harness completed the development task, how easily the resulting software could be tested and inspected, and how effectively it reported its results. AI Harness Delivery Performance Scores /10 — Cost & Completion Time: 🥇 Pi — 8.46/10 — $2.634 — 17 minutes 🥈 Claude Code — 8.16/10 — $4.390 — 13 minutes 🥉 OpenCode — 7.68/10 — $2.370 — 25 minutes 🏅 Cursor — 7.24/10 — $3.780 — 19 minutes Key Findings: — Best balance between speed and cost — Pi. Ranking second in both metrics, Pi achieved the highest overall score. — Fastest completion — Claude Code. It completed the task in just 13 minutes, but also had the highest execution cost. — Lowest cost — OpenCode. However, it also recorded the longest completion time at 25 minutes. — Explicit testing and state inspection interfaces — Claude Code and Cursor. Both delivered implementations with named access points for externally stepping through execution and inspecting the current simulation state. — Restart behavior issue — OpenCode. Its implementation failed to clear the previous benchmark results after restarting. The cost–time trade-off was particularly interesting: — Pi completed the task 8 minutes faster than OpenCode for just $0.264 more. Compared with Cursor, Pi was 2 minutes faster and $1.146 cheaper. — These results demonstrate that the lowest execution cost does not necessarily translate into the highest overall efficiency. More importantly, this benchmark highlights a key observation: Using the same model, the same development prompt, and the same effort configuration does not guarantee identical completion times, execution costs, or output characteristics. — In this particular run, Pi delivered the most balanced performance, Claude Code was the fastest, and OpenCode was the most cost-efficient in absolute spending. Although Cursor's explicit testing interfaces were a positive aspect, it still fell behind Pi in both completion time and cost. Ultimately, the harness itself matters. Even when the underlying model remains identical, orchestration, execution workflows, and delivery behavior can significantly influence the final outcome.