Hi everyone, this time I compared the outputs developed by Gemini 4, GPT-6.1 Sol, DeepSeek V4.1 Flash 1017, and Qwen3.8-120B in a highly challenging RAILWAY INTERLOCKING — Autonomous Railway Benchmark scenario. 🎉 I used Pi Harness for all models and carried out the development task using the single-shot prompt method. I evaluated the models not only based on their visual outputs but also on simulation accuracy, traffic management, interlocking safety, physical braking, deterministic stability, independent validation, code quality, and cost. Task Summary: I asked the models to develop a fully autonomous railway simulation featuring 10 autonomous trains, dynamic route planning, concurrent traffic management, atomic track reservations, point-locking mechanisms, signalling failures, collision prevention, and physics-based emergency braking. The simulation had to run for 75 seconds across 5 phases, executing 4,500 physics ticks, with trains moving according to real kinematic calculations, track reservations managed safely, and all results evaluated out of 100 in a way that supported independent validation. In particular, the requirement that any safety violation would reduce the official benchmark score to zero meant that the models had to develop not just working software, but an engineering system whose safety could be verified. Overall Scores — Independent Engineering, Coding Quality /100, and Costs: 🥇 GPT-6.1 Sol — 89/100 — $3.233 🥈 DeepSeek V4.1 Flash 1017 — 75/100 — $0.270 🥉 Qwen3.8-120B — 52/100 — $1.028 Gemini 4 — 45/100 — $4.177 I calculated these scores using a weighted engineering evaluation based on safety and specification compliance (35%), operational efficiency (20%), multi-seed task completion stability (20%), and code, validator, and observability quality (25%). Simulation Performance: - Qwen3.8-120B — 10/10 tasks | Last completion at 66.58 seconds | 5.55 seconds average queue wait - Gemini 4 — 10/10 tasks | Last completion at 72.67 seconds | 8.60 seconds average queue wait - GPT-6.1 Sol — 10/10 tasks | Last completion at 73.27 seconds | 9.39 seconds average queue wait - DeepSeek V4.1 Flash — 10/10 tasks | Last completion at 73.57 seconds | 6.71 seconds average queue wait Results: 🥇 Best overall engineering and validation quality — GPT-6.1 Sol. — Best price/performance ratio — DeepSeek V4.1 Flash 1017. — Best dynamic traffic and queue management — Qwen3.8-120B. — Fastest completion of all tasks — Qwen3.8-120B (66.58 seconds). — Lowest average and maximum queue wait times — Qwen3.8-120B (5.55 / 17.18 seconds). — Best independent event-history and safety validation — GPT-6.1 Sol. — Lowest development cost — DeepSeek V4.1 Flash ($0.270). — Best emergency braking performance — DeepSeek V4.1 Flash (1.00 second, 3.00-meter safety margin). — Highest multi-seed task completion stability — GPT-6.1 Sol, Qwen, and Gemini (10/10 tasks completed in all 6/6 runs). — Most critical safety violations — Gemini 4 and Qwen3.8-120B. — Weakest cost-to-engineering-quality balance despite the highest cost — Gemini 4. Evaluation of the Results: I observed 30 premature resource releases in Gemini 4 and 20 premature point-lock releases in Qwen3.8-120B. Although these incidents did not result in actual train collisions, they violated the benchmark's mandatory 3-meter safety margin. Therefore, when the strict safety rule in the specification is applied, the official benchmark scores of Gemini and Qwen drop to 0/100 in the default test. GPT-6.1 Sol and DeepSeek, on the other hand, completed the examined safety checks without any violations. In terms of operational efficiency, Qwen3.8-120B stood out by a significant margin. Its ability to complete all tasks in 66.58 seconds, its low queue wait times, and its success in dynamic traffic distribution were particularly valuable. However, the shortcomings in safety validation demonstrated that high throughput alone does not constitute engineering success. GPT-6.1 Sol ranked first in engineering quality thanks to its comprehensive validator architecture, event-history revalidation, and task completion stability across different seeds. DeepSeek V4.1 Flash, at a cost of just $0.270, delivered an impressive price/performance result by completing all 10 tasks in the default test and passing the independent safety checks. Its ability to produce an output of this quality at approximately 15.5 times lower cost than Gemini and 12 times lower cost than GPT-6.1 Sol is significant. However, in the additional tests I conducted using six different seeds, it managed to complete all tasks on time in only two runs, indicating that there are still areas for improvement in operational stability.