03 — Investigation · Is it worth the cost?
Is the expensive model worth it?
Decision it informs: Which model to pay for when running many games (or many agent tasks like them).
Key takeaway generated
Not in this field: Jev Router delivered the cheapest wins ($0.16 per win), and per-game cost ranged from $0.03 to $0.46.
At a glance computed
Key figure computed
Cost per game vs win rate
Each dot is one model over its round-robin games. Up and to the left is better.
Table view
| Model | Cost / game | Win rate | Cost / win |
|---|---|---|---|
| Jev Router | $0.12 | 73% | $0.16 |
| Claude Sonnet 5 | $0.36 | 67% | $0.54 |
| GPT-5.6 Terra | $0.46 | 64% | $0.72 |
| GPT-5.6 Luna | $0.03 | 0% | – |
Evaluations computed
Every run's evals-v2 validation report, scored by Jetty: deterministic code checks and the reviewer checklist.
| Evaluation | Passed | Rate | Share |
|---|---|---|---|
| Code checks | 161/161 | 100% | |
| Checklist items | 69/69 | 100% | |
| Runs with a passing verdict | 23/23 | 100% |
Findings generated
Jev Router buys a win for the least money
medium confidence$0.16 of model spend per game won (8 wins at $0.12 a game). Jev Router $0.16; Claude Sonnet 5 $0.54; GPT-5.6 Terra $0.72; GPT-5.6 Luna no wins.
Cost per game spans 13×
high confidenceFrom $0.03 (GPT-5.6 Luna) to $0.46 (GPT-5.6 Terra) per game at the same number of turns.
The winner is also not the most expensive
medium confidenceJev Router topped the table at $0.12 a game. Spend and strength are only loosely related in this field.
Proposed improvements generated
Trim the briefing
Most tokens are the per-turn briefing; summarizing idle units and static map facts would cut prompt cost for every model.
Add the cheapest strong model to the ladder
Run Jev Router against the built-in AI too, to see if the best value also holds up there.
Caveats
- Costs are OpenRouter charges for the competing models only, not the runbook agent or the sandbox.
- Router pricing varies per request.
- 4 games per pair.
Runs and provenance
| Run | Stage | West | East | Winner | Evaluation | Game min | Wall min | Model cost | Jetty run ID |
|---|---|---|---|---|---|---|---|---|---|
| r2-rr-gpt56terra-gpt56luna-g4 | Round robin | GPT-5.6 Luna | GPT-5.6 Terra | GPT-5.6 Terra | passchecks 10/10 | 6.1 | 7.8 | $0.36 | 07fd5299 |
| r2-rr-gpt56terra-gpt56luna-g3 | Round robin | GPT-5.6 Terra | GPT-5.6 Luna | GPT-5.6 Terra | passchecks 10/10 | 7.8 | 10.2 | $0.50 | 77076920 |
| r2-rr-gpt56terra-gpt56luna-g2 | Round robin | GPT-5.6 Luna | GPT-5.6 Terra | GPT-5.6 Terra | passchecks 10/10 | 5.9 | 7.7 | $0.36 | 7caea988 |
| r2-rr-sonnet5-gpt56luna-g1 | Round robin | Claude Sonnet 5 | GPT-5.6 Luna | Claude Sonnet 5 | passchecks 10/10 | 6.1 | 8.6 | $0.34 | 7fa30839 |
| r2-rr-sonnet5-gpt56terra-g4 | Round robin | GPT-5.6 Terra | Claude Sonnet 5 | Claude Sonnet 5 | passchecks 10/10 | 8.7 | 12.4 | $0.91 | fc8c2b35 |
| r2-rr-gpt56terra-gpt56luna-g1 | Round robin | GPT-5.6 Terra | GPT-5.6 Luna | GPT-5.6 Terra | passchecks 10/10 | 4.7 | 6.0 | $0.28 | 53107caf |
| r2-rr-sonnet5-gpt56luna-g4 | Round robin | GPT-5.6 Luna | Claude Sonnet 5 | Claude Sonnet 5 | passchecks 10/10 | 5.4 | 6.7 | $0.30 | a816405d |
| r2-rr-sonnet5-gpt56luna-g3 | Round robin | Claude Sonnet 5 | GPT-5.6 Luna | Claude Sonnet 5 | passchecks 10/10 | 7.2 | 10.1 | $0.43 | 67220f81 |
| r2-rr-sonnet5-gpt56luna-g2 | Round robin | GPT-5.6 Luna | Claude Sonnet 5 | Claude Sonnet 5 | passchecks 10/10 | 5.9 | 7.8 | $0.32 | a5fa0d15 |
| r2-rr-sonnet5-gpt56terra-g3 | Round robin | Claude Sonnet 5 | GPT-5.6 Terra | GPT-5.6 Terra | passchecks 10/10 | 4.8 | 7.3 | $0.50 | c243bf0a |
| r2-rr-sonnet5-gpt56terra-g2 | Round robin | GPT-5.6 Terra | Claude Sonnet 5 | Claude Sonnet 5 | passchecks 10/10 | 11.7 | 16.8 | $1.32 | bcce8dad |
| r2-rr-sonnet5-gpt56terra-g1 | Round robin | Claude Sonnet 5 | GPT-5.6 Terra | GPT-5.6 Terra | passchecks 10/10 | 6.1 | 8.4 | $0.61 | 410f87f9 |
| r2-rr-jev-gpt56luna-g4 | Round robin | GPT-5.6 Luna | Jev Router | Jev Router | passchecks 10/10 | 5.1 | 7.9 | $0.11 | a5d0f3fc |
| r2-rr-jev-gpt56luna-g3 | Round robin | Jev Router | GPT-5.6 Luna | Jev Router | passchecks 10/10 | 5.4 | 7.9 | $0.10 | 9e9a7d94 |
| r2-rr-jev-gpt56luna-g2 | Round robin | GPT-5.6 Luna | Jev Router | Jev Router | passchecks 10/10 | 4.9 | 6.6 | $0.08 | c5d4707d |
| r2-rr-jev-gpt56luna-g1 | Round robin | Jev Router | GPT-5.6 Luna | Jev Router | passchecks 10/10 | 5.6 | 8.3 | $0.12 | 3531a1f4 |
| r2-rr-jev-gpt56terra-g4 | Round robin | GPT-5.6 Terra | Jev Router | Jev Router | passchecks 10/10 | 9.3 | 14.9 | $0.62 | 6d4b5d06 |
| r2-rr-jev-gpt56terra-g3 | Round robin | Jev Router | GPT-5.6 Terra | Jev Router | passchecks 10/10 | 14.6 | 21.8 | $1.11 | eda9e5a6 |
| r2-rr-jev-gpt56terra-g2 | Round robin | GPT-5.6 Terra | Jev Router | GPT-5.6 Terra | passchecks 10/10 | 9.5 | 15.2 | $0.74 | 9872e42e |
| r2-rr-jev-sonnet5-g4 | Round robin | Claude Sonnet 5 | Jev Router | Claude Sonnet 5 | passchecks 10/10 | 5.6 | 8.5 | $0.35 | 55fe7dbc |
| r2-rr-jev-sonnet5-g3 | Round robin | Jev Router | Claude Sonnet 5 | Jev Router | passchecks 10/10 | 11.5 | 17.9 | $0.77 | 188481ef |
| r2-rr-jev-sonnet5-g2 | Round robin | Claude Sonnet 5 | Jev Router | Jev Router | passchecks 10/10 | 5.0 | 7.2 | $0.29 | dcf50a3a |
| r2-rr-jev-sonnet5-g1 | Round robin | Jev Router | Claude Sonnet 5 | Claude Sonnet 5 | passchecks 10/10 | 6.9 | 10.1 | $0.48 | 80bb2b04 |