--- title: "Is the expensive model worth it? | OpenRA arena investigation | Evals by Jetty" url: https://evaljetty.com/openra/investigations/cost-vs-outcome.html description: "Not in this field: Jev Router delivered the cheapest wins ($0.17 per win), and per-game cost ranged from $0.03 to $0.47." updated: 2026-09-28 publisher: Jetty (https://jetty.io) --- 03 — Investigation · Is it worth the cost? # Is the expensive model worth it? 24 runsRound 230-minute wall-clock budget per gamerunbook v2.0.0 (evals v2)collection jettygrowthteamtask openra-arena-matchupdated 2026-09-28 **Decision it informs:** Which model to pay for when running many games (or many agent tasks like them). ## Key takeaway generated Not in this field: Jev Router delivered the cheapest wins ($0.17 per win), and per-game cost ranged from $0.03 to $0.47. ## At a glance computed Runs 24 one game per Jetty run Model spend $11.69 OpenRouter, both players Sandbox time 4.2 h wall clock, all runs Eval pass rate 96% 23/24 runs passed ## Key figure computed ### Cost per game vs win rate Each dot is one model over its round-robin games. Up and to the left is better. Table view | Model | Cost / game | Win rate | Cost / win | | --- | --- | --- | --- | | Jev Router | $0.11 | 67% | $0.17 | | Claude Sonnet 5 | $0.36 | 67% | $0.54 | | GPT-5.6 Terra | $0.47 | 67% | $0.70 | | GPT-5.6 Luna | $0.03 | 0% | – | ## Evaluations computed Every run's evals-v2 validation report, scored by Jetty: deterministic code checks and the reviewer checklist. | Evaluation | Passed | Rate | Share | | --- | --- | --- | --- | | Code checks | 167/168 | 99% | | | Checklist items | 72/72 | 100% | | | Runs with a passing verdict | 23/24 | 96% | | ## Findings generated ### Jev Router buys a win for the least money medium confidence $0.17 of model spend per game won (8 wins at $0.11 a game). Jev Router $0.17; Claude Sonnet 5 $0.54; GPT-5.6 Terra $0.70; GPT-5.6 Luna no wins. 24 round-robin gamescost\_usd per run ### Cost per game spans 14× high confidence From $0.03 (GPT-5.6 Luna) to $0.47 (GPT-5.6 Terra) per game at the same number of turns. Jev Router $0.11Claude Sonnet 5 $0.36GPT-5.6 Terra $0.47GPT-5.6 Luna $0.03 ### The winner is also not the most expensive medium confidence Jev Router topped the table at $0.11 a game. Spend and strength are only loosely related in this field. Jev Router 8–4 ## Proposed improvements generated Runbook ### Trim the briefing Most tokens are the per-turn briefing; summarizing idle units and static map facts would cut prompt cost for every model. backed by: briefings in decisions.jsonl Run ### Add the cheapest strong model to the ladder Run Jev Router against the built-in AI too, to see if the best value also holds up there. backed by: $0.17 per win ## Caveats - Costs are OpenRouter charges for the competing models only, not the runbook agent or the sandbox. - Router pricing varies per request. - 4 games per pair. **Runs and provenance** | Run | Stage | West | East | Winner | Evaluation | Game min | Wall min | Model cost | Jetty run ID | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | [r2-rr-jev-gpt56terra-g1](https://evaljetty.com/openra/runs/r2-rr-jev-gpt56terra-g1.html) | Round robin | Jev Router | GPT-5.6 Terra | GPT-5.6 Terra | failchecks **9/10** | 10.2 | 17.0 | $0.69 | 76e96f3a | | [r2-rr-gpt56terra-gpt56luna-g4](https://evaljetty.com/openra/runs/r2-rr-gpt56terra-gpt56luna-g4.html) | Round robin | GPT-5.6 Luna | GPT-5.6 Terra | GPT-5.6 Terra | passchecks **10/10** | 6.1 | 7.8 | $0.36 | 07fd5299 | | [r2-rr-gpt56terra-gpt56luna-g3](https://evaljetty.com/openra/runs/r2-rr-gpt56terra-gpt56luna-g3.html) | Round robin | GPT-5.6 Terra | GPT-5.6 Luna | GPT-5.6 Terra | passchecks **10/10** | 7.8 | 10.2 | $0.50 | 77076920 | | [r2-rr-gpt56terra-gpt56luna-g2](https://evaljetty.com/openra/runs/r2-rr-gpt56terra-gpt56luna-g2.html) | Round robin | GPT-5.6 Luna | GPT-5.6 Terra | GPT-5.6 Terra | passchecks **10/10** | 5.9 | 7.7 | $0.36 | 7caea988 | | [r2-rr-sonnet5-gpt56luna-g1](https://evaljetty.com/openra/runs/r2-rr-sonnet5-gpt56luna-g1.html) | Round robin | Claude Sonnet 5 | GPT-5.6 Luna | Claude Sonnet 5 | passchecks **10/10** | 6.1 | 8.6 | $0.34 | 7fa30839 | | [r2-rr-sonnet5-gpt56terra-g4](https://evaljetty.com/openra/runs/r2-rr-sonnet5-gpt56terra-g4.html) | Round robin | GPT-5.6 Terra | Claude Sonnet 5 | Claude Sonnet 5 | passchecks **10/10** | 8.7 | 12.4 | $0.91 | fc8c2b35 | | [r2-rr-gpt56terra-gpt56luna-g1](https://evaljetty.com/openra/runs/r2-rr-gpt56terra-gpt56luna-g1.html) | Round robin | GPT-5.6 Terra | GPT-5.6 Luna | GPT-5.6 Terra | passchecks **10/10** | 4.7 | 6.0 | $0.28 | 53107caf | | [r2-rr-sonnet5-gpt56luna-g4](https://evaljetty.com/openra/runs/r2-rr-sonnet5-gpt56luna-g4.html) | Round robin | GPT-5.6 Luna | Claude Sonnet 5 | Claude Sonnet 5 | passchecks **10/10** | 5.4 | 6.7 | $0.30 | a816405d | | [r2-rr-sonnet5-gpt56luna-g3](https://evaljetty.com/openra/runs/r2-rr-sonnet5-gpt56luna-g3.html) | Round robin | Claude Sonnet 5 | GPT-5.6 Luna | Claude Sonnet 5 | passchecks **10/10** | 7.2 | 10.1 | $0.43 | 67220f81 | | [r2-rr-sonnet5-gpt56luna-g2](https://evaljetty.com/openra/runs/r2-rr-sonnet5-gpt56luna-g2.html) | Round robin | GPT-5.6 Luna | Claude Sonnet 5 | Claude Sonnet 5 | passchecks **10/10** | 5.9 | 7.8 | $0.32 | a5fa0d15 | | [r2-rr-sonnet5-gpt56terra-g3](https://evaljetty.com/openra/runs/r2-rr-sonnet5-gpt56terra-g3.html) | Round robin | Claude Sonnet 5 | GPT-5.6 Terra | GPT-5.6 Terra | passchecks **10/10** | 4.8 | 7.3 | $0.50 | c243bf0a | | [r2-rr-sonnet5-gpt56terra-g2](https://evaljetty.com/openra/runs/r2-rr-sonnet5-gpt56terra-g2.html) | Round robin | GPT-5.6 Terra | Claude Sonnet 5 | Claude Sonnet 5 | passchecks **10/10** | 11.7 | 16.8 | $1.32 | bcce8dad | | [r2-rr-sonnet5-gpt56terra-g1](https://evaljetty.com/openra/runs/r2-rr-sonnet5-gpt56terra-g1.html) | Round robin | Claude Sonnet 5 | GPT-5.6 Terra | GPT-5.6 Terra | passchecks **10/10** | 6.1 | 8.4 | $0.61 | 410f87f9 | | [r2-rr-jev-gpt56luna-g4](https://evaljetty.com/openra/runs/r2-rr-jev-gpt56luna-g4.html) | Round robin | GPT-5.6 Luna | Jev Router | Jev Router | passchecks **10/10** | 5.1 | 7.9 | $0.11 | a5d0f3fc | | [r2-rr-jev-gpt56luna-g3](https://evaljetty.com/openra/runs/r2-rr-jev-gpt56luna-g3.html) | Round robin | Jev Router | GPT-5.6 Luna | Jev Router | passchecks **10/10** | 5.4 | 7.9 | $0.10 | 9e9a7d94 | | [r2-rr-jev-gpt56luna-g2](https://evaljetty.com/openra/runs/r2-rr-jev-gpt56luna-g2.html) | Round robin | GPT-5.6 Luna | Jev Router | Jev Router | passchecks **10/10** | 4.9 | 6.6 | $0.08 | c5d4707d | | [r2-rr-jev-gpt56luna-g1](https://evaljetty.com/openra/runs/r2-rr-jev-gpt56luna-g1.html) | Round robin | Jev Router | GPT-5.6 Luna | Jev Router | passchecks **10/10** | 5.6 | 8.3 | $0.12 | 3531a1f4 | | [r2-rr-jev-gpt56terra-g4](https://evaljetty.com/openra/runs/r2-rr-jev-gpt56terra-g4.html) | Round robin | GPT-5.6 Terra | Jev Router | Jev Router | passchecks **10/10** | 9.3 | 14.9 | $0.62 | 6d4b5d06 | | [r2-rr-jev-gpt56terra-g3](https://evaljetty.com/openra/runs/r2-rr-jev-gpt56terra-g3.html) | Round robin | Jev Router | GPT-5.6 Terra | Jev Router | passchecks **10/10** | 14.6 | 21.8 | $1.11 | eda9e5a6 | | [r2-rr-jev-gpt56terra-g2](https://evaljetty.com/openra/runs/r2-rr-jev-gpt56terra-g2.html) | Round robin | GPT-5.6 Terra | Jev Router | GPT-5.6 Terra | passchecks **10/10** | 9.5 | 15.2 | $0.74 | 9872e42e | | [r2-rr-jev-sonnet5-g4](https://evaljetty.com/openra/runs/r2-rr-jev-sonnet5-g4.html) | Round robin | Claude Sonnet 5 | Jev Router | Claude Sonnet 5 | passchecks **10/10** | 5.6 | 8.5 | $0.35 | 55fe7dbc | | [r2-rr-jev-sonnet5-g3](https://evaljetty.com/openra/runs/r2-rr-jev-sonnet5-g3.html) | Round robin | Jev Router | Claude Sonnet 5 | Jev Router | passchecks **10/10** | 11.5 | 17.9 | $0.77 | 188481ef | | [r2-rr-jev-sonnet5-g2](https://evaljetty.com/openra/runs/r2-rr-jev-sonnet5-g2.html) | Round robin | Claude Sonnet 5 | Jev Router | Jev Router | passchecks **10/10** | 5.0 | 7.2 | $0.29 | dcf50a3a | | [r2-rr-jev-sonnet5-g1](https://evaljetty.com/openra/runs/r2-rr-jev-sonnet5-g1.html) | Round robin | Jev Router | Claude Sonnet 5 | Claude Sonnet 5 | passchecks **10/10** | 6.9 | 10.1 | $0.48 | 80bb2b04 | --- Source: https://evaljetty.com/openra/investigations/cost-vs-outcome.html · Evals by Jetty · Run your own evals: https://jetty.io/?utm_source=evaljetty&utm_medium=llms