--- title: "Which model should command a Red Alert army? | OpenRA arena investigation | Evals by Jetty" url: https://evaljetty.com/openra/investigations/which-model.html description: "Jev Router won the round robin 8\u20134, ahead of Claude Sonnet 5 (8\u20134), GPT-5.6 Terra (8\u20134) and GPT-5.6 Luna (0\u201312). With 4 games per pair the order at the top is a signal, not a verdict." updated: 2026-09-28 publisher: Jetty (https://jetty.io) --- 03 — Investigation · Compare setups # Which model should command a Red Alert army? 24 runsRound 230-minute wall-clock budget per gamerunbook v2.0.0 (evals v2)collection jettygrowthteamtask openra-arena-matchupdated 2026-09-28 **Decision it informs:** Which model to put in charge of a real-time, fog-of-war agent task, and how much to trust the ranking. ## Key takeaway generated Jev Router won the round robin 8–4, ahead of Claude Sonnet 5 (8–4), GPT-5.6 Terra (8–4) and GPT-5.6 Luna (0–12). With 4 games per pair the order at the top is a signal, not a verdict. ## At a glance computed Runs 24 one game per Jetty run Model spend $11.69 OpenRouter, both players Sandbox time 4.2 h wall clock, all runs Eval pass rate 96% 23/24 runs passed ## Key figure computed ### Model profiles on eight metrics Each metric scaled 0–100 against the best model on it (Survival and Speed invert loss and latency). Click a name to hide it. Table view | # | Model | W–L | Elo | Destroyed/game | Lost/game | Latency | Cost/game | | --- | --- | --- | --- | --- | --- | --- | --- | | 1 | Jev Router | 8–4 | 1552 | $12,725 | $5,850 | 5.0s | $0.11 | | 2 | Claude Sonnet 5 | 8–4 | 1548 | $9,754 | $7,275 | 4.4s | $0.36 | | 3 | GPT-5.6 Terra | 8–4 | 1545 | $10,038 | $10,742 | 3.6s | $0.47 | | 4 | GPT-5.6 Luna | 0–12 | 1355 | $1,567 | $10,217 | 3.3s | $0.03 | | Row beat column | Jev Router | Claude Sonnet 5 | GPT-5.6 Terra | GPT-5.6 Luna | | --- | --- | --- | --- | --- | | Jev Router | — | 2–2 | 2–2 | 4–0 | | Claude Sonnet 5 | 2–2 | — | 2–2 | 4–0 | | GPT-5.6 Terra | 2–2 | 2–2 | — | 4–0 | | GPT-5.6 Luna | 0–4 | 0–4 | 0–4 | — | | Model | Win rate | Firepower | Survival | Economy | Army size | Decisiveness | Speed | Reliability | | --- | --- | --- | --- | --- | --- | --- | --- | --- | | Jev Router | 67 | 100 | 100 | 84 | 75 | 67 | 66 | 92 | | Claude Sonnet 5 | 67 | 77 | 80 | 100 | 100 | 67 | 73 | 100 | | GPT-5.6 Terra | 67 | 79 | 54 | 89 | 80 | 67 | 91 | 100 | | GPT-5.6 Luna | 0 | 12 | 57 | 53 | 33 | 0 | 100 | 100 | ## Evaluations computed Every run's evals-v2 validation report, scored by Jetty: deterministic code checks and the reviewer checklist. | Evaluation | Passed | Rate | Share | | --- | --- | --- | --- | | Code checks | 167/168 | 99% | | | Checklist items | 72/72 | 100% | | | Runs with a passing verdict | 23/24 | 96% | | ## Findings generated ### Jev Router is the strongest commander in this field medium confidence It finished 8–4 with Elo 1552, and did not lose a head-to-head series: 2–2 vs Claude Sonnet 5; 2–2 vs GPT-5.6 Terra; 4–0 vs GPT-5.6 Luna. 24 round-robin games4 games per pairr2-rr-jev-sonnet5-g2r2-rr-jev-sonnet5-g3r2-rr-jev-gpt56terra-g3r2-rr-jev-gpt56terra-g4 ### GPT-5.6 Luna finished last at 0–12 medium confidence It lost 6.5× as much value as it destroyed (K/D 0.15) K/D 0.15Elo 1355 ### Styles differ more than win rates suggest medium confidence Highest firepower (value destroyed per game): Jev Router. Best survival (least lost): Jev Router. Largest peak army: Claude Sonnet 5. Most decisive (wins by base destruction): Jev Router. radar profile, 8 metricsleaderboard.json ### Decision speed matters under a wall-clock budget high confidence GPT-5.6 Luna answered in 3.3s per turn on average and Jev Router in 5.0s. Because both players wait for the slower one, slow pairs play fewer game-minutes in the same 30 minutes (game length by pair: Jev Router–Claude Sonnet 5 7.3 min; GPT-5.6 Terra–Jev Router 10.9 min; GPT-5.6 Luna–Jev Router 5.3 min; GPT-5.6 Terra–Claude Sonnet 5 7.8 min; GPT-5.6 Luna–Claude Sonnet 5 6.2 min; GPT-5.6 Luna–GPT-5.6 Terra 6.1 min). mean latency 3.3s vs 5.0s ### 24 of 24 games were won by destroying the base high confidence The other 0 went to the wall-clock or tick limit and were decided on score, so the ranking rewards both winning fights outright and building value when a game is still open. all base kills ## Proposed improvements generated Run ### Play more games per pair 4 games per pair leaves close records within noise; 8–10 would separate the middle of the table. backed by: series records 2-2, 2-2, 4-0 Runbook ### Give every model the same game time, not the same wall time A game-tick budget (with a wall-clock backstop) removes the latency penalty from the ranking; keep wall-clock limits for cost control. backed by: latency 3.3s vs 5.0s Eval ### Add an LLM judge on strategy quality Score each model's reasoning (scouting, economy, army composition) from decisions.jsonl, so a loss with good play is visible. backed by: decision logs for every run ## Caveats - Small sample: 4 games per pair on one map (Singles). - Factions are random per game; Allied and Soviet armies are not symmetric. - Routers such as Jev Router choose a model per request; the model used each turn is in the decision log. - Games end on a 30-minute wall-clock budget, which favours fast models (see findings). **Runs and provenance** | Run | Stage | West | East | Winner | Evaluation | Game min | Wall min | Model cost | Jetty run ID | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | [r2-rr-jev-gpt56terra-g1](https://evaljetty.com/openra/runs/r2-rr-jev-gpt56terra-g1.html) | Round robin | Jev Router | GPT-5.6 Terra | GPT-5.6 Terra | failchecks **9/10** | 10.2 | 17.0 | $0.69 | 76e96f3a | | [r2-rr-gpt56terra-gpt56luna-g4](https://evaljetty.com/openra/runs/r2-rr-gpt56terra-gpt56luna-g4.html) | Round robin | GPT-5.6 Luna | GPT-5.6 Terra | GPT-5.6 Terra | passchecks **10/10** | 6.1 | 7.8 | $0.36 | 07fd5299 | | [r2-rr-gpt56terra-gpt56luna-g3](https://evaljetty.com/openra/runs/r2-rr-gpt56terra-gpt56luna-g3.html) | Round robin | GPT-5.6 Terra | GPT-5.6 Luna | GPT-5.6 Terra | passchecks **10/10** | 7.8 | 10.2 | $0.50 | 77076920 | | [r2-rr-gpt56terra-gpt56luna-g2](https://evaljetty.com/openra/runs/r2-rr-gpt56terra-gpt56luna-g2.html) | Round robin | GPT-5.6 Luna | GPT-5.6 Terra | GPT-5.6 Terra | passchecks **10/10** | 5.9 | 7.7 | $0.36 | 7caea988 | | [r2-rr-sonnet5-gpt56luna-g1](https://evaljetty.com/openra/runs/r2-rr-sonnet5-gpt56luna-g1.html) | Round robin | Claude Sonnet 5 | GPT-5.6 Luna | Claude Sonnet 5 | passchecks **10/10** | 6.1 | 8.6 | $0.34 | 7fa30839 | | [r2-rr-sonnet5-gpt56terra-g4](https://evaljetty.com/openra/runs/r2-rr-sonnet5-gpt56terra-g4.html) | Round robin | GPT-5.6 Terra | Claude Sonnet 5 | Claude Sonnet 5 | passchecks **10/10** | 8.7 | 12.4 | $0.91 | fc8c2b35 | | [r2-rr-gpt56terra-gpt56luna-g1](https://evaljetty.com/openra/runs/r2-rr-gpt56terra-gpt56luna-g1.html) | Round robin | GPT-5.6 Terra | GPT-5.6 Luna | GPT-5.6 Terra | passchecks **10/10** | 4.7 | 6.0 | $0.28 | 53107caf | | [r2-rr-sonnet5-gpt56luna-g4](https://evaljetty.com/openra/runs/r2-rr-sonnet5-gpt56luna-g4.html) | Round robin | GPT-5.6 Luna | Claude Sonnet 5 | Claude Sonnet 5 | passchecks **10/10** | 5.4 | 6.7 | $0.30 | a816405d | | [r2-rr-sonnet5-gpt56luna-g3](https://evaljetty.com/openra/runs/r2-rr-sonnet5-gpt56luna-g3.html) | Round robin | Claude Sonnet 5 | GPT-5.6 Luna | Claude Sonnet 5 | passchecks **10/10** | 7.2 | 10.1 | $0.43 | 67220f81 | | [r2-rr-sonnet5-gpt56luna-g2](https://evaljetty.com/openra/runs/r2-rr-sonnet5-gpt56luna-g2.html) | Round robin | GPT-5.6 Luna | Claude Sonnet 5 | Claude Sonnet 5 | passchecks **10/10** | 5.9 | 7.8 | $0.32 | a5fa0d15 | | [r2-rr-sonnet5-gpt56terra-g3](https://evaljetty.com/openra/runs/r2-rr-sonnet5-gpt56terra-g3.html) | Round robin | Claude Sonnet 5 | GPT-5.6 Terra | GPT-5.6 Terra | passchecks **10/10** | 4.8 | 7.3 | $0.50 | c243bf0a | | [r2-rr-sonnet5-gpt56terra-g2](https://evaljetty.com/openra/runs/r2-rr-sonnet5-gpt56terra-g2.html) | Round robin | GPT-5.6 Terra | Claude Sonnet 5 | Claude Sonnet 5 | passchecks **10/10** | 11.7 | 16.8 | $1.32 | bcce8dad | | [r2-rr-sonnet5-gpt56terra-g1](https://evaljetty.com/openra/runs/r2-rr-sonnet5-gpt56terra-g1.html) | Round robin | Claude Sonnet 5 | GPT-5.6 Terra | GPT-5.6 Terra | passchecks **10/10** | 6.1 | 8.4 | $0.61 | 410f87f9 | | [r2-rr-jev-gpt56luna-g4](https://evaljetty.com/openra/runs/r2-rr-jev-gpt56luna-g4.html) | Round robin | GPT-5.6 Luna | Jev Router | Jev Router | passchecks **10/10** | 5.1 | 7.9 | $0.11 | a5d0f3fc | | [r2-rr-jev-gpt56luna-g3](https://evaljetty.com/openra/runs/r2-rr-jev-gpt56luna-g3.html) | Round robin | Jev Router | GPT-5.6 Luna | Jev Router | passchecks **10/10** | 5.4 | 7.9 | $0.10 | 9e9a7d94 | | [r2-rr-jev-gpt56luna-g2](https://evaljetty.com/openra/runs/r2-rr-jev-gpt56luna-g2.html) | Round robin | GPT-5.6 Luna | Jev Router | Jev Router | passchecks **10/10** | 4.9 | 6.6 | $0.08 | c5d4707d | | [r2-rr-jev-gpt56luna-g1](https://evaljetty.com/openra/runs/r2-rr-jev-gpt56luna-g1.html) | Round robin | Jev Router | GPT-5.6 Luna | Jev Router | passchecks **10/10** | 5.6 | 8.3 | $0.12 | 3531a1f4 | | [r2-rr-jev-gpt56terra-g4](https://evaljetty.com/openra/runs/r2-rr-jev-gpt56terra-g4.html) | Round robin | GPT-5.6 Terra | Jev Router | Jev Router | passchecks **10/10** | 9.3 | 14.9 | $0.62 | 6d4b5d06 | | [r2-rr-jev-gpt56terra-g3](https://evaljetty.com/openra/runs/r2-rr-jev-gpt56terra-g3.html) | Round robin | Jev Router | GPT-5.6 Terra | Jev Router | passchecks **10/10** | 14.6 | 21.8 | $1.11 | eda9e5a6 | | [r2-rr-jev-gpt56terra-g2](https://evaljetty.com/openra/runs/r2-rr-jev-gpt56terra-g2.html) | Round robin | GPT-5.6 Terra | Jev Router | GPT-5.6 Terra | passchecks **10/10** | 9.5 | 15.2 | $0.74 | 9872e42e | | [r2-rr-jev-sonnet5-g4](https://evaljetty.com/openra/runs/r2-rr-jev-sonnet5-g4.html) | Round robin | Claude Sonnet 5 | Jev Router | Claude Sonnet 5 | passchecks **10/10** | 5.6 | 8.5 | $0.35 | 55fe7dbc | | [r2-rr-jev-sonnet5-g3](https://evaljetty.com/openra/runs/r2-rr-jev-sonnet5-g3.html) | Round robin | Jev Router | Claude Sonnet 5 | Jev Router | passchecks **10/10** | 11.5 | 17.9 | $0.77 | 188481ef | | [r2-rr-jev-sonnet5-g2](https://evaljetty.com/openra/runs/r2-rr-jev-sonnet5-g2.html) | Round robin | Claude Sonnet 5 | Jev Router | Jev Router | passchecks **10/10** | 5.0 | 7.2 | $0.29 | dcf50a3a | | [r2-rr-jev-sonnet5-g1](https://evaljetty.com/openra/runs/r2-rr-jev-sonnet5-g1.html) | Round robin | Jev Router | Claude Sonnet 5 | Claude Sonnet 5 | passchecks **10/10** | 6.9 | 10.1 | $0.48 | 80bb2b04 | --- Source: https://evaljetty.com/openra/investigations/which-model.html · Evals by Jetty · Run your own evals: https://jetty.io/?utm_source=evaljetty&utm_medium=llms