03 — Investigation · Compare setups
Which model should command a Red Alert army?
Decision it informs: Which model to put in charge of a real-time, fog-of-war agent task, and how much to trust the ranking.
Key takeaway generated
Jev Router won the round robin 8–3, ahead of Claude Sonnet 5 (8–4), GPT-5.6 Terra (7–4) and GPT-5.6 Luna (0–12). With 4 games per pair the order at the top is a signal, not a verdict.
At a glance computed
Key figure computed
Model profiles on eight metrics
Each metric scaled 0–100 against the best model on it (Survival and Speed invert loss and latency). Click a name to hide it.
Table view
| # | Model | W–L | Elo | Destroyed/game | Lost/game | Latency | Cost/game |
|---|---|---|---|---|---|---|---|
| 1 | Jev Router | 8–3 | 1570 | $13,291 | $4,823 | 4.9s | $0.12 |
| 2 | Claude Sonnet 5 | 8–4 | 1548 | $9,754 | $7,275 | 4.4s | $0.36 |
| 3 | GPT-5.6 Terra | 7–4 | 1527 | $9,382 | $11,118 | 3.6s | $0.46 |
| 4 | GPT-5.6 Luna | 0–12 | 1355 | $1,567 | $10,217 | 3.3s | $0.03 |
| Row beat column | Jev Router | Claude Sonnet 5 | GPT-5.6 Terra | GPT-5.6 Luna |
|---|---|---|---|---|
| Jev Router | — | 2–2 | 2–1 | 4–0 |
| Claude Sonnet 5 | 2–2 | — | 2–2 | 4–0 |
| GPT-5.6 Terra | 1–2 | 2–2 | — | 4–0 |
| GPT-5.6 Luna | 0–4 | 0–4 | 0–4 | — |
| Model | Win rate | Firepower | Survival | Economy | Army size | Decisiveness | Speed | Reliability |
|---|---|---|---|---|---|---|---|---|
| Jev Router | 73 | 100 | 100 | 85 | 78 | 73 | 67 | 99 |
| Claude Sonnet 5 | 67 | 73 | 66 | 100 | 100 | 67 | 73 | 100 |
| GPT-5.6 Terra | 64 | 71 | 43 | 87 | 80 | 64 | 92 | 100 |
| GPT-5.6 Luna | 0 | 12 | 47 | 53 | 33 | 0 | 100 | 100 |
Evaluations computed
Every run's evals-v2 validation report, scored by Jetty: deterministic code checks and the reviewer checklist.
| Evaluation | Passed | Rate | Share |
|---|---|---|---|
| Code checks | 161/161 | 100% | |
| Checklist items | 69/69 | 100% | |
| Runs with a passing verdict | 23/23 | 100% |
Findings generated
Jev Router is the strongest commander in this field
medium confidenceIt finished 8–3 with Elo 1570, and did not lose a head-to-head series: 2–2 vs Claude Sonnet 5; 2–1 vs GPT-5.6 Terra; 4–0 vs GPT-5.6 Luna.
GPT-5.6 Luna finished last at 0–12
medium confidenceIt lost 6.5× as much value as it destroyed (K/D 0.15)
Styles differ more than win rates suggest
medium confidenceHighest firepower (value destroyed per game): Jev Router. Best survival (least lost): Jev Router. Largest peak army: Claude Sonnet 5. Most decisive (wins by base destruction): Jev Router.
Decision speed matters under a wall-clock budget
high confidenceGPT-5.6 Luna answered in 3.3s per turn on average and Jev Router in 4.9s. Because both players wait for the slower one, slow pairs play fewer game-minutes in the same 30 minutes (game length by pair: Jev Router–Claude Sonnet 5 7.3 min; GPT-5.6 Terra–Jev Router 11.1 min; GPT-5.6 Luna–Jev Router 5.3 min; GPT-5.6 Terra–Claude Sonnet 5 7.8 min; GPT-5.6 Luna–Claude Sonnet 5 6.2 min; GPT-5.6 Luna–GPT-5.6 Terra 6.1 min).
23 of 23 games were won by destroying the base
high confidenceThe other 0 went to the wall-clock or tick limit and were decided on score, so the ranking rewards both winning fights outright and building value when a game is still open.
Proposed improvements generated
Play more games per pair
4 games per pair leaves close records within noise; 8–10 would separate the middle of the table.
Give every model the same game time, not the same wall time
A game-tick budget (with a wall-clock backstop) removes the latency penalty from the ranking; keep wall-clock limits for cost control.
Add an LLM judge on strategy quality
Score each model's reasoning (scouting, economy, army composition) from decisions.jsonl, so a loss with good play is visible.
Caveats
- Small sample: 4 games per pair on one map (Singles).
- Factions are random per game; Allied and Soviet armies are not symmetric.
- Routers such as Jev Router choose a model per request; the model used each turn is in the decision log.
- Games end on a 30-minute wall-clock budget, which favours fast models (see findings).
Runs and provenance
| Run | Stage | West | East | Winner | Evaluation | Game min | Wall min | Model cost | Jetty run ID |
|---|---|---|---|---|---|---|---|---|---|
| r2-rr-gpt56terra-gpt56luna-g4 | Round robin | GPT-5.6 Luna | GPT-5.6 Terra | GPT-5.6 Terra | passchecks 10/10 | 6.1 | 7.8 | $0.36 | 07fd5299 |
| r2-rr-gpt56terra-gpt56luna-g3 | Round robin | GPT-5.6 Terra | GPT-5.6 Luna | GPT-5.6 Terra | passchecks 10/10 | 7.8 | 10.2 | $0.50 | 77076920 |
| r2-rr-gpt56terra-gpt56luna-g2 | Round robin | GPT-5.6 Luna | GPT-5.6 Terra | GPT-5.6 Terra | passchecks 10/10 | 5.9 | 7.7 | $0.36 | 7caea988 |
| r2-rr-sonnet5-gpt56luna-g1 | Round robin | Claude Sonnet 5 | GPT-5.6 Luna | Claude Sonnet 5 | passchecks 10/10 | 6.1 | 8.6 | $0.34 | 7fa30839 |
| r2-rr-sonnet5-gpt56terra-g4 | Round robin | GPT-5.6 Terra | Claude Sonnet 5 | Claude Sonnet 5 | passchecks 10/10 | 8.7 | 12.4 | $0.91 | fc8c2b35 |
| r2-rr-gpt56terra-gpt56luna-g1 | Round robin | GPT-5.6 Terra | GPT-5.6 Luna | GPT-5.6 Terra | passchecks 10/10 | 4.7 | 6.0 | $0.28 | 53107caf |
| r2-rr-sonnet5-gpt56luna-g4 | Round robin | GPT-5.6 Luna | Claude Sonnet 5 | Claude Sonnet 5 | passchecks 10/10 | 5.4 | 6.7 | $0.30 | a816405d |
| r2-rr-sonnet5-gpt56luna-g3 | Round robin | Claude Sonnet 5 | GPT-5.6 Luna | Claude Sonnet 5 | passchecks 10/10 | 7.2 | 10.1 | $0.43 | 67220f81 |
| r2-rr-sonnet5-gpt56luna-g2 | Round robin | GPT-5.6 Luna | Claude Sonnet 5 | Claude Sonnet 5 | passchecks 10/10 | 5.9 | 7.8 | $0.32 | a5fa0d15 |
| r2-rr-sonnet5-gpt56terra-g3 | Round robin | Claude Sonnet 5 | GPT-5.6 Terra | GPT-5.6 Terra | passchecks 10/10 | 4.8 | 7.3 | $0.50 | c243bf0a |
| r2-rr-sonnet5-gpt56terra-g2 | Round robin | GPT-5.6 Terra | Claude Sonnet 5 | Claude Sonnet 5 | passchecks 10/10 | 11.7 | 16.8 | $1.32 | bcce8dad |
| r2-rr-sonnet5-gpt56terra-g1 | Round robin | Claude Sonnet 5 | GPT-5.6 Terra | GPT-5.6 Terra | passchecks 10/10 | 6.1 | 8.4 | $0.61 | 410f87f9 |
| r2-rr-jev-gpt56luna-g4 | Round robin | GPT-5.6 Luna | Jev Router | Jev Router | passchecks 10/10 | 5.1 | 7.9 | $0.11 | a5d0f3fc |
| r2-rr-jev-gpt56luna-g3 | Round robin | Jev Router | GPT-5.6 Luna | Jev Router | passchecks 10/10 | 5.4 | 7.9 | $0.10 | 9e9a7d94 |
| r2-rr-jev-gpt56luna-g2 | Round robin | GPT-5.6 Luna | Jev Router | Jev Router | passchecks 10/10 | 4.9 | 6.6 | $0.08 | c5d4707d |
| r2-rr-jev-gpt56luna-g1 | Round robin | Jev Router | GPT-5.6 Luna | Jev Router | passchecks 10/10 | 5.6 | 8.3 | $0.12 | 3531a1f4 |
| r2-rr-jev-gpt56terra-g4 | Round robin | GPT-5.6 Terra | Jev Router | Jev Router | passchecks 10/10 | 9.3 | 14.9 | $0.62 | 6d4b5d06 |
| r2-rr-jev-gpt56terra-g3 | Round robin | Jev Router | GPT-5.6 Terra | Jev Router | passchecks 10/10 | 14.6 | 21.8 | $1.11 | eda9e5a6 |
| r2-rr-jev-gpt56terra-g2 | Round robin | GPT-5.6 Terra | Jev Router | GPT-5.6 Terra | passchecks 10/10 | 9.5 | 15.2 | $0.74 | 9872e42e |
| r2-rr-jev-sonnet5-g4 | Round robin | Claude Sonnet 5 | Jev Router | Claude Sonnet 5 | passchecks 10/10 | 5.6 | 8.5 | $0.35 | 55fe7dbc |
| r2-rr-jev-sonnet5-g3 | Round robin | Jev Router | Claude Sonnet 5 | Jev Router | passchecks 10/10 | 11.5 | 17.9 | $0.77 | 188481ef |
| r2-rr-jev-sonnet5-g2 | Round robin | Claude Sonnet 5 | Jev Router | Jev Router | passchecks 10/10 | 5.0 | 7.2 | $0.29 | dcf50a3a |
| r2-rr-jev-sonnet5-g1 | Round robin | Jev Router | Claude Sonnet 5 | Claude Sonnet 5 | passchecks 10/10 | 6.9 | 10.1 | $0.48 | 80bb2b04 |