evals
Create your eval

03 — Investigation · Compare setups

Which model should command a Red Alert army?

24 runsRound 230-minute wall-clock budget per gamerunbook v2.0.0 (evals v2)collection jettygrowthteamtask openra-arena-matchupdated 2026-09-28

Decision it informs: Which model to put in charge of a real-time, fog-of-war agent task, and how much to trust the ranking.

Key takeaway generated

A 3-way tie at the top: Jev Router, Claude Sonnet 5 and GPT-5.6 Terra all went 8–4, ahead of GPT-5.6 Luna (0–12). Jev Router leads on Elo (1552 vs 1548 and 1545), a margin 4 games per pair can't settle.

At a glance computed

Runs
24
one game per Jetty run
Model spend
$11.69
OpenRouter, both players
Sandbox time
4.2 h
wall clock, all runs
Eval pass rate
96%
23/24 runs passed

Key figure computed

Model profiles on eight metrics

Each metric scaled 0–100 against the best model on it (Survival and Speed invert loss and latency). Click a name to hide it.

Table view
#ModelW–LEloDestroyed/gameLost/gameLatencyCost/game
1Jev Router8–41552$12,725$5,8505.0s$0.11
2Claude Sonnet 58–41548$9,754$7,2754.4s$0.36
3GPT-5.6 Terra8–41545$10,038$10,7423.6s$0.47
4GPT-5.6 Luna0–121355$1,567$10,2173.3s$0.03
Row beat columnJev RouterClaude Sonnet 5GPT-5.6 TerraGPT-5.6 Luna
Jev Router—2–22–24–0
Claude Sonnet 52–2—2–24–0
GPT-5.6 Terra2–22–2—4–0
GPT-5.6 Luna0–40–40–4—
ModelWin rateFirepowerSurvivalEconomyArmy sizeDecisivenessSpeedReliability
Jev Router671001008475676692
Claude Sonnet 56777801001006773100
GPT-5.6 Terra67795489806791100
GPT-5.6 Luna0125753330100100

Evaluations computed

Every run's evals-v2 validation report, scored by Jetty: deterministic code checks and the reviewer checklist.

EvaluationPassedRateShare
Code checks167/16899%
Checklist items72/72100%
Runs with a passing verdict23/2496%

Findings generated

Jev Router is the strongest commander in this field

medium confidence

It finished 8–4 with Elo 1552, and did not lose a head-to-head series: 2–2 vs Claude Sonnet 5; 2–2 vs GPT-5.6 Terra; 4–0 vs GPT-5.6 Luna.

24 round-robin games4 games per pairr2-rr-jev-sonnet5-g2r2-rr-jev-sonnet5-g3r2-rr-jev-gpt56terra-g3r2-rr-jev-gpt56terra-g4

GPT-5.6 Luna finished last at 0–12

medium confidence

It lost 6.5× as much value as it destroyed (K/D 0.15)

K/D 0.15Elo 1355

Styles differ more than win rates suggest

medium confidence

Highest firepower (value destroyed per game): Jev Router. Best survival (least lost): Jev Router. Largest peak army: Claude Sonnet 5. Most decisive (wins by base destruction): Jev Router.

radar profile, 8 metricsleaderboard.json

Decision speed matters under a wall-clock budget

high confidence

GPT-5.6 Luna answered in 3.3s per turn on average and Jev Router in 5.0s. Because both players wait for the slower one, slow pairs play fewer game-minutes in the same 30 minutes (game length by pair: Jev Router–Claude Sonnet 5 7.3 min; GPT-5.6 Terra–Jev Router 10.9 min; GPT-5.6 Luna–Jev Router 5.3 min; GPT-5.6 Terra–Claude Sonnet 5 7.8 min; GPT-5.6 Luna–Claude Sonnet 5 6.2 min; GPT-5.6 Luna–GPT-5.6 Terra 6.1 min).

mean latency 3.3s vs 5.0s

24 of 24 games were won by destroying the base

high confidence

The other 0 went to the wall-clock or tick limit and were decided on score, so the ranking rewards both winning fights outright and building value when a game is still open.

all base kills

Proposed improvements generated

Run

Play more games per pair

4 games per pair leaves close records within noise; 8–10 would separate the middle of the table.

backed by: series records 2-2, 2-2, 4-0

Runbook

Give every model the same game time, not the same wall time

A game-tick budget (with a wall-clock backstop) removes the latency penalty from the ranking; keep wall-clock limits for cost control.

backed by: latency 3.3s vs 5.0s

Eval

Add an LLM judge on strategy quality

Score each model's reasoning (scouting, economy, army composition) from decisions.jsonl, so a loss with good play is visible.

backed by: decision logs for every run

Caveats

  • Small sample: 4 games per pair on one map (Singles).
  • Factions are random per game; Allied and Soviet armies are not symmetric.
  • Routers such as Jev Router choose a model per request; the model used each turn is in the decision log.
  • Games end on a 30-minute wall-clock budget, which favours fast models (see findings).
Runs and provenance
RunStageWestEastWinnerEvaluationGame minWall minModel costJetty run ID
r2-rr-jev-gpt56terra-g1Round robinJev RouterGPT-5.6 TerraGPT-5.6 Terrafailchecks 9/1010.217.0$0.6976e96f3a
r2-rr-gpt56terra-gpt56luna-g4Round robinGPT-5.6 LunaGPT-5.6 TerraGPT-5.6 Terrapasschecks 10/106.17.8$0.3607fd5299
r2-rr-gpt56terra-gpt56luna-g3Round robinGPT-5.6 TerraGPT-5.6 LunaGPT-5.6 Terrapasschecks 10/107.810.2$0.5077076920
r2-rr-gpt56terra-gpt56luna-g2Round robinGPT-5.6 LunaGPT-5.6 TerraGPT-5.6 Terrapasschecks 10/105.97.7$0.367caea988
r2-rr-sonnet5-gpt56luna-g1Round robinClaude Sonnet 5GPT-5.6 LunaClaude Sonnet 5passchecks 10/106.18.6$0.347fa30839
r2-rr-sonnet5-gpt56terra-g4Round robinGPT-5.6 TerraClaude Sonnet 5Claude Sonnet 5passchecks 10/108.712.4$0.91fc8c2b35
r2-rr-gpt56terra-gpt56luna-g1Round robinGPT-5.6 TerraGPT-5.6 LunaGPT-5.6 Terrapasschecks 10/104.76.0$0.2853107caf
r2-rr-sonnet5-gpt56luna-g4Round robinGPT-5.6 LunaClaude Sonnet 5Claude Sonnet 5passchecks 10/105.46.7$0.30a816405d
r2-rr-sonnet5-gpt56luna-g3Round robinClaude Sonnet 5GPT-5.6 LunaClaude Sonnet 5passchecks 10/107.210.1$0.4367220f81
r2-rr-sonnet5-gpt56luna-g2Round robinGPT-5.6 LunaClaude Sonnet 5Claude Sonnet 5passchecks 10/105.97.8$0.32a5fa0d15
r2-rr-sonnet5-gpt56terra-g3Round robinClaude Sonnet 5GPT-5.6 TerraGPT-5.6 Terrapasschecks 10/104.87.3$0.50c243bf0a
r2-rr-sonnet5-gpt56terra-g2Round robinGPT-5.6 TerraClaude Sonnet 5Claude Sonnet 5passchecks 10/1011.716.8$1.32bcce8dad
r2-rr-sonnet5-gpt56terra-g1Round robinClaude Sonnet 5GPT-5.6 TerraGPT-5.6 Terrapasschecks 10/106.18.4$0.61410f87f9
r2-rr-jev-gpt56luna-g4Round robinGPT-5.6 LunaJev RouterJev Routerpasschecks 10/105.17.9$0.11a5d0f3fc
r2-rr-jev-gpt56luna-g3Round robinJev RouterGPT-5.6 LunaJev Routerpasschecks 10/105.47.9$0.109e9a7d94
r2-rr-jev-gpt56luna-g2Round robinGPT-5.6 LunaJev RouterJev Routerpasschecks 10/104.96.6$0.08c5d4707d
r2-rr-jev-gpt56luna-g1Round robinJev RouterGPT-5.6 LunaJev Routerpasschecks 10/105.68.3$0.123531a1f4
r2-rr-jev-gpt56terra-g4Round robinGPT-5.6 TerraJev RouterJev Routerpasschecks 10/109.314.9$0.626d4b5d06
r2-rr-jev-gpt56terra-g3Round robinJev RouterGPT-5.6 TerraJev Routerpasschecks 10/1014.621.8$1.11eda9e5a6
r2-rr-jev-gpt56terra-g2Round robinGPT-5.6 TerraJev RouterGPT-5.6 Terrapasschecks 10/109.515.2$0.749872e42e
r2-rr-jev-sonnet5-g4Round robinClaude Sonnet 5Jev RouterClaude Sonnet 5passchecks 10/105.68.5$0.3555fe7dbc
r2-rr-jev-sonnet5-g3Round robinJev RouterClaude Sonnet 5Jev Routerpasschecks 10/1011.517.9$0.77188481ef
r2-rr-jev-sonnet5-g2Round robinClaude Sonnet 5Jev RouterJev Routerpasschecks 10/105.07.2$0.29dcf50a3a
r2-rr-jev-sonnet5-g1Round robinJev RouterClaude Sonnet 5Claude Sonnet 5passchecks 10/106.910.1$0.4880bb2b04

Ready to stop guessing if your outputs are good?

Create your eval Book a 15-minute walkthrough
Connect your agent to Jetty →