evals
Create your eval

01 — Author

Tell Jetty
the job to be done.

One runbook plus a sandbox snapshot. Jetty scores every run against the runbook's checks.

The job

OpenRA is an open-source engine for Command & Conquer: Red Alert. Each run is one 1v1 skirmish on Singles, a two-player map, with random factions, one MCV and $5,000 each. A player wins by destroying every enemy building. Games end after a 30-minute wall-clock budget, when the side with the lower score (assets value plus value of enemy units and buildings destroyed) surrenders.

Lockstep decisions and fairness

A C# bot trait inside the engine pauses the simulation every 200 ticks (8 game-seconds), serialises what each player can see (its units, buildings, production, economy, and enemies currently visible through fog of war) and sends both views at the same tick. Both models are called in parallel and the game resumes only when both have answered, so neither sees the other's move first and model latency never costs game time. Latency does cost wall-clock time, so under the wall-clock budget a slow pair plays fewer game-minutes than a fast one.

Each turn a model gets a text briefing (economy, power, what it can build with prices, every unit and building with IDs and cells, visible enemies, the result of its previous commands, and a memory note it wrote last turn) and replies with JSON: its reasoning, a note for next turn, and commands such as build, place, train, attack_move, attack, deploy, rally, repair and sell. Invalid commands are rejected with an explanation the model sees next turn. Models run through OpenRouter with low reasoning effort where supported, up to 4000 output tokens and a 120-second timeout; a failed reply is a turn with no orders.

How a run is checked (evals v2)

Deterministic code checks and a reviewer checklist, taken from the runbook. The agent writes them into validation_report.json in Jetty's evals-v2 shape and Jetty computes the verdict: any failing check fails the run.

Code checkWhat it verifies
outputs-existEvery required output file exists and is non-empty ```bash cd {{results_dir}} && for f in result.json metrics.json decisions.jsonl replay.orarep match.mp4 thumb.jpg runtime_config.json summary.md; do test -s "$f" || { echo "missing $f"; exit 1; }; done ``` ### game-has-winner — The game finished with a winner ```bash python3 -c "import json; r=json.load(open('{{results_dir}}/result.json')); assert r.get('winner'), 'no winner'; print(r['winner'], r['end_reason'])" ``` ### decisions-logged — Every model player has decision turns in decisions.jsonl ```bash python3 -c "import json; r=json.load(open('{{results_dir}}/result.json')); s={json.loads(l)['slot'] for l in open('{{results_dir}}/decisions.jsonl')}; need={k for k,v in r['providers'].items() if v!='cpu'}; assert need<=s, need-s" ``` ### llm-turns-valid — At most 20% of each model's turns failed ```bash python3 -c "import json; r=json.load(open('{{results_dir}}/result.json')); c,f=r['llm_calls'],r['llm_failures']; bad={s:(f[s],c[s]) for s in c if f[s]>max(3,0.2*c[s])}; assert not bad, bad" ``` ### video-decodes — match.mp4 decodes ```bash test "$(ffprobe -v error -show_entries format=duration -of csv=p=0 {{results_dir}}/match.mp4 | cut -d. -f1)" -gt 0 ``` ### video-complete — The recording covers the whole game ```bash python3 -c "import json; r=json.load(open('{{results_dir}}/render.json')); assert not r.get('truncated'), r" ``` ### limits-respected — The game ended within the wall-clock budget ```bash python3 -c "import json; r=json.load(open('{{results_dir}}/result.json')); assert r['wall_seconds']/60 <= {{max_wall_minutes}}+8, r['wall_seconds']" ``` ## Checklist Confirm each item by inspection; a failed item fails the run. - [ ] `notes-explain-win`: summary.md has a Notes section explaining how the game was won, citing both models' own reasoning - [ ] `players-match-request`: the players and models recorded in result.json are the ones requested in `{{player_a}}` and `{{player_b}}` - [ ] `no-silent-outage`: run.log shows no long stretch of failed or empty model turns that the code checks did not already flag ## Write Validation Report Write `{{results_dir}}/validation_report.json` in the evals-v2 shape with `arena-report`. It runs every code check above, records one `step` entry per step you pass in, and one `checklist` entry per item you pass in (status, a colon, then a one-line reason). Jetty computes the verdict from these checks.
Checklist itemReviewer confirms
notes-explain-winsummary.md has a Notes section explaining how the game was won, citing both models' own reasoning
players-match-requestthe players and models recorded in result.json are the ones requested in `{{player_a}}` and `{{player_b}}`
no-silent-outagerun.log shows no long stretch of failed or empty model turns that the code checks did not already flag

The environment: sandbox snapshot openra-arena

4 vCPU / 8 GB. Debian bookworm with the .NET 8 runtime, OpenRA built from the OpenRA-RL fork at 9b271c1 plus the arena patch, Red Alert freeware data, Xvfb with Mesa software OpenGL (games play with the real renderer so replays stay deterministic), ffmpeg, Python and the orchestrator. Recordings re-simulate the replay in a capture mode: 1920×1080, 30 fps, 6× speed.

Players

A preset ID, openrouter:<model>|<Display name> for any OpenRouter chat model, or cpu:<level> for OpenRA's scripted AI.

SpecNameModel
jevJev Routertypesafe/jev-router
sonnet5Claude Sonnet 5anthropic/claude-sonnet-5
gpt56terraGPT-5.6 Terraopenai/gpt-5.6-terra
gpt56lunaGPT-5.6 Lunaopenai/gpt-5.6-luna
gpt6lunaGPT-6 Lunaopenai/gpt-6-luna
cpu-beginnerOpenRA AI (Beginner)openra/beginner-ai
cpu-easyOpenRA AI (Easy)openra/easy-ai
cpu-mediumOpenRA AI (Medium)openra/medium-ai
cpu-normalOpenRA AI (Normal)openra/normal-ai

The runbook

A runbook is one Markdown file that gives an agent its job, its bar for done and its checks. Jetty runs it in a sandbox and computes a verdict from the checks. This one has these parts:

  1. Objective: play one 1v1 skirmish between the two requested players and deliver a video, the replay, a decision log and a results file.
  2. Parameters: the two players, a game ID, the wall-clock budget (30 minutes) and a tick cap.
  3. Steps: check the sandbox, launch the match, supervise it in the foreground until it ends, render the video and write short notes.
  4. Code checks: deterministic commands that verify the outputs, listed above.
  5. Checklist: items the agent confirms from the logs and the video, listed above.
  6. Validation report: the agent writes validation_report.json in Jetty's evals-v2 shape and Jetty computes the pass/fail verdict from it.

Fork and run copies this runbook, unchanged, into your Jetty collection and launches its first run with the inputs you choose; Jetty asks for an OpenRouter key if your collection has none.

Improve: what changed between rounds

Every round's lessons go back into the runbook, the Improve step of the loop. The biggest changes so far: v1.1.0 made supervision foreground-only after agents ended their sessions mid-game, and v2.0.0 moved to Jetty's evals-v2 checks and a 30-minute wall-clock budget per game.

Ready to stop guessing if your outputs are good?

Create your eval Book a 15-minute walkthrough
Connect your agent to Jetty →