evals
Create your eval

OpenRA arena · Round 2

Can decision models beat LLMs in real-time strategy?

LLMs command Red Alert armies in real time, against each other and OpenRA's scripted AI. Every game is a Jetty run with a video and a scored evaluation.

Jev Router vs Claude Sonnet 5 · Jev Router destroyed the enemy base · Open the run

How each model plays

Eight metrics, each scaled 0–100 against the best model on it. Click a name to hide it.

Table view
ModelWin rateFirepowerSurvivalEconomyArmy sizeDecisivenessSpeedReliability
Jev Router671001008475676692
Claude Sonnet 56777801001006773100
GPT-5.6 Terra67795489806791100
GPT-5.6 Luna0125753330100100

Jev Router

8–4 · Elo 1552

Claude Sonnet 5

8–4 · Elo 1548

GPT-5.6 Terra

8–4 · Elo 1545

GPT-5.6 Luna

0–12 · Elo 1355

FAQ

Which LLM is best at Command & Conquer: Red Alert?

In the OpenRA arena round robin (updated 2026-09-28), Jev Router ranked first with a 8–4 record and Elo 1552. Full standings: Jev Router 8–4; Claude Sonnet 5 8–4; GPT-5.6 Terra 8–4; GPT-5.6 Luna 0–12. Each pair played 4 games, alternating starting positions, on the map Singles, each game capped at a 30-minute wall-clock budget.

How much does it cost for an LLM to play a game of Red Alert?

Average model spend per game through OpenRouter: Jev Router $0.11; Claude Sonnet 5 $0.36; GPT-5.6 Terra $0.47; GPT-5.6 Luna $0.03. GPT-5.6 Luna was the cheapest. A model is called once every 8 game-seconds, roughly 50 to 150 calls per game.

How does the OpenRA arena keep games fair between models?

The engine pauses every 8 game-seconds and gives both models their own fog-of-war view at the same tick, then resumes only when both have answered, so a slower model never loses game time. Every model gets the same prompt, commands, map and time limit, and starting positions alternate between games.

Can I run the OpenRA LLM arena myself?

Yes. Click Fork and run on Jetty, at the top of this page and on the Author page: it copies the runbook (openra-arena-match, on the prebuilt sandbox snapshot openra-arena) into your Jetty collection and launches one game between the players you pick, any OpenRouter model or cpu:<level> for OpenRA's AI; you add an OPENROUTER_API_KEY when asked. The runbook itself: evaljetty.com/openra/RUNBOOK.md. How it works: evaljetty.com/openra/author.html.

What data is published for each game?

Every game has a 1080p video, the OpenRA replay file, the full decision log (what each model saw, its reasoning and its commands), result and metrics JSON, and the exact runtime configuration. Aggregates: evaljetty.com/openra/data/leaderboard.json, leaderboard.csv and games.json.

Ready to stop guessing if your outputs are good?

Create your eval Book a 15-minute walkthrough
Connect your agent to Jetty →