A launchpad for agent benchmarks and evals

Ground your team in benchmarks you care about.

Public agent benchmarks are great for providing a general level of performance, but they don’t know your context or your task. What matters is the task that you’re building and how it performs. This website provides worked examples of how you can author, run, investigate and improve agentic workloads.

See the worked examples Create your eval

OpenRA arena · Round 2

Can decision models beat LLMs in real-time strategy?

Jev Router tops the round robin on Elo (8–4, 1552). Each model plays differently: the radar scores all of them on eight metrics.

Fork and run on Jetty Open the eval Read the investigation

How each model plays

Each metric scaled 0–100 against the best model on it.

Jev Router vs Claude Sonnet 5 · Jev Router destroyed the enemy base, destroying $29,700 of enemy value for $6,350 lost. Open the run

Pelican benchmark · generative SVG

Do coding agents grade their own work honestly?

No. Self-scores pass 107 of 108 drawings at 28/40; an external judge passes 58. The two scores correlate at only 0.25.

Fork and run on Jetty Open the eval Read the investigation

hermes · Sonnet 4.6 pelican, round 1 — First draft · round 1self 32 · judge **26.5**

hermes · Sonnet 4.6 pelican, round 8 — After self-critique · round 8self 36 · judge 18

hermes · Sonnet 4.6: the agent’s score for its own drawing rose from 32 to 36 while the external judge’s fell from 26.5 to 18. See every round

Micropolis city builder · open-source SimCity

Can an LLM plan a city for twenty years?

Mostly not: 1 of 32 runs met their goal (GPT-5.6 Terra won Tokyo 1957). Cities boom, overspend and collapse.

Fork and run on Jetty Open the eval Read the investigation

Jev Router · Grow a city · 19,660 people (bankrupt), city score 596. Open the run

Goals met by model

Grow a city for 20 years, and rescue Tokyo, San Francisco and Detroit; two seeds each. Grow pop. is the mean population after 20 years.

Mayor	Goals	Grow pop.	$ / run
GPT-5.6 Terra	1/8	3,290	$0.94
Jev Router	0/8	12,120	$1.76
GPT-5.6 Luna	0/8	7,210	$0.08
Claude Sonnet 5	0/8	1,070	$0.72

Ground your team in benchmarks you care about.

Can decision models beat LLMs in real-time strategy?

How each model plays

Do coding agents grade their own work honestly?

Can an LLM plan a city for twenty years?

Goals met by model

Ready to stop guessing if your outputs are good?