evals
Create your eval

02 — Run · Round 1 · Grow a city · sonnet5-grow-s2

Claude Sonnet 5 plays Grow a city

Empty generated map, $20,000, 20 years: reach 10,000 people without going bankrupt. Ended Jan 1920: 1,200 people, city score 431.

GOAL MISSED seed 2
Claude Sonnet 5
anthropic/claude-sonnet-5

population 1,200 (goal 10,000), city score 431

Population (start → end)0 → 1,200 Peak population14,880City score (start → end)500 → 431 Funds at the end$3,876Crime / pollution / traffic8 / 52 / 0 Zones R / C / I39 / 17 / 26 Power plants (coal / nuclear)2 / 0Police / fire stations2 / 1 Turns (model / failed)80 (80 / 0) Actions applied / rejected211 / 55 Mean turn latency10.5sModel cost$1.390Wall clock14.2 min Jetty run ID40c8cf3e

The city over time

Population

People, after every turn

Funds

Treasury, after every turn

City score

0–1000, re-evaluated each January

Table view
TurnDatePopulationFundsCity scoreCrimePollution
0Jan 19000$20,00050000
1Apr 19000$15,29571200
2Jul 1900220$13,2117122041
3Oct 1900920$12,753712444
4Jan 19011,400$11,704532444
5Apr 19012,220$11,316532350
6Jul 19013,000$10,8375321148
7Oct 19013,840$10,3275321148
8Jan 19024,560$9,1325501647
9Apr 19026,420$8,4835502247
10Jul 19027,100$4,6915502259
11Oct 19028,220$4,3685503559
12Jan 190310,100$4,5756974161
13Apr 190310,720$4,2576974165
14Jul 190311,540$3,9256975065
15Oct 190313,300$3,9146975367
16Jan 190413,820$3,3066915378
17Apr 190414,640$2,7656916079
18Jul 190414,880$2,5656916679
19Oct 190413,920$2,2636916679
20Jan 190512,040$2,6135056573
21Apr 190510,620$2,5985055773
22Jul 19059,880$2,5885055765
23Oct 19058,700$2,5885055062
24Jan 19067,500$2,7893913662
25Apr 19065,500$2,7893913661
26Jul 19065,160$2,7893912760
27Oct 19064,300$2,7893912062
28Jan 19073,940$2,8623272062
29Apr 19073,780$2,8573272066
30Jul 19073,460$2,8573271955
31Oct 19073,260$2,8573271955
32Jan 19082,900$3,0913531655
33Apr 19082,960$3,0913531549
34Jul 19082,640$3,0913531549
35Oct 19082,460$3,0913531549
36Jan 19092,460$3,2094161452
37Apr 19092,100$3,2094161449
38Jul 19092,280$3,2094161349
39Oct 19093,220$3,2094161243
40Jan 19102,840$3,3565371244
41Apr 19102,540$3,3565371344
42Jul 19102,180$3,3565371545
43Oct 19102,200$3,3565371548
44Jan 19112,140$3,4244791448
45Apr 19111,920$3,4244791248
46Jul 19111,780$3,4244791242
47Oct 19112,600$3,4244791243
48Jan 19122,520$3,5646111343
49Apr 19121,960$3,5646111349
50Jul 19121,680$3,5646111155
51Oct 19121,780$3,5646111047
52Jan 19132,560$3,5425831047
53Apr 19132,960$3,5275831441
54Jul 19132,180$3,5275831743
55Oct 19132,000$3,5275831743
56Jan 19141,840$3,6084671443
57Apr 19141,480$3,6084671441
58Jul 19141,360$3,6084671441
59Oct 19141,240$3,6084671141
60Jan 19151,260$3,6483751143
61Apr 19151,280$3,6483751141
62Jul 19151,080$3,6483751241
63Oct 19151,120$3,6483751141
64Jan 19161,120$3,7053621141
65Apr 1916960$3,7053621141
66Jul 1916980$3,7053621041
67Oct 19161,720$3,7053621041
68Jan 19174,160$3,8156811048
69Apr 19174,060$3,7836811748
70Jul 19173,300$3,7836811748
71Oct 19172,720$3,7836811447
72Jan 19181,880$3,8364501247
73Apr 19181,620$3,8364501252
74Jul 19181,680$3,8364501052
75Oct 19181,660$3,836450952
76Jan 19191,640$3,877466952
77Apr 19191,420$3,877466952
78Jul 19191,340$3,877466952
79Oct 19191,240$3,877466952
80Jan 19201,200$3,876431852

Final map

Final city map: Claude Sonnet 5 on Grow a city

Evaluation

failchecks 9/10 verdict: valid run (code checks and checklist) and task goal met (judge)

StatusKindCheckDetail
passstepenvironmentEnvironment check (snapshot, engine, key)
passsteplaunchLaunch run-city as a detached process
passstepsuperviseSupervise in the foreground until run-city exits
passsteprenderTimelapse video rendered
passstepnotesAppend Notes to summary.md
passcode_checkoutputs-existall present
passcode_checksim-reached-deadlineended Jan 1920 after 80 turns; scenario verdict None
passcode_checkdecisions-logged80 rows for 80 turns
passcode_checkllm-turns-valid0/80 model turns failed
passcode_checkvideo-decodesduration 35.6s
passcode_checklimits-respected14.2 wall-minutes (model budget 45.0 min)
failjudgecity-goalpopulation 1,200 (goal 10,000), city score 431 (score 1,200 / threshold 10,000)
passchecklistnotes-explain-outcomesummary.md has a Notes section with quotes from turns 1, 18, 30, and 80 explaining the rise-and-fall pattern
passchecklistplayer-matches-requestresult.json shows player.id=sonnet5, task=grow, seed=2 matching the runbook parameters
passchecklistno-silent-outage0 LLM failures across all 80 turns, no autopilot engaged, run.log shows continuous turn-by-turn progress

Decision timeline

What the mayor thought and did each turn. Click a turn to jump the timelapse there.

Ready to stop guessing if your outputs are good?

Create your eval Book a 15-minute walkthrough
Connect your agent to Jetty →