evals
Create your eval

02 — Run · Round 1 · San Francisco 1906: earthquake · sonnet5-sanfrancisco-s2

Claude Sonnet 5 plays San Francisco 1906

An earthquake hits San Francisco in 1906. Be a Metropolis (100,000 people) within 5 years. Ended Jan 1911: lost, city score 27.

GOAL MISSED seed 2
Claude Sonnet 5
anthropic/claude-sonnet-5

LOST: Population 72,300 (goal >= 100,000); engine verdict: lost

Population (start → end)107,540 → 72,300 Peak population107,540City score (start → end)633 → 27 Funds at the end$388Crime / pollution / traffic102 / 61 / 21 Zones R / C / I163 / 59 / 76 Power plants (coal / nuclear)4 / 0Police / fire stations8 / 4 Turns (model / failed)20 (20 / 0) Actions applied / rejected195 / 23 Mean turn latency12.8sModel cost$0.451Wall clock4.3 min Jetty run IDfd728217

The city over time

Population

People, after every turn

Funds

Treasury, after every turn

City score

0–1000, re-evaluated each January

Table view
TurnDatePopulationFundsCity scoreCrimePollution
0Jan 1906107,540$20,0006337157
1Apr 1906105,020$19,4396697157
2Jul 1906101,180$18,79166911461
3Oct 190698,920$18,76066912560
4Jan 190796,960$17,36433412560
5Apr 190795,940$16,03633411559
6Jul 190794,440$14,70133411257
7Oct 190793,060$12,64033411257
8Jan 190893,280$9,63619610857
9Apr 190891,900$8,30419610760
10Jul 190893,940$6,67519610761
11Oct 190892,720$5,52019610761
12Jan 190990,740$2,70110910560
13Apr 190990,160$1,98610910561
14Jul 190989,860$1,95910910561
15Oct 190987,100$1,91910910862
16Jan 191085,420$1,2835410858
17Apr 191083,140$1,2355410658
18Jul 191079,220$1,1325410458
19Oct 191075,020$1,1075410462
20Jan 191171,680$3882710261

Final map

Final city map: Claude Sonnet 5 on San Francisco 1906: earthquake

Evaluation

failchecks 9/10 verdict: valid run (code checks and checklist) and task goal met (judge)

StatusKindCheckDetail
passstepenvironmentEnvironment check (snapshot, engine, key)
passsteplaunchLaunch run-city as a detached process
passstepsuperviseSupervise in the foreground until run-city exits
passsteprenderTimelapse video rendered
passstepnotesAppend Notes to summary.md
passcode_checkoutputs-existall present
passcode_checksim-reached-deadlineended Jan 1911 after 20 turns; scenario verdict False
passcode_checkdecisions-logged20 rows for 20 turns
passcode_checkllm-turns-valid0/20 model turns failed
passcode_checkvideo-decodesduration 30.0s
passcode_checklimits-respected4.3 wall-minutes (model budget 45.0 min)
failjudgescenario-goalLOST: Population 72,300 (goal >= 100,000); engine verdict: lost (score 0 / threshold 1)
passchecklistnotes-explain-outcomesummary.md Notes section cites model thoughts from turns 1, 9, and 20 and explains the fire-driven population decline
passchecklistplayer-matches-requestresult.json shows player=sonnet5 model=anthropic/claude-sonnet-5 task=sanfrancisco seed=2
passchecklistno-silent-outagerun.log shows 20 consecutive successful turns with 0 LLM failures and no gaps

Decision timeline

What the mayor thought and did each turn. Click a turn to jump the timelapse there.

Ready to stop guessing if your outputs are good?

Create your eval Book a 15-minute walkthrough
Connect your agent to Jetty →