evals
Create your eval

02 — Run · Round 1 · San Francisco 1906: earthquake · gpt56terra-sanfrancisco-s2

GPT-5.6 Terra plays San Francisco 1906

An earthquake hits San Francisco in 1906. Be a Metropolis (100,000 people) within 5 years. Ended Jan 1911: lost, city score 37.

GOAL MISSED seed 2
GPT-5.6 Terra
openai/gpt-5.6-terra

LOST: Population 38,880 (goal >= 100,000); engine verdict: lost

Population (start → end)107,540 → 38,880 Peak population107,540City score (start → end)633 → 37 Funds at the end$0Crime / pollution / traffic78 / 47 / 9 Zones R / C / I151 / 68 / 55 Power plants (coal / nuclear)6 / 0Police / fire stations8 / 15 Turns (model / failed)20 (20 / 0) Actions applied / rejected225 / 23 Mean turn latency19.1sModel cost$0.682Wall clock6.4 min Jetty run IDb8e5bc93

The city over time

Population

People, after every turn

Funds

Treasury, after every turn

City score

0–1000, re-evaluated each January

Table view
TurnDatePopulationFundsCity scoreCrimePollution
0Jan 1906107,540$20,0006337159
1Apr 1906104,280$14,9466677159
2Jul 1906100,520$13,86566711460
3Oct 190698,100$12,54466712159
4Jan 190794,180$7,91133612159
5Apr 190793,360$5,82933612859
6Jul 190790,020$2,98933612462
7Oct 190786,220$2,05833612462
8Jan 190883,780$1,53016812060
9Apr 190883,060$41916811361
10Jul 190881,020$516811357
11Oct 190879,560$516811257
12Jan 190976,400$1,6178611056
13Apr 190973,520$9338611055
14Jul 190967,160$3068610455
15Oct 190962,900$158869851
16Jan 191056,540$2,086439849
17Apr 191052,180$1,626439348
18Jul 191046,360$1,510438548
19Oct 191042,540$1,334438546
20Jan 191138,000$0377847

Final map

Final city map: GPT-5.6 Terra on San Francisco 1906: earthquake

Evaluation

failchecks 9/10 verdict: valid run (code checks and checklist) and task goal met (judge)

StatusKindCheckDetail
passstepenvironmentEnvironment check (snapshot, engine, key)
passsteplaunchLaunch run-city as a detached process
passstepsuperviseSupervise in the foreground until run-city exits
passsteprenderTimelapse video rendered
passstepnotesAppend Notes to summary.md
passcode_checkoutputs-existall present
passcode_checksim-reached-deadlineended Jan 1911 after 20 turns; scenario verdict False
passcode_checkdecisions-logged20 rows for 20 turns
passcode_checkllm-turns-valid0/20 model turns failed
passcode_checkvideo-decodesduration 30.0s
passcode_checklimits-respected6.4 wall-minutes (model budget 45.0 min)
failjudgescenario-goalLOST: Population 38,880 (goal >= 100,000); engine verdict: lost (score 0 / threshold 1)
passchecklistnotes-explain-outcomesummary.md has a Notes section explaining the scenario loss with model thoughts from turns 1, 4, 8, and 20
passchecklistplayer-matches-requestresult.json shows player=gpt56terra, task=sanfrancisco, seed=2 matching the requested parameters
passchecklistno-silent-outagerun.log shows 0 LLM failures across all 20 turns, no gaps or repeated failures

Decision timeline

What the mayor thought and did each turn. Click a turn to jump the timelapse there.

Ready to stop guessing if your outputs are good?

Create your eval Book a 15-minute walkthrough
Connect your agent to Jetty →