evals
Create your eval

02 — Run · Round 1 · San Francisco 1906: earthquake · gpt56luna-sanfrancisco-s2

GPT-5.6 Luna plays San Francisco 1906

An earthquake hits San Francisco in 1906. Be a Metropolis (100,000 people) within 5 years. Ended Jan 1911: lost, city score 22.

GOAL MISSED seed 2
GPT-5.6 Luna
openai/gpt-5.6-luna

LOST: Population 48,220 (goal >= 100,000); engine verdict: lost

Population (start → end)107,540 → 48,220 Peak population107,540City score (start → end)633 → 22 Funds at the end$156Crime / pollution / traffic81 / 58 / 16 Zones R / C / I159 / 59 / 73 Power plants (coal / nuclear)8 / 0Police / fire stations8 / 10 Turns (model / failed)20 (20 / 0) Actions applied / rejected142 / 32 Mean turn latency7.2sModel cost$0.043Wall clock2.5 min Jetty run IDcd12e212

The city over time

Population

People, after every turn

Funds

Treasury, after every turn

City score

0–1000, re-evaluated each January

Table view
TurnDatePopulationFundsCity scoreCrimePollution
0Jan 1906107,540$20,0006337155
1Apr 1906103,840$18,4516717155
2Jul 1906102,400$17,47467111159
3Oct 1906103,100$16,22667111961
4Jan 1907102,980$12,24633511961
5Apr 1907101,380$11,47233511560
6Jul 1907100,240$4,93733511562
7Oct 190796,180$4,24333511562
8Jan 190894,420$016711361
9Apr 190890,760$016711060
10Jul 190888,760$016711059
11Oct 190886,100$016710659
12Jan 190984,100$1,4578910359
13Apr 190981,100$7328910360
14Jul 190976,980$4438910060
15Oct 190971,740$74899761
16Jan 191066,180$0449762
17Apr 191060,940$0449259
18Jul 191056,800$0448559
19Oct 191051,860$0448557
20Jan 191147,320$156228158

Final map

Final city map: GPT-5.6 Luna on San Francisco 1906: earthquake

Evaluation

failchecks 9/10 verdict: valid run (code checks and checklist) and task goal met (judge)

StatusKindCheckDetail
passstepenvironmentEnvironment check (snapshot, engine, key)
passsteplaunchLaunch run-city as a detached process
passstepsuperviseSupervise in the foreground until run-city exits
passsteprenderTimelapse video rendered
passstepnotesAppend Notes to summary.md
passcode_checkoutputs-existall present
passcode_checksim-reached-deadlineended Jan 1911 after 20 turns; scenario verdict False
passcode_checkdecisions-logged20 rows for 20 turns
passcode_checkllm-turns-valid0/20 model turns failed
passcode_checkvideo-decodesduration 30.0s
passcode_checklimits-respected2.5 wall-minutes (model budget 45.0 min)
failjudgescenario-goalLOST: Population 48,220 (goal >= 100,000); engine verdict: lost (score 0 / threshold 1)
passchecklistnotes-explain-outcomeNotes section cites model thoughts from turns 1, 9 and 20 and explains the treasury collapse that caused population decline
passchecklistplayer-matches-requestresult.json confirms player gpt56luna, model openai/gpt-5.6-luna, task sanfrancisco, seed 2
passchecklistno-silent-outagerun.log shows 20 turns with 0 LLM failures and no gaps; max_broke_streak 7 is a game event, not a model outage

Decision timeline

What the mayor thought and did each turn. Click a turn to jump the timelapse there.

Ready to stop guessing if your outputs are good?

Create your eval Book a 15-minute walkthrough
Connect your agent to Jetty →