evals
Create your eval

02 — Run · Round 1 · San Francisco 1906: earthquake · jev-sanfrancisco-s2

Jev Router plays San Francisco 1906

An earthquake hits San Francisco in 1906. Be a Metropolis (100,000 people) within 5 years. Ended Jan 1911: lost, city score 500.

GOAL MISSED seed 2
Jev Router
typesafe/jev-router

LOST: Population 92,360 (goal >= 100,000); engine verdict: lost

Population (start → end)107,540 → 92,360 Peak population107,540City score (start → end)633 → 500 Funds at the end$0Crime / pollution / traffic101 / 53 / 33 Zones R / C / I195 / 59 / 56 Power plants (coal / nuclear)8 / 0Police / fire stations8 / 14 Turns (model / failed)20 (20 / 3) Actions applied / rejected149 / 2 Mean turn latency55.1sModel cost$1.999Wall clock18.4 min Jetty run IDd76eb040

The city over time

Population

People, after every turn

Funds

Treasury, after every turn

City score

0–1000, re-evaluated each January

Table view
TurnDatePopulationFundsCity scoreCrimePollution
0Jan 1906107,540$20,0006337158
1Apr 1906103,900$16,4736677158
2Jul 1906103,360$15,88966711157
3Oct 1906102,880$14,26266711756
4Jan 1907101,940$11,60439511756
5Apr 190799,740$11,60439511356
6Jul 190798,980$11,60439511258
7Oct 190798,040$10,96739511258
8Jan 190897,060$6,91224211157
9Apr 190895,760$6,19724210955
10Jul 190895,860$6,19724210957
11Oct 190895,440$5,51024210757
12Jan 190997,600$4,73535410554
13Apr 190998,480$3,78535410554
14Jul 190999,520$2,21135410554
15Oct 190998,500$1,79435410558
16Jan 191096,140$1,05740810556
17Apr 191093,740$68240810852
18Jul 191092,200$7640810352
19Oct 191091,880$7640810354
20Jan 191193,100$050010153

Final map

Final city map: Jev Router on San Francisco 1906: earthquake

Evaluation

failchecks 9/10 verdict: valid run (code checks and checklist) and task goal met (judge)

StatusKindCheckDetail
passstepenvironmentEnvironment check (snapshot, engine, key)
passsteplaunchLaunch run-city as a detached process
passstepsuperviseSupervise in the foreground until run-city exits
passsteprenderTimelapse video rendered
passstepnotesAppend Notes to summary.md
passcode_checkoutputs-existall present
passcode_checksim-reached-deadlineended Jan 1911 after 20 turns; scenario verdict False
passcode_checkdecisions-logged20 rows for 20 turns
passcode_checkllm-turns-valid3/20 model turns failed
passcode_checkvideo-decodesduration 30.0s
passcode_checklimits-respected18.4 wall-minutes (model budget 45.0 min)
failjudgescenario-goalLOST: Population 92,360 (goal >= 100,000); engine verdict: lost (score 0 / threshold 1)
passchecklistnotes-explain-outcomesummary.md Notes section explains population decline, cites model thoughts from turns 1, 12, and 20, and attributes the loss to budget exhaustion after earthquake rebuilding
passchecklistplayer-matches-requestresult.json shows player=jev task=sanfrancisco seed=2 matching the runbook parameters
passchecklistno-silent-outagerun.log shows 3 individual FAILED turns (5, 6, 10) from provider timeouts, each isolated and within the 20% llm-turns-valid threshold; no prolonged silent stretch

Decision timeline

What the mayor thought and did each turn. Click a turn to jump the timelapse there.

Ready to stop guessing if your outputs are good?

Create your eval Book a 15-minute walkthrough
Connect your agent to Jetty →