evals
Create your eval

02 — Run · Round 1 · San Francisco 1906: earthquake · jev-sanfrancisco-s1

Jev Router plays San Francisco 1906

An earthquake hits San Francisco in 1906. Be a Metropolis (100,000 people) within 5 years. Ended Jan 1911: lost, city score 286.

GOAL MISSED seed 1
Jev Router
typesafe/jev-router

LOST: Population 36,300 (goal >= 100,000); engine verdict: lost

Population (start → end)107,540 → 36,300 Peak population107,540City score (start → end)633 → 286 Funds at the end$0Crime / pollution / traffic69 / 49 / 7 Zones R / C / I144 / 59 / 52 Power plants (coal / nuclear)9 / 0Police / fire stations7 / 16 Turns (model / failed)20 (20 / 0) Actions applied / rejected104 / 13 Mean turn latency32.0sModel cost$0.863Wall clock10.7 min Jetty run IDb6be516e

The city over time

Population

People, after every turn

Funds

Treasury, after every turn

City score

0–1000, re-evaluated each January

Table view
TurnDatePopulationFundsCity scoreCrimePollution
0Jan 1906107,540$20,0006337158
1Apr 1906103,160$15,9576697158
2Jul 190698,420$13,43866911857
3Oct 190694,420$7,36066913359
4Jan 190791,700$033413359
5Apr 190787,780$033413158
6Jul 190782,500$033411759
7Oct 190777,180$033411759
8Jan 190872,200$3,47816711158
9Apr 190868,500$3,29916710459
10Jul 190864,280$2,77816710456
11Oct 190860,000$2,5841679856
12Jan 190957,620$8971499356
13Apr 190954,200$8481499354
14Jul 190952,000$3031498754
15Oct 190949,500$2931498152
16Jan 191045,660$5571288151
17Apr 191041,660$121287649
18Jul 191037,800$121287249
19Oct 191036,400$11287248
20Jan 191136,100$02866949

Final map

Final city map: Jev Router on San Francisco 1906: earthquake

Evaluation

failchecks 9/10 verdict: valid run (code checks and checklist) and task goal met (judge)

StatusKindCheckDetail
passstepenvironmentEnvironment check (snapshot, engine, key)
passsteplaunchLaunch run-city as a detached process
passstepsuperviseSupervise in the foreground until run-city exits
passsteprenderTimelapse video rendered
passstepnotesAppend Notes to summary.md
passcode_checkoutputs-existall present
passcode_checksim-reached-deadlineended Jan 1911 after 20 turns; scenario verdict False
passcode_checkdecisions-logged20 rows for 20 turns
passcode_checkllm-turns-valid0/20 model turns failed
passcode_checkvideo-decodesduration 30.0s
passcode_checklimits-respected10.7 wall-minutes (model budget 45.0 min)
failjudgescenario-goalLOST: Population 36,300 (goal >= 100,000); engine verdict: lost (score 0 / threshold 1)
passchecklistnotes-explain-outcomeNotes section explains population decline, cites model thoughts from turns 1, 4, and 20
passchecklistplayer-matches-requestresult.json shows player=jev task=sanfrancisco seed=1 matching the runbook parameters
passchecklistno-silent-outagerun.log shows 0 failed turns across all 20 turns with no stretch of empty or failed model responses

Decision timeline

What the mayor thought and did each turn. Click a turn to jump the timelapse there.

Ready to stop guessing if your outputs are good?

Create your eval Book a 15-minute walkthrough
Connect your agent to Jetty →