03 — Investigation · Break down by case
Which scenarios break which models?
Decision it informs: Whether a model can recover a system in crisis (a disaster, a failing city), not just grow one from scratch.
Key takeaway generated
Scenario wins: Tokyo 1957 1/8; San Francisco 1906 0/8; Detroit 1972 0/8. Best at recovery: GPT-5.6 Terra (1/6).
At a glance computed
Key figure computed
Outcome by model and task
✓ goal met, ✗ missed, one mark per seed; hover a mark for the judge's detail. The last row is an idle mayor that never acts.
| Mayor | Grow a city | Tokyo 1957 | San Francisco 1906 | Detroit 1972 | Goals met |
|---|---|---|---|---|---|
| GPT-5.6 Terra | ✗ 3.7k · ✗ 2.9k | ✓ won · ✗ lost | ✗ lost · ✗ lost | ✗ lost · ✗ lost | 1/8 |
| Jev Router | ✗ 19.7k · ✗ 4.6k | ✗ lost · ✗ lost | ✗ lost · ✗ lost | ✗ lost · ✗ lost | 0/8 |
| GPT-5.6 Luna | ✗ 8.1k · ✗ 6.4k | ✗ lost · ✗ lost | ✗ lost · ✗ lost | ✗ lost · ✗ lost | 0/8 |
| Claude Sonnet 5 | ✗ 1.2k · ✗ 0.9k | ✗ lost · ✗ lost | ✗ lost · ✗ lost | ✗ lost · ✗ lost | 0/8 |
| Idle mayor (baseline: no actions) | ✗ 0.0k · ✗ 0.0k | ✗ lost · ✗ lost | ✗ lost · ✗ lost | ✗ lost · ✗ lost | 0/8 |
Table view
| Run | Mayor | Result | Judge detail |
|---|---|---|---|
| jev-tokyo-s1 | Jev Router | lost | LOST: City score 372 (goal > 500); engine verdict: lost |
| sonnet5-tokyo-s1 | Claude Sonnet 5 | lost | LOST: City score 333 (goal > 500); engine verdict: lost |
| gpt56terra-tokyo-s1 | GPT-5.6 Terra | won | WON: City score 501 (goal > 500); engine verdict: won |
| gpt56luna-tokyo-s1 | GPT-5.6 Luna | lost | LOST: City score 330 (goal > 500); engine verdict: lost |
| jev-tokyo-s2 | Jev Router | lost | LOST: City score 368 (goal > 500); engine verdict: lost |
| sonnet5-tokyo-s2 | Claude Sonnet 5 | lost | LOST: City score 346 (goal > 500); engine verdict: lost |
| gpt56terra-tokyo-s2 | GPT-5.6 Terra | lost | LOST: City score 237 (goal > 500); engine verdict: lost |
| gpt56luna-tokyo-s2 | GPT-5.6 Luna | lost | LOST: City score 458 (goal > 500); engine verdict: lost |
| jev-sanfrancisco-s1 | Jev Router | lost | LOST: Population 36,300 (goal >= 100,000); engine verdict: lost |
| sonnet5-sanfrancisco-s1 | Claude Sonnet 5 | lost | LOST: Population 36,840 (goal >= 100,000); engine verdict: lost |
| gpt56terra-sanfrancisco-s1 | GPT-5.6 Terra | lost | LOST: Population 47,680 (goal >= 100,000); engine verdict: lost |
| gpt56luna-sanfrancisco-s1 | GPT-5.6 Luna | lost | LOST: Population 23,680 (goal >= 100,000); engine verdict: lost |
| jev-sanfrancisco-s2 | Jev Router | lost | LOST: Population 92,360 (goal >= 100,000); engine verdict: lost |
| sonnet5-sanfrancisco-s2 | Claude Sonnet 5 | lost | LOST: Population 72,300 (goal >= 100,000); engine verdict: lost |
| gpt56terra-sanfrancisco-s2 | GPT-5.6 Terra | lost | LOST: Population 38,880 (goal >= 100,000); engine verdict: lost |
| gpt56luna-sanfrancisco-s2 | GPT-5.6 Luna | lost | LOST: Population 48,220 (goal >= 100,000); engine verdict: lost |
| jev-detroit-s1 | Jev Router | lost | LOST: Average crime 22 (goal < 60), population 7,380 (floor 25,000); engine verdict: won |
| sonnet5-detroit-s1 | Claude Sonnet 5 | lost | LOST: Average crime 30 (goal < 60), population 3,520 (floor 25,000); engine verdict: won |
| gpt56terra-detroit-s1 | GPT-5.6 Terra | lost | LOST: Average crime 33 (goal < 60), population 6,120 (floor 25,000); engine verdict: won |
| gpt56luna-detroit-s1 | GPT-5.6 Luna | lost | LOST: Average crime 24 (goal < 60), population 7,400 (floor 25,000); engine verdict: won |
| jev-detroit-s2 | Jev Router | lost | LOST: Average crime 40 (goal < 60), population 10,040 (floor 25,000); engine verdict: won |
| sonnet5-detroit-s2 | Claude Sonnet 5 | lost | LOST: Average crime 28 (goal < 60), population 3,280 (floor 25,000); engine verdict: won |
| gpt56terra-detroit-s2 | GPT-5.6 Terra | lost | LOST: Average crime 56 (goal < 60), population 6,880 (floor 25,000); engine verdict: won |
| gpt56luna-detroit-s2 | GPT-5.6 Luna | lost | LOST: Average crime 30 (goal < 60), population 15,280 (floor 25,000); engine verdict: won |
Evaluations computed
Every run's evals-v2 validation report, scored by Jetty: code checks (is the run valid), the judge (was the goal met) and the reviewer checklist.
| Evaluation | Passed | Rate | Share |
|---|---|---|---|
| Code checks (run is valid) | 144/144 | 100% | |
| Checklist items | 72/72 | 100% | |
| Judge: task goal met | 1/24 | 4% | |
| Valid runs (all code checks and checklist items) | 24/24 | 100% | |
| Runs with a passing verdict (valid and goal met) | 1/24 | 4% |
Findings generated
Tokyo 1957: monster attack: 1 of 8 runs won
medium confidenceGoal: Scenario won (city score > 500 in 1962). Jev Router s1 lost (city score 372); Claude Sonnet 5 s1 lost (city score 333); GPT-5.6 Terra s1 won (city score 501); GPT-5.6 Luna s1 lost (city score 330); Jev Router s2 lost (city score 368); Claude Sonnet 5 s2 lost (city score 346); GPT-5.6 Terra s2 lost (city score 237); GPT-5.6 Luna s2 lost (city score 458).
San Francisco 1906: earthquake: 0 of 8 runs won
medium confidenceGoal: Scenario won (Metropolis, 100,000 people, in 1911). Jev Router s1 lost (population 36,300); Claude Sonnet 5 s1 lost (population 36,840); GPT-5.6 Terra s1 lost (population 47,680); GPT-5.6 Luna s1 lost (population 23,680); Jev Router s2 lost (population 92,360); Claude Sonnet 5 s2 lost (population 72,300); GPT-5.6 Terra s2 lost (population 38,880); GPT-5.6 Luna s2 lost (population 48,220).
Detroit 1972: crime: 0 of 8 runs won
medium confidenceGoal: Scenario won (crime < 60 in 1982, population >= 25,000). Jev Router s1 lost (average crime 22, pop 7,380); Claude Sonnet 5 s1 lost (average crime 30, pop 3,520); GPT-5.6 Terra s1 lost (average crime 33, pop 6,120); GPT-5.6 Luna s1 lost (average crime 24, pop 7,400); Jev Router s2 lost (average crime 40, pop 10,040); Claude Sonnet 5 s2 lost (average crime 28, pop 3,280); GPT-5.6 Terra s2 lost (average crime 56, pop 6,880); GPT-5.6 Luna s2 lost (average crime 30, pop 15,280).
Doing nothing loses every scenario
high confidenceAn idle mayor (same harness and seeds, no actions) as a reference: Detroit 1972 s1 LOST: Average crime 56 (goal < 60), population 9,320 (floor 25,000); Detroit 1972 s2 LOST: Average crime 46 (goal < 60), population 2,020 (floor 25,000); San Francisco 1906 s1 LOST: Population 32,240 (goal >= 100,000); San Francisco 1906 s2 LOST: Population 40,900 (goal >= 100,000); Tokyo 1957 s1 LOST: City score 360 (goal > 500); Tokyo 1957 s2 LOST: City score 315 (goal > 500). These cities decline on their own, so a model has to act to win.
GPT-5.6 Terra handles disasters best
medium confidenceScenario wins by model: GPT-5.6 Terra 1/6; Jev Router 0/6; GPT-5.6 Luna 0/6; Claude Sonnet 5 0/6.
Proposed improvements generated
Show the scenario rule's trend
Scenarios are judged once, at the deadline; a per-year projection of the judged metric would reward steering over reacting.
Run more seeds per scenario
One or two repeats per model and scenario cannot separate a lucky win from a skill; four to six would.
Score partial progress
A 0/1 judge hides near misses; recording the judged metric's distance to the goal would separate close losses from collapses.
Caveats
- Repeats per model: Tokyo 1957 2 (seeds 1, 2); San Francisco 1906 2 (seeds 1, 2); Detroit 1972 2 (seeds 1, 2). The seed moves where disasters land.
- Win rules are the engine's own; Detroit adds this eval's population floor (25,000).
- Scenario cities decline without intervention in this engine, so holding the line is part of the job.
Runs and provenance
| Run | Task | Seed | Mayor | Outcome | Population | City score | Evaluation | Model cost | Wall min | Jetty run ID |
|---|---|---|---|---|---|---|---|---|---|---|
| jev-tokyo-s1 | Tokyo 1957 | 1 | Jev Router | lost | 23,120 | 372 | fail | $1.21 | 14.4 | 96f24e6b |
| sonnet5-tokyo-s1 | Tokyo 1957 | 1 | Claude Sonnet 5 | lost | 23,800 | 333 | fail | $0.39 | 3.4 | 58cccc15 |
| gpt56terra-tokyo-s1 | Tokyo 1957 | 1 | GPT-5.6 Terra | won | 29,640 | 501 | pass | $0.61 | 6.0 | 00ff7869 |
| gpt56luna-tokyo-s1 | Tokyo 1957 | 1 | GPT-5.6 Luna | lost | 23,700 | 330 | fail | $0.04 | 2.5 | 4675adce |
| jev-tokyo-s2 | Tokyo 1957 | 2 | Jev Router | lost | 21,900 | 368 | fail | $0.65 | 10.1 | d78fe456 |
| sonnet5-tokyo-s2 | Tokyo 1957 | 2 | Claude Sonnet 5 | lost | 22,400 | 346 | fail | $0.38 | 3.1 | 72ed89e1 |
| gpt56terra-tokyo-s2 | Tokyo 1957 | 2 | GPT-5.6 Terra | lost | 31,020 | 237 | fail | $0.55 | 5.4 | 2cf5d9c4 |
| gpt56luna-tokyo-s2 | Tokyo 1957 | 2 | GPT-5.6 Luna | lost | 29,120 | 458 | fail | $0.04 | 2.5 | a57b0db6 |
| jev-sanfrancisco-s1 | San Francisco 1906 | 1 | Jev Router | lost | 36,300 | 286 | fail | $0.86 | 10.7 | b6be516e |
| sonnet5-sanfrancisco-s1 | San Francisco 1906 | 1 | Claude Sonnet 5 | lost | 36,840 | 23 | fail | $0.42 | 3.7 | df629564 |
| gpt56terra-sanfrancisco-s1 | San Francisco 1906 | 1 | GPT-5.6 Terra | lost | 47,680 | 39 | fail | $0.65 | 6.4 | d5eb8764 |
| gpt56luna-sanfrancisco-s1 | San Francisco 1906 | 1 | GPT-5.6 Luna | lost | 23,680 | 21 | fail | $0.05 | 2.6 | 7a99b8d9 |
| jev-sanfrancisco-s2 | San Francisco 1906 | 2 | Jev Router | lost | 92,360 | 500 | fail | $2.00 | 18.4 | d76eb040 |
| sonnet5-sanfrancisco-s2 | San Francisco 1906 | 2 | Claude Sonnet 5 | lost | 72,300 | 27 | fail | $0.45 | 4.3 | fd728217 |
| gpt56terra-sanfrancisco-s2 | San Francisco 1906 | 2 | GPT-5.6 Terra | lost | 38,880 | 37 | fail | $0.68 | 6.4 | b8e5bc93 |
| gpt56luna-sanfrancisco-s2 | San Francisco 1906 | 2 | GPT-5.6 Luna | lost | 48,220 | 22 | fail | $0.04 | 2.5 | cd12e212 |
| jev-detroit-s1 | Detroit 1972 | 1 | Jev Router | lost | 7,380 | 595 | fail | $2.21 | 26.2 | aea16d25 |
| sonnet5-detroit-s1 | Detroit 1972 | 1 | Claude Sonnet 5 | lost | 3,520 | 586 | fail | $0.73 | 6.9 | 650b3bc0 |
| gpt56terra-detroit-s1 | Detroit 1972 | 1 | GPT-5.6 Terra | lost | 6,120 | 597 | fail | $1.06 | 9.8 | 175f2d53 |
| gpt56luna-detroit-s1 | Detroit 1972 | 1 | GPT-5.6 Luna | lost | 7,400 | 547 | fail | $0.08 | 4.8 | fde1c078 |
| jev-detroit-s2 | Detroit 1972 | 2 | Jev Router | lost | 10,040 | 456 | fail | $3.11 | 45.9 | 85fe296a |
| sonnet5-detroit-s2 | Detroit 1972 | 2 | Claude Sonnet 5 | lost | 3,280 | 341 | fail | $0.72 | 6.6 | c55e7a2b |
| gpt56terra-detroit-s2 | Detroit 1972 | 2 | GPT-5.6 Terra | lost | 6,880 | 693 | fail | $1.05 | 10.8 | ef93e671 |
| gpt56luna-detroit-s2 | Detroit 1972 | 2 | GPT-5.6 Luna | lost | 15,280 | 704 | fail | $0.09 | 5.4 | aba6000b |