03 — Investigation · Compare setups
Which model plans a city best?
Decision it informs: Which model to trust with a long-horizon planning task where early mistakes compound for years.
Key takeaway generated
Jev Router built the largest cities (mean 12,120 people after 20 years), ahead of GPT-5.6 Luna (7,210), GPT-5.6 Terra (3,290) and Claude Sonnet 5 (1,070). With two maps per model, the order is a signal, not a verdict.
At a glance computed
Key figure computed
Population in January 1920, every grow run
One bar per run, colored by model; the dashed line is the 10,000 goal. Hover for the city score.
Table view
| Run | Task | Seed | Mayor | Outcome | Population | City score | Evaluation | Model cost | Wall min | Jetty run ID |
|---|---|---|---|---|---|---|---|---|---|---|
| jev-grow-s2 | Grow a city | 2 | Jev Router | 19,660 people (bankrupt) | 19,660 | 596 | fail | $2.14 | 46.3 | 5de24fe4 |
| gpt56luna-grow-s2 | Grow a city | 2 | GPT-5.6 Luna | 8,060 people | 8,060 | 417 | fail | $0.14 | 8.4 | 831ede26 |
| gpt56luna-grow-s6 | Grow a city | 6 | GPT-5.6 Luna | 6,360 people | 6,360 | 311 | fail | $0.15 | 9.7 | 349eb345 |
| jev-grow-s6 | Grow a city | 6 | Jev Router | 4,580 people | 4,580 | 367 | fail | $1.91 | 46.6 | 981094de |
| gpt56terra-grow-s2 | Grow a city | 2 | GPT-5.6 Terra | 3,660 people | 3,660 | 322 | fail | $1.54 | 13.0 | 476d94f4 |
| gpt56terra-grow-s6 | Grow a city | 6 | GPT-5.6 Terra | 2,920 people (bankrupt) | 2,920 | 756 | fail | $1.41 | 12.6 | 4c859227 |
| sonnet5-grow-s2 | Grow a city | 2 | Claude Sonnet 5 | 1,200 people | 1,200 | 431 | fail | $1.39 | 14.2 | 40c8cf3e |
| sonnet5-grow-s6 | Grow a city | 6 | Claude Sonnet 5 | 940 people | 940 | 514 | fail | $1.32 | 13.1 | 0e6f2bec |
Evaluations computed
Every run's evals-v2 validation report, scored by Jetty: code checks (is the run valid), the judge (was the goal met) and the reviewer checklist.
| Evaluation | Passed | Rate | Share |
|---|---|---|---|
| Code checks (run is valid) | 48/48 | 100% | |
| Checklist items | 24/24 | 100% | |
| Judge: task goal met | 0/8 | 0% | |
| Valid runs (all code checks and checklist items) | 8/8 | 100% | |
| Runs with a passing verdict (valid and goal met) | 0/8 | 0% |
Findings generated
Jev Router builds the biggest cities
medium confidenceMean population in January 1920: Jev Router 12,120 (best 19,660); GPT-5.6 Luna 7,210 (best 8,060); GPT-5.6 Terra 3,290 (best 3,660); Claude Sonnet 5 1,070 (best 1,200). 0 of 8 grow runs met the goal (10,000 people in 1920 without going bankrupt).
The best single city: 19,660 people (Jev Router, seed 2)
high confidenceIt peaked at 25,140 with city score 596 and $2,545 left, but it still missed the goal: the treasury sat under $500 for two years first, which counts as bankruptcy. The weakest model, Claude Sonnet 5, averaged 1,070.
7 of 8 cities peaked and then shrank by 40% or more
medium confidenceThe usual pattern: early overbuilding drains the treasury, the model cannot afford the next power plant, raises taxes to rebuild cash, and demand collapses. Claude Sonnet 5 s2: peak 14,880 → 1,200; GPT-5.6 Terra s2: peak 20,900 → 3,660; GPT-5.6 Luna s2: peak 25,820 → 8,060; Jev Router s6: peak 24,600 → 4,580.
Placement accuracy differs a lot
high confidenceShare of actions the engine rejected (blocked site, no money, off-map), all tasks: Jev Router 4% of 3.5 actions/turn; GPT-5.6 Terra 7% of 4.7 actions/turn; GPT-5.6 Luna 17% of 6.5 actions/turn; Claude Sonnet 5 20% of 4.6 actions/turn.
Proposed improvements generated
Add seeds
Two maps per model separate large gaps but not close ones; four to six seeds would.
Give the mayor a budget forecast
Most collapses start with an unaffordable power plant; a projected January budget in the briefing would test planning, not arithmetic.
Judge the plan, not just the result
An LLM judge over decisions.jsonl could score zoning ratios, power planning and budget discipline per turn.
Caveats
- Two generated maps (seeds 2 and 6), one run each per model.
- Routers such as Jev Router pick a model per request; the model used each turn is in the decision log.
- Each run gives the model 45 wall-clock minutes; slower models may hand the last years to autopilot (flagged per run).
- Population is Micropolis's own census; the city score is a tie-breaker, not part of the goal.
Runs and provenance
| Run | Task | Seed | Mayor | Outcome | Population | City score | Evaluation | Model cost | Wall min | Jetty run ID |
|---|---|---|---|---|---|---|---|---|---|---|
| jev-grow-s2 | Grow a city | 2 | Jev Router | 19,660 people (bankrupt) | 19,660 | 596 | fail | $2.14 | 46.3 | 5de24fe4 |
| sonnet5-grow-s2 | Grow a city | 2 | Claude Sonnet 5 | 1,200 people | 1,200 | 431 | fail | $1.39 | 14.2 | 40c8cf3e |
| gpt56terra-grow-s2 | Grow a city | 2 | GPT-5.6 Terra | 3,660 people | 3,660 | 322 | fail | $1.54 | 13.0 | 476d94f4 |
| gpt56luna-grow-s2 | Grow a city | 2 | GPT-5.6 Luna | 8,060 people | 8,060 | 417 | fail | $0.14 | 8.4 | 831ede26 |
| jev-grow-s6 | Grow a city | 6 | Jev Router | 4,580 people | 4,580 | 367 | fail | $1.91 | 46.6 | 981094de |
| sonnet5-grow-s6 | Grow a city | 6 | Claude Sonnet 5 | 940 people | 940 | 514 | fail | $1.32 | 13.1 | 0e6f2bec |
| gpt56terra-grow-s6 | Grow a city | 6 | GPT-5.6 Terra | 2,920 people (bankrupt) | 2,920 | 756 | fail | $1.41 | 12.6 | 4c859227 |
| gpt56luna-grow-s6 | Grow a city | 6 | GPT-5.6 Luna | 6,360 people | 6,360 | 311 | fail | $0.15 | 9.7 | 349eb345 |