--- title: "Is the expensive model worth it? | Micropolis investigation | Evals by Jetty" url: https://evaljetty.com/micropolis/investigations/cost-vs-outcome.html description: "GPT-5.6 Terra met goals for the least money ($7.55 per goal met); per-run cost ranged from $0.08 (GPT-5.6 Luna) to $1.76 (Jev Router)." updated: 2026-09-28 publisher: Jetty (https://jetty.io) --- 03 — Investigation · Is it worth the cost? # Is the expensive model worth it? 32 runsRound 13 game months per turnrunbook v1.0.0collection jettygrowthteamtask micropolis-cityupdated 2026-09-28 **Decision it informs:** Which model to pay for when a long-horizon agent task runs many times. ## Key takeaway generated GPT-5.6 Terra met goals for the least money ($7.55 per goal met); per-run cost ranged from $0.08 (GPT-5.6 Luna) to $1.76 (Jev Router). ## At a glance computed Runs 32 one city per Jetty run Model spend $28.08 OpenRouter, the mayor models Goals met 1/32 judge score ≥ threshold Eval pass rate 3% 1/32 runs valid and goal met ## Key figure computed ### Cost per run vs share of goals met One dot per model over all its runs. Up and to the left is better. Table view | Model | Cost / run | Goals met | Cost / goal met | Latency | Grow pop. | | --- | --- | --- | --- | --- | --- | | GPT-5.6 Terra | $0.94 | 1/8 | $7.55 | 15s | 3,290 | | Jev Router | $1.76 | 0/8 | – | 48s | 12,120 | | GPT-5.6 Luna | $0.08 | 0/8 | – | 7s | 7,210 | | Claude Sonnet 5 | $0.72 | 0/8 | – | 10s | 1,070 | ## Evaluations computed Every run's evals-v2 validation report, scored by Jetty: code checks (is the run valid), the judge (was the goal met) and the reviewer checklist. | Evaluation | Passed | Rate | Share | | --- | --- | --- | --- | | Code checks (run is valid) | 192/192 | 100% | | | Checklist items | 96/96 | 100% | | | Judge: task goal met | 1/32 | 3% | | | Valid runs (all code checks and checklist items) | 32/32 | 100% | | | Runs with a passing verdict (valid and goal met) | 1/32 | 3% | | ## Findings generated ### Cost per run spans 22× high confidence Mean model spend per run: GPT-5.6 Luna $0.08; Claude Sonnet 5 $0.72; GPT-5.6 Terra $0.94; Jev Router $1.76. 32 runsmodel\_cost\_usd in runs.csv ### GPT-5.6 Terra buys a met goal for the least money medium confidence Model spend per goal met: GPT-5.6 Terra $7.55; Jev Router no goals met; GPT-5.6 Luna no goals met; Claude Sonnet 5 no goals met. GPT-5.6 Terra 1/8 ### Latency, not game time, is what slow models pay high confidence Mean seconds per turn: GPT-5.6 Luna 7s; Claude Sonnet 5 10s; GPT-5.6 Terra 15s; Jev Router 48s. The game waits for every answer, but the 45-minute budget does not: 3 runs handed their last turns to autopilot. Jev Router 3 autopilot runs ## Proposed improvements generated Runbook ### Trim the text map The 60×50 map is most of each prompt; sending only changed rows after turn 1 would cut prompt cost for every model. backed by: briefings in decisions.jsonl Run ### Try the cheapest model with more turns Shorter turns (1 month) cost the cheap model little and may close the gap with the expensive one. backed by: GPT-5.6 Luna $0.08 a run ## Caveats - Costs are OpenRouter charges for the mayor model only, not the runbook agent or the sandbox. - Router pricing varies per request. - One or two runs per model and task (see the matrix). **Runs and provenance** | Run | Task | Seed | Mayor | Outcome | Population | City score | Evaluation | Model cost | Wall min | Jetty run ID | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | [jev-grow-s2](https://evaljetty.com/micropolis/runs/jev-grow-s2.html) | Grow a city | 2 | Jev Router | 19,660 people (bankrupt) | 19,660 | 596 | fail | $2.14 | 46.3 | 5de24fe4 | | [sonnet5-grow-s2](https://evaljetty.com/micropolis/runs/sonnet5-grow-s2.html) | Grow a city | 2 | Claude Sonnet 5 | 1,200 people | 1,200 | 431 | fail | $1.39 | 14.2 | 40c8cf3e | | [gpt56terra-grow-s2](https://evaljetty.com/micropolis/runs/gpt56terra-grow-s2.html) | Grow a city | 2 | GPT-5.6 Terra | 3,660 people | 3,660 | 322 | fail | $1.54 | 13.0 | 476d94f4 | | [gpt56luna-grow-s2](https://evaljetty.com/micropolis/runs/gpt56luna-grow-s2.html) | Grow a city | 2 | GPT-5.6 Luna | 8,060 people | 8,060 | 417 | fail | $0.14 | 8.4 | 831ede26 | | [jev-grow-s6](https://evaljetty.com/micropolis/runs/jev-grow-s6.html) | Grow a city | 6 | Jev Router | 4,580 people | 4,580 | 367 | fail | $1.91 | 46.6 | 981094de | | [sonnet5-grow-s6](https://evaljetty.com/micropolis/runs/sonnet5-grow-s6.html) | Grow a city | 6 | Claude Sonnet 5 | 940 people | 940 | 514 | fail | $1.32 | 13.1 | 0e6f2bec | | [gpt56terra-grow-s6](https://evaljetty.com/micropolis/runs/gpt56terra-grow-s6.html) | Grow a city | 6 | GPT-5.6 Terra | 2,920 people (bankrupt) | 2,920 | 756 | fail | $1.41 | 12.6 | 4c859227 | | [gpt56luna-grow-s6](https://evaljetty.com/micropolis/runs/gpt56luna-grow-s6.html) | Grow a city | 6 | GPT-5.6 Luna | 6,360 people | 6,360 | 311 | fail | $0.15 | 9.7 | 349eb345 | | [jev-tokyo-s1](https://evaljetty.com/micropolis/runs/jev-tokyo-s1.html) | Tokyo 1957 | 1 | Jev Router | lost | 23,120 | 372 | fail | $1.21 | 14.4 | 96f24e6b | | [sonnet5-tokyo-s1](https://evaljetty.com/micropolis/runs/sonnet5-tokyo-s1.html) | Tokyo 1957 | 1 | Claude Sonnet 5 | lost | 23,800 | 333 | fail | $0.39 | 3.4 | 58cccc15 | | [gpt56terra-tokyo-s1](https://evaljetty.com/micropolis/runs/gpt56terra-tokyo-s1.html) | Tokyo 1957 | 1 | GPT-5.6 Terra | won | 29,640 | 501 | pass | $0.61 | 6.0 | 00ff7869 | | [gpt56luna-tokyo-s1](https://evaljetty.com/micropolis/runs/gpt56luna-tokyo-s1.html) | Tokyo 1957 | 1 | GPT-5.6 Luna | lost | 23,700 | 330 | fail | $0.04 | 2.5 | 4675adce | | [jev-tokyo-s2](https://evaljetty.com/micropolis/runs/jev-tokyo-s2.html) | Tokyo 1957 | 2 | Jev Router | lost | 21,900 | 368 | fail | $0.65 | 10.1 | d78fe456 | | [sonnet5-tokyo-s2](https://evaljetty.com/micropolis/runs/sonnet5-tokyo-s2.html) | Tokyo 1957 | 2 | Claude Sonnet 5 | lost | 22,400 | 346 | fail | $0.38 | 3.1 | 72ed89e1 | | [gpt56terra-tokyo-s2](https://evaljetty.com/micropolis/runs/gpt56terra-tokyo-s2.html) | Tokyo 1957 | 2 | GPT-5.6 Terra | lost | 31,020 | 237 | fail | $0.55 | 5.4 | 2cf5d9c4 | | [gpt56luna-tokyo-s2](https://evaljetty.com/micropolis/runs/gpt56luna-tokyo-s2.html) | Tokyo 1957 | 2 | GPT-5.6 Luna | lost | 29,120 | 458 | fail | $0.04 | 2.5 | a57b0db6 | | [jev-sanfrancisco-s1](https://evaljetty.com/micropolis/runs/jev-sanfrancisco-s1.html) | San Francisco 1906 | 1 | Jev Router | lost | 36,300 | 286 | fail | $0.86 | 10.7 | b6be516e | | [sonnet5-sanfrancisco-s1](https://evaljetty.com/micropolis/runs/sonnet5-sanfrancisco-s1.html) | San Francisco 1906 | 1 | Claude Sonnet 5 | lost | 36,840 | 23 | fail | $0.42 | 3.7 | df629564 | | [gpt56terra-sanfrancisco-s1](https://evaljetty.com/micropolis/runs/gpt56terra-sanfrancisco-s1.html) | San Francisco 1906 | 1 | GPT-5.6 Terra | lost | 47,680 | 39 | fail | $0.65 | 6.4 | d5eb8764 | | [gpt56luna-sanfrancisco-s1](https://evaljetty.com/micropolis/runs/gpt56luna-sanfrancisco-s1.html) | San Francisco 1906 | 1 | GPT-5.6 Luna | lost | 23,680 | 21 | fail | $0.05 | 2.6 | 7a99b8d9 | | [jev-sanfrancisco-s2](https://evaljetty.com/micropolis/runs/jev-sanfrancisco-s2.html) | San Francisco 1906 | 2 | Jev Router | lost | 92,360 | 500 | fail | $2.00 | 18.4 | d76eb040 | | [sonnet5-sanfrancisco-s2](https://evaljetty.com/micropolis/runs/sonnet5-sanfrancisco-s2.html) | San Francisco 1906 | 2 | Claude Sonnet 5 | lost | 72,300 | 27 | fail | $0.45 | 4.3 | fd728217 | | [gpt56terra-sanfrancisco-s2](https://evaljetty.com/micropolis/runs/gpt56terra-sanfrancisco-s2.html) | San Francisco 1906 | 2 | GPT-5.6 Terra | lost | 38,880 | 37 | fail | $0.68 | 6.4 | b8e5bc93 | | [gpt56luna-sanfrancisco-s2](https://evaljetty.com/micropolis/runs/gpt56luna-sanfrancisco-s2.html) | San Francisco 1906 | 2 | GPT-5.6 Luna | lost | 48,220 | 22 | fail | $0.04 | 2.5 | cd12e212 | | [jev-detroit-s1](https://evaljetty.com/micropolis/runs/jev-detroit-s1.html) | Detroit 1972 | 1 | Jev Router | lost | 7,380 | 595 | fail | $2.21 | 26.2 | aea16d25 | | [sonnet5-detroit-s1](https://evaljetty.com/micropolis/runs/sonnet5-detroit-s1.html) | Detroit 1972 | 1 | Claude Sonnet 5 | lost | 3,520 | 586 | fail | $0.73 | 6.9 | 650b3bc0 | | [gpt56terra-detroit-s1](https://evaljetty.com/micropolis/runs/gpt56terra-detroit-s1.html) | Detroit 1972 | 1 | GPT-5.6 Terra | lost | 6,120 | 597 | fail | $1.06 | 9.8 | 175f2d53 | | [gpt56luna-detroit-s1](https://evaljetty.com/micropolis/runs/gpt56luna-detroit-s1.html) | Detroit 1972 | 1 | GPT-5.6 Luna | lost | 7,400 | 547 | fail | $0.08 | 4.8 | fde1c078 | | [jev-detroit-s2](https://evaljetty.com/micropolis/runs/jev-detroit-s2.html) | Detroit 1972 | 2 | Jev Router | lost | 10,040 | 456 | fail | $3.11 | 45.9 | 85fe296a | | [sonnet5-detroit-s2](https://evaljetty.com/micropolis/runs/sonnet5-detroit-s2.html) | Detroit 1972 | 2 | Claude Sonnet 5 | lost | 3,280 | 341 | fail | $0.72 | 6.6 | c55e7a2b | | [gpt56terra-detroit-s2](https://evaljetty.com/micropolis/runs/gpt56terra-detroit-s2.html) | Detroit 1972 | 2 | GPT-5.6 Terra | lost | 6,880 | 693 | fail | $1.05 | 10.8 | ef93e671 | | [gpt56luna-detroit-s2](https://evaljetty.com/micropolis/runs/gpt56luna-detroit-s2.html) | Detroit 1972 | 2 | GPT-5.6 Luna | lost | 15,280 | 704 | fail | $0.09 | 5.4 | aba6000b | --- Source: https://evaljetty.com/micropolis/investigations/cost-vs-outcome.html · Evals by Jetty · Run your own evals: https://jetty.io/?utm_source=evaljetty&utm_medium=llms