03 — Investigation · Is it worth the cost?
Is the expensive model worth it?
Decision it informs: Which model to pay for when a long-horizon agent task runs many times.
Key takeaway generated
GPT-5.6 Terra met goals for the least money ($7.55 per goal met); per-run cost ranged from $0.08 (GPT-5.6 Luna) to $1.76 (Jev Router).
At a glance computed
Key figure computed
Cost per run vs share of goals met
One dot per model over all its runs. Up and to the left is better.
Table view
| Model | Cost / run | Goals met | Cost / goal met | Latency | Grow pop. |
|---|---|---|---|---|---|
| GPT-5.6 Terra | $0.94 | 1/8 | $7.55 | 15s | 3,290 |
| Jev Router | $1.76 | 0/8 | – | 48s | 12,120 |
| GPT-5.6 Luna | $0.08 | 0/8 | – | 7s | 7,210 |
| Claude Sonnet 5 | $0.72 | 0/8 | – | 10s | 1,070 |
Evaluations computed
Every run's evals-v2 validation report, scored by Jetty: code checks (is the run valid), the judge (was the goal met) and the reviewer checklist.
| Evaluation | Passed | Rate | Share |
|---|---|---|---|
| Code checks (run is valid) | 192/192 | 100% | |
| Checklist items | 96/96 | 100% | |
| Judge: task goal met | 1/32 | 3% | |
| Valid runs (all code checks and checklist items) | 32/32 | 100% | |
| Runs with a passing verdict (valid and goal met) | 1/32 | 3% |
Findings generated
Cost per run spans 22×
high confidenceMean model spend per run: GPT-5.6 Luna $0.08; Claude Sonnet 5 $0.72; GPT-5.6 Terra $0.94; Jev Router $1.76.
GPT-5.6 Terra buys a met goal for the least money
medium confidenceModel spend per goal met: GPT-5.6 Terra $7.55; Jev Router no goals met; GPT-5.6 Luna no goals met; Claude Sonnet 5 no goals met.
Latency, not game time, is what slow models pay
high confidenceMean seconds per turn: GPT-5.6 Luna 7s; Claude Sonnet 5 10s; GPT-5.6 Terra 15s; Jev Router 48s. The game waits for every answer, but the 45-minute budget does not: 3 runs handed their last turns to autopilot.
Proposed improvements generated
Trim the text map
The 60×50 map is most of each prompt; sending only changed rows after turn 1 would cut prompt cost for every model.
Try the cheapest model with more turns
Shorter turns (1 month) cost the cheap model little and may close the gap with the expensive one.
Caveats
- Costs are OpenRouter charges for the mayor model only, not the runbook agent or the sandbox.
- Router pricing varies per request.
- One or two runs per model and task (see the matrix).
Runs and provenance
| Run | Task | Seed | Mayor | Outcome | Population | City score | Evaluation | Model cost | Wall min | Jetty run ID |
|---|---|---|---|---|---|---|---|---|---|---|
| jev-grow-s2 | Grow a city | 2 | Jev Router | 19,660 people (bankrupt) | 19,660 | 596 | fail | $2.14 | 46.3 | 5de24fe4 |
| sonnet5-grow-s2 | Grow a city | 2 | Claude Sonnet 5 | 1,200 people | 1,200 | 431 | fail | $1.39 | 14.2 | 40c8cf3e |
| gpt56terra-grow-s2 | Grow a city | 2 | GPT-5.6 Terra | 3,660 people | 3,660 | 322 | fail | $1.54 | 13.0 | 476d94f4 |
| gpt56luna-grow-s2 | Grow a city | 2 | GPT-5.6 Luna | 8,060 people | 8,060 | 417 | fail | $0.14 | 8.4 | 831ede26 |
| jev-grow-s6 | Grow a city | 6 | Jev Router | 4,580 people | 4,580 | 367 | fail | $1.91 | 46.6 | 981094de |
| sonnet5-grow-s6 | Grow a city | 6 | Claude Sonnet 5 | 940 people | 940 | 514 | fail | $1.32 | 13.1 | 0e6f2bec |
| gpt56terra-grow-s6 | Grow a city | 6 | GPT-5.6 Terra | 2,920 people (bankrupt) | 2,920 | 756 | fail | $1.41 | 12.6 | 4c859227 |
| gpt56luna-grow-s6 | Grow a city | 6 | GPT-5.6 Luna | 6,360 people | 6,360 | 311 | fail | $0.15 | 9.7 | 349eb345 |
| jev-tokyo-s1 | Tokyo 1957 | 1 | Jev Router | lost | 23,120 | 372 | fail | $1.21 | 14.4 | 96f24e6b |
| sonnet5-tokyo-s1 | Tokyo 1957 | 1 | Claude Sonnet 5 | lost | 23,800 | 333 | fail | $0.39 | 3.4 | 58cccc15 |
| gpt56terra-tokyo-s1 | Tokyo 1957 | 1 | GPT-5.6 Terra | won | 29,640 | 501 | pass | $0.61 | 6.0 | 00ff7869 |
| gpt56luna-tokyo-s1 | Tokyo 1957 | 1 | GPT-5.6 Luna | lost | 23,700 | 330 | fail | $0.04 | 2.5 | 4675adce |
| jev-tokyo-s2 | Tokyo 1957 | 2 | Jev Router | lost | 21,900 | 368 | fail | $0.65 | 10.1 | d78fe456 |
| sonnet5-tokyo-s2 | Tokyo 1957 | 2 | Claude Sonnet 5 | lost | 22,400 | 346 | fail | $0.38 | 3.1 | 72ed89e1 |
| gpt56terra-tokyo-s2 | Tokyo 1957 | 2 | GPT-5.6 Terra | lost | 31,020 | 237 | fail | $0.55 | 5.4 | 2cf5d9c4 |
| gpt56luna-tokyo-s2 | Tokyo 1957 | 2 | GPT-5.6 Luna | lost | 29,120 | 458 | fail | $0.04 | 2.5 | a57b0db6 |
| jev-sanfrancisco-s1 | San Francisco 1906 | 1 | Jev Router | lost | 36,300 | 286 | fail | $0.86 | 10.7 | b6be516e |
| sonnet5-sanfrancisco-s1 | San Francisco 1906 | 1 | Claude Sonnet 5 | lost | 36,840 | 23 | fail | $0.42 | 3.7 | df629564 |
| gpt56terra-sanfrancisco-s1 | San Francisco 1906 | 1 | GPT-5.6 Terra | lost | 47,680 | 39 | fail | $0.65 | 6.4 | d5eb8764 |
| gpt56luna-sanfrancisco-s1 | San Francisco 1906 | 1 | GPT-5.6 Luna | lost | 23,680 | 21 | fail | $0.05 | 2.6 | 7a99b8d9 |
| jev-sanfrancisco-s2 | San Francisco 1906 | 2 | Jev Router | lost | 92,360 | 500 | fail | $2.00 | 18.4 | d76eb040 |
| sonnet5-sanfrancisco-s2 | San Francisco 1906 | 2 | Claude Sonnet 5 | lost | 72,300 | 27 | fail | $0.45 | 4.3 | fd728217 |
| gpt56terra-sanfrancisco-s2 | San Francisco 1906 | 2 | GPT-5.6 Terra | lost | 38,880 | 37 | fail | $0.68 | 6.4 | b8e5bc93 |
| gpt56luna-sanfrancisco-s2 | San Francisco 1906 | 2 | GPT-5.6 Luna | lost | 48,220 | 22 | fail | $0.04 | 2.5 | cd12e212 |
| jev-detroit-s1 | Detroit 1972 | 1 | Jev Router | lost | 7,380 | 595 | fail | $2.21 | 26.2 | aea16d25 |
| sonnet5-detroit-s1 | Detroit 1972 | 1 | Claude Sonnet 5 | lost | 3,520 | 586 | fail | $0.73 | 6.9 | 650b3bc0 |
| gpt56terra-detroit-s1 | Detroit 1972 | 1 | GPT-5.6 Terra | lost | 6,120 | 597 | fail | $1.06 | 9.8 | 175f2d53 |
| gpt56luna-detroit-s1 | Detroit 1972 | 1 | GPT-5.6 Luna | lost | 7,400 | 547 | fail | $0.08 | 4.8 | fde1c078 |
| jev-detroit-s2 | Detroit 1972 | 2 | Jev Router | lost | 10,040 | 456 | fail | $3.11 | 45.9 | 85fe296a |
| sonnet5-detroit-s2 | Detroit 1972 | 2 | Claude Sonnet 5 | lost | 3,280 | 341 | fail | $0.72 | 6.6 | c55e7a2b |
| gpt56terra-detroit-s2 | Detroit 1972 | 2 | GPT-5.6 Terra | lost | 6,880 | 693 | fail | $1.05 | 10.8 | ef93e671 |
| gpt56luna-detroit-s2 | Detroit 1972 | 2 | GPT-5.6 Luna | lost | 15,280 | 704 | fail | $0.09 | 5.4 | aba6000b |