---
version: "1.0.0"
evals_version: 2
evaluation: programmatic
agent: claude-code
model: anthropic/claude-sonnet-4.6
model_provider: openrouter
snapshot: micropolis-city
primary_outputs:
  - timelapse.mp4
  - summary.md
  - result.json
secrets:
  OPENROUTER_API_KEY:
    env: OPENROUTER_API_KEY
    description: "OpenRouter key used by the model playing mayor (and by this runbook's agent)"
    required: true
---

# Micropolis: one LLM plays one city

## Objective

Let **one decision model play one Micropolis task** (Micropolis is the open-source release of the original SimCity,
GPL v3) and deliver the timelapse video, the final map, the save file, every decision, the per-turn city statistics and
an evals-v2 validation report to `{{results_dir}}`.

The game is turn-based: every turn the model gets a text briefing of the city (date, money, population, demand,
problems, power, a coarse map with coordinates, free building sites, last turn's action results and its own notes) and
replies with JSON actions (zone, build, road/rail/power line, bulldoze, tax, funding). The simulation then runs
`{{months_per_turn}}` game months before the next turn. Model latency never costs game time.

Tasks (`{{task}}`):

| Task | Start | Deadline | Goal (the judge) |
|------|-------|----------|------------------|
| `grow` | empty generated map (`{{seed}}` = map seed), $20,000 | 20 years (Jan 1920) | population >= 10,000, never bankrupt (treasury under $500 for 8 turns in a row) |
| `tokyo` | Tokyo 1957, a monster attacks | 5 years | city score > 500 (the scenario's own rule) |
| `sanfrancisco` | San Francisco 1906, an earthquake hits | 5 years | Metropolis, population >= 100,000 (the scenario's own rule) |
| `detroit` | Detroit 1972, chronic crime | 10 years | average crime < 60 (the scenario's own rule) and population >= 25,000 |

Rules are the same for every model: no random disasters (a scenario's scripted disaster still happens), manual
budget (the model's funding levels hold), trees and rubble are auto-bulldozed when building, easy difficulty.
`{{wall_minutes}}` is the model's wall-clock budget: after it (or after `{{turns}}` model turns, if set), the city keeps
running to the deadline with no new actions, so the outcome is always measured at the same game date.

The `micropolis-city` snapshot ships everything prebuilt: the Micropolis engine compiled as a Python extension, the
game resources at `/opt/micropolis-engine`, the harness at `/opt/micropolis-eval`, and ffmpeg. The entry points are
`run-city` (plays, records and scores a task) and `city-report` (writes the validation report). Do not build or
install anything.

## REQUIRED OUTPUT FILES

- `{{results_dir}}/result.json`: task, player, seed, final city (date, population, funds, score, crime...), the judge
  outcome, bankruptcy, model cost, latency and failure counts
- `{{results_dir}}/metrics.json`: headline metrics (final and peak population, city score, actions applied/rejected, cost)
- `{{results_dir}}/decisions.jsonl`: one row per turn (the briefing the model saw, its thoughts, notes, actions and their results)
- `{{results_dir}}/series.json` and `series.csv`: city statistics after every turn (for charts)
- `{{results_dir}}/timelapse.mp4`: the whole game, one frame per game week, and `thumb.jpg`
- `{{results_dir}}/final_map.png`: the final city at full resolution (1920x1600)
- `{{results_dir}}/city.cty`: Micropolis save file of the final city (opens in Micropolis / micropolisJS)
- `{{results_dir}}/runtime_config.json`: player, model, task, seed, limits, engine commit, harness hashes, host
- `{{results_dir}}/summary.md`: result table plus the Notes you add in Step 5
- `{{results_dir}}/validation_report.json`: evals-v2 report (`"version": 2`, `checks[]`), written in the last section

## Parameters

| Parameter | Template Variable | Default | Description |
|-----------|-------------------|---------|-------------|
| Results directory | `{{results_dir}}` | `/app/results` | Output directory (persisted by Jetty) |
| Player | `{{player}}` | `jev` | Who plays mayor (see Player specs) |
| Task | `{{task}}` | `grow` | `grow`, `tokyo`, `sanfrancisco` or `detroit` |
| Seed | `{{seed}}` | `2` | Map seed for `grow`; simulation random seed for scenarios |
| Run ID | `{{run_id}}` | `city` | Label for this run (used in logs and the result) |
| Months per turn | `{{months_per_turn}}` | `3` | Game months simulated between two model turns |
| Max model turns | `{{turns}}` | `0` | Cap on model turns (0 = every turn to the deadline) |
| Wall-clock budget | `{{wall_minutes}}` | `45` | Real-time minutes the model may use; afterwards the city runs to the deadline untouched |

Player specs:

- `jev`, `sonnet5`, `gpt56terra`, `gpt56luna`: roster presets (Jev Router, Claude Sonnet 5, GPT-5.6 Terra, GPT-5.6 Luna;
  the last three with `reasoning: {effort: low}`)
- `openrouter:<model>` or `openrouter:<model>|<Display Name>`: any OpenRouter chat model
- `idle`: a baseline mayor that never acts (no model calls)

## Dependencies

- `micropolis-city` snapshot: engine `SimHacker/micropolis@c98f6b0` compiled with `engine/binding.cpp`, resources at
  `/opt/micropolis-engine`, `run-city` and `city-report` on PATH
- `OPENROUTER_API_KEY` collection environment variable (not needed for `idle`)

## Steps

### Step 1: Environment Setup

```bash
mkdir -p {{results_dir}}
test -x /usr/local/bin/run-city && test -x /usr/local/bin/city-report && test -f /opt/micropolis-engine/res/snro.555 \
  && /opt/micropolis-eval/.venv/bin/python -c "import sys; sys.path.insert(0,'/opt/micropolis-eval'); import mpcity._micropolis" \
  && echo "PASS: micropolis-city snapshot present" || echo "FAIL: not running on the micropolis-city snapshot"
test -n "${OPENROUTER_API_KEY:-}" && echo "PASS: OPENROUTER_API_KEY set" || echo "WARN: OPENROUTER_API_KEY missing (only the idle player will work)"
nproc; free -g | head -2
```

If the snapshot check fails, stop: record `environment` as failed in the validation report (last section) and explain
in `summary.md`. Do not try to build Micropolis inside the sandbox.

### Step 2: Launch the run as a detached process

A run takes up to `{{wall_minutes}}` minutes, longer than one shell call may block. Start it with `nohup` (a plain shell
command, not an agent background task):

```bash
cd {{results_dir}}
nohup run-city --player "{{player}}" --task "{{task}}" --seed {{seed}} --run-id "{{run_id}}" \
  --months-per-turn {{months_per_turn}} --turns {{turns}} --wall-minutes {{wall_minutes}} --out {{results_dir}} \
  > {{results_dir}}/run.log 2>&1 &
echo $! > /tmp/run-city.pid
sleep 45; tail -5 {{results_dir}}/run.log
```

Within a minute `run.log` shows `Starting ...` and then one line per turn:
`[turn   3] Jul 1900 pop     100 $ 11,426 score 719 | 13 actions, 12 ok | 7.1s $0.006`.

### Step 3: Supervise until the run ends (foreground only)

> **CRITICAL:** the sandbox, and the city with it, is destroyed the moment your session ends. Stay in this step until
> `run-city` has exited. Do NOT use background tasks, `run_in_background`, `tail -f`, cron or any other asynchronous
> watcher, and do NOT end your turn "to check back later": there is no later.

Run this exact command. It blocks for at most 9 minutes and prints the latest progress:

```bash
PID=$(cat /tmp/run-city.pid)
for i in $(seq 1 54); do
  if ! kill -0 "$PID" 2>/dev/null; then echo "RUN-CITY EXITED"; break; fi
  sleep 10
done
tail -3 {{results_dir}}/run.log
```

Repeat it until the output contains `RUN-CITY EXITED` (expect 1 to 6 repetitions). Never kill a run that is still
logging turns.

### Step 4: Handle failures (max 1 retry)

Read the preliminary `{{results_dir}}/validation_report.json` that `run-city` wrote, and `run.log`.

- No `result.json` (crash, or the process died): check the end of `run.log`, then relaunch Step 2 **once** with
  `--run-id "{{run_id}}-retry"` into the same results directory and supervise it again.
- `llm-turns-valid` failed because the model provider was down (many `FAILED` turns in `run.log`): relaunch once as above.
- Never rerun a finished city just to get a better outcome: a lost scenario or a small city is a valid result.

### Step 5: Notes

Append a `## Notes` section to `{{results_dir}}/summary.md`: anything unusual in `run.log` (model errors, retries,
autopilot), then three or four sentences on how the city developed and why the goal was or was not met, citing the
model's own `thoughts` from two or three turns of `decisions.jsonl` (for example the first turn, a turning point where
population, money or the score changed sharply in `series.csv`, and the last turn).

## Code Checks

Each check is a deterministic command; a failing check fails the run (it means the run is not a valid measurement).
`city-report` runs the same checks and records them, so run these for your own verification and then write the report
as described below.

### outputs-exist — Every required output file exists and is non-empty

```bash
cd {{results_dir}} && for f in result.json metrics.json decisions.jsonl series.json timelapse.mp4 thumb.jpg final_map.png city.cty runtime_config.json summary.md; do test -s "$f" || { echo "missing $f"; exit 1; }; done
```

### sim-reached-deadline — The simulation ran to the task's deadline (or the scenario was judged)

```bash
python3 -c "import json; r=json.load(open('{{results_dir}}/result.json')); assert r['reached_deadline'], r['final']['date']; print(r['final']['date'], r['turns'])"
```

### decisions-logged — decisions.jsonl has one row per turn

```bash
python3 -c "import json; r=json.load(open('{{results_dir}}/result.json')); n=sum(1 for _ in open('{{results_dir}}/decisions.jsonl')); assert n and n==r['turns'], (n, r['turns'])"
```

### llm-turns-valid — At most 20% of the model's turns failed

```bash
python3 -c "import json; r=json.load(open('{{results_dir}}/result.json')); c,f=r['llm_calls'],r['llm_failures']; assert c and f<=max(2,0.2*c), (f,c)"
```

### video-decodes — timelapse.mp4 decodes

```bash
test "$(ffprobe -v error -show_entries format=duration -of csv=p=0 {{results_dir}}/timelapse.mp4 | cut -d. -f1)" -gt 0
```

### limits-respected — The run finished within the wall-clock budget

```bash
python3 -c "import json; r=json.load(open('{{results_dir}}/result.json')); assert r['wall_seconds']/60 <= {{wall_minutes}}+10, r['wall_seconds']"
```

## Checklist

Confirm each item by inspection; a failed item fails the run.

- [ ] `notes-explain-outcome`: summary.md has a Notes section explaining the outcome, citing the model's own thoughts
- [ ] `player-matches-request`: the player, model, task and seed in result.json are the ones requested in `{{player}}`, `{{task}}` and `{{seed}}`
- [ ] `no-silent-outage`: run.log shows no long stretch of failed or empty model turns that the code checks did not already flag

## Write Validation Report

Write `{{results_dir}}/validation_report.json` in the evals-v2 shape with `city-report`. It runs every code check above,
adds one `judge` check for the task goal (from `result.json`: for `grow` the score is the final population, threshold
10,000, and a bankrupt city scores 0; for a scenario the score is 1 if the scenario was won and 0 if lost, threshold 1),
records one `step` entry per step you pass in, and one `checklist` entry per item you pass in (status, a colon, then a
one-line reason). Jetty computes the verdict from these checks: the run passes only if it is valid (every code check
and checklist item passes) AND the model met the goal (judge score >= threshold).

```bash
city-report --results {{results_dir}} --iterations 1 \
  --steps '{"environment": "pass", "launch": "pass", "supervise": "pass", "render": "pass", "notes": "pass"}' \
  --checklist '{"notes-explain-outcome": "pass: <why>", "player-matches-request": "pass: <why>", "no-silent-outage": "pass: <why>"}'
```

Use `fail: <reason>` for any step or checklist item that did not hold, and `--iterations 2` if you used the retry in
Step 4. The command prints every check and the verdict. A failed judge (the model did not reach the goal) is a normal
outcome: report it, do not retry. Do not hand-edit the file and do not invent a different format: it must stay
`{"version": 2, "verdict", "overall_passed", "iterations", "checks": [...]}`.

## Tips

- Model cost is small (roughly $0.02 to $2 per run); most of a run's cost is this runbook's agent.
- `decisions.jsonl` rows contain the full briefing the model saw, which is useful for debugging strange play.
- `city.cty` loads in any Micropolis build (e.g. micropolisJS) to explore the final city.

## Changelog

- 1.0.0: Initial version: four tasks (grow, tokyo, sanfrancisco, detroit), 3-month turns, evals v2 with a judge check.
