evals
Create your eval

02 — Run · Round 1 · Detroit 1972: crime · sonnet5-detroit-s1

Claude Sonnet 5 plays Detroit 1972

Detroit's crime is out of control in 1972. Bring average crime under 60 in 10 years, without emptying the city. Ended Jan 1982: lost, city score 586.

GOAL MISSED seed 1
Claude Sonnet 5
anthropic/claude-sonnet-5

LOST: Average crime 30 (goal < 60), population 3,520 (floor 25,000); engine verdict: won

Population (start → end)76,160 → 3,520 Peak population76,160City score (start → end)285 → 586 Funds at the end$0Crime / pollution / traffic30 / 50 / 0 Zones R / C / I138 / 65 / 89 Power plants (coal / nuclear)2 / 3Police / fire stations13 / 7 Turns (model / failed)40 (40 / 0) Actions applied / rejected118 / 36 Mean turn latency10.1sModel cost$0.730Wall clock6.9 min Jetty run ID650b3bc0

The city over time

Population

People, after every turn

Funds

Treasury, after every turn

City score

0–1000, re-evaluated each January

Table view
TurnDatePopulationFundsCity scoreCrimePollution
0Jan 197276,160$20,0002859262
1Apr 197274,700$18,3435459262
2Jul 197272,160$17,93254510564
3Oct 197267,800$16,80454510864
4Jan 197363,340$14,36247710864
5Apr 197358,400$13,8414779460
6Jul 197350,600$13,1654778257
7Oct 197344,120$12,1394778257
8Jan 197441,340$8,8183915855
9Apr 197438,080$8,8113915354
10Jul 197434,600$8,8113915355
11Oct 197431,760$8,3083914955
12Jan 197528,940$7,1353504652
13Apr 197525,160$7,1353504648
14Jul 197523,840$7,1353504348
15Oct 197522,020$7,1353504248
16Jan 197619,620$3,2263694243
17Apr 197618,060$3,2213693946
18Jul 197615,540$3,2163693846
19Oct 197613,980$3,2163693843
20Jan 197712,740$1,4873303640
21Apr 197711,340$1,4873303540
22Jul 19779,800$1,4873303542
23Oct 19779,160$1,4873303343
24Jan 19788,220$6723203243
25Apr 19787,160$6723203244
26Jul 19786,720$6723203242
27Oct 19785,700$6723203240
28Jan 19795,700$1183243240
29Apr 19795,360$1183243241
30Jul 19794,500$1183243143
31Oct 19794,140$1183243143
32Jan 19803,760$02803147
33Apr 19804,000$02803146
34Jul 19803,440$02803149
35Oct 19803,400$02803149
36Jan 19813,040$03063153
37Apr 19813,040$03063149
38Jul 19812,720$03063049
39Oct 19812,740$03063049
40Jan 19824,620$05863050

Final map

Final city map: Claude Sonnet 5 on Detroit 1972: crime

Evaluation

failchecks 9/10 verdict: valid run (code checks and checklist) and task goal met (judge)

StatusKindCheckDetail
passstepenvironmentEnvironment check (snapshot, engine, key)
passsteplaunchLaunch run-city as a detached process
passstepsuperviseSupervise in the foreground until run-city exits
passsteprenderTimelapse video rendered
passstepnotesAppend Notes to summary.md
passcode_checkoutputs-existall present
passcode_checksim-reached-deadlineended Jan 1982 after 40 turns; scenario verdict True
passcode_checkdecisions-logged40 rows for 40 turns
passcode_checkllm-turns-valid0/40 model turns failed
passcode_checkvideo-decodesduration 36.9s
passcode_checklimits-respected6.9 wall-minutes (model budget 45.0 min)
failjudgescenario-goalLOST: Average crime 30 (goal < 60), population 3,520 (floor 25,000); engine verdict: won (score 0 / threshold 1)
passchecklistnotes-explain-outcomesummary.md Notes section cites model thoughts from turns 1, 28, and 40 and explains why crime goal was met but population goal was not
passchecklistplayer-matches-requestresult.json shows player sonnet5 / anthropic/claude-sonnet-5, task detroit, seed 1
passchecklistno-silent-outagerun.log shows 40 consecutive turns with no failed model calls and no gaps

Decision timeline

What the mayor thought and did each turn. Click a turn to jump the timelapse there.

Ready to stop guessing if your outputs are good?

Create your eval Book a 15-minute walkthrough
Connect your agent to Jetty →