03 — Investigation · Compare setups
Which agent and model draw the best pelican on a bicycle?
Decision it informs: Which agent and model to use for SVG drawing work, and whether a newer model is worth switching to.
Key takeaway generated
opencode · Gemini 3.8 Flash and claude-code · Opus 5.5 tie for the best pelican at 32.8/40 from the external judge, ahead of hermes · Jev Router at 31.7. Current models beat last spring's: the v2 cohort averages 30.1 against 27.9 for v1. The gaps are a few points on one judge with one lineage per setup, so rank the top three as a group and re-run before choosing between them.
At a glance computed
Key figure computed
opencode · Gemini 3.8 Flash and claude-code · Opus 5.5 share the top; v2 models fill most of it
One bar per agent × model: the external judge's score (out of 40) for the round the agent picked as its best, coloured by cohort. The ring marks the judge's favourite round in that lineage's climb; the dashed line is the 28/40 pass threshold. Click a bar to open that lineage's runs.
Table view
| Agent · model | Cohort | Agent's pick | Judge /40 | Self /40 | Judge's best round |
|---|---|---|---|---|---|
| opencode · Gemini 3.8 Flashgoogle/gemini-3.8-flash | v2 | v1 | 32.8 | 38.5 | v1 · 32.8 |
| claude-code · Opus 5.5anthropic/claude-opus-5.5 | v2 | v3 | 32.8 | 36 | v5 · 33.3 |
| hermes · Jev Routertypesafe/jev-router | v2 | v4 | 31.7 | 37 | v4 · 31.7 |
| gemini-cli · Gemini 3.5 Flashgemini-3.5-flash | v1 | v1 | 29.8 | 40 | v5 · 32.3 |
| opencode · Gemini 3.5 Flashgoogle/gemini-3.5-flash | v1 | v2 | 29.3 | 40 | v9 · 31.5 |
| hermes · Fusion routeropenrouter/fusion | v1 | v1 | 28.8 | 34 | v3 · 29 |
| claude-code · Opus 4.7claude-opus-4-7 | v1 | v3 | 28.7 | 36 | v4 · 29.7 |
| opencode · Fusion routeropenrouter/fusion | v1 | v2 | 28.3 | 36 | v1 · 29.3 |
| hermes · Gemini 3.5 Flashopenrouter/google/gemini-3.5-flash | v1 | v9 | 28.2 | 39.5 | v5 · 30 |
| opencode · GPT-5.6 Terraopenai/gpt-5.6-terra | v2 | v1 | 28 | 38 | v5 · 29.8 |
| hermes · GLM 5.1openrouter/z-ai/glm-5.1 | v1 | v2 | 27.3 | 36.5 | v3 · 28.8 |
| claude-code · Sonnet 4.6claude-sonnet-4-6 | v1 | v5 | 26.2 | 37 | v10 · 28 |
| claude-code · Sonnet 5anthropic/claude-sonnet-5 | v2 | v4 | 25.3 | 36 | v2 · 27.3 |
| hermes · Sonnet 4.6openrouter/anthropic/claude-sonnet-4.6 | v1 | v5 | 24 | 36 | v2 · 27 |
Evaluations computed
These runs used runbook v1, which wrote no validation report, so the scorecard is replayed here: the v2 runbook's code checks re-run on every stored SVG, plus two judge checks at 28/40 (the external judge, and the agent's own self-score, which is what the v2 runbook gates on). The overall row is the derived verdict shown on Runs.
| Evaluation | Passed | Rate | Share |
|---|---|---|---|
| Code check · svg-parses | 98/98 | 100% | |
| Code check · svg-size-under-50kb | 98/98 | 100% | |
| Code check · no-image-tags | 98/98 | 100% | |
| Code check · png-rendered | 98/98 | 100% | |
| Judge · external judge ≥ 28/40 | 52/98 | 53% | |
| Judge · agent self-score ≥ 28/40 (the v2 runbook's judge check) | 98/98 | 100% | |
| Overall · derived verdict (code checks + external judge) | 52/98 | 53% |
Findings generated
The top three are within 1.2 points, and two of them are v2 models
medium confidenceBy the judge's score for each agent's own pick, opencode · Gemini 3.8 Flash and claude-code · Opus 5.5 tie at 32.8/40, then hermes · Jev Router (31.7). opencode · Gemini 3.8 Flash got there in a single round before two OpenRouter outages ended its lineage. Differences this small sit inside what one judge on one lineage can resolve.
Current models draw better pelicans than last spring's
high confidenceThe v2 cohort (5 lineages) averages 30.1/40 from the judge against 27.9 for v1 (9 lineages). 3 of 5 v2 setups beat the best v1 setup, gemini-cli · Gemini 3.5 Flash at 29.8, and v2 did it in at most five rounds instead of ten.
The model matters more than the agent harness around it
medium confidenceThe same model run through different agents lands within 2.3 points of itself on its best round by the judge: gemini-cli · Gemini 3.5 Flash 32.3; opencode · Gemini 3.5 Flash 31.5; hermes · Gemini 3.5 Flash 30. The spread between models in the same cohort is larger, so pick the model first.
Agents often keep the wrong drawing
high confidenceIn 12 of 14 lineages the round the agent picked as its best is not the round the judge scored highest; the judge's favourite scored 1.6 points more on average. Worst case: hermes · Sonnet 4.6 kept v5 (24) over v2 (27). A ranking by the agents' picks understates what each setup can draw.
Proposed improvements generated
Pick the kept round with the external judge, not the self-score
Would change the kept drawing in 12 of 14 lineages, +1.6 judge points each on average.
Re-run the top three setups three times each
Tells whether the 1.2-point spread at the top is real; today each setup has one lineage.
Add a second judge from a different vendor
The judge (Opus 5.5) is also a contestant; a second judge shows whether the lead survives a different yardstick.
Caveats
- One external judge (Claude Opus 5.5), and it is also a contestant: weigh the Opus 5.5 row accordingly.
- One lineage per agent × model; no repeat runs, so there are no error bars on any setup.
- v1 and v2 differ in round count (10 vs 5) and in the models' era, not only in the model.
- opencode · Gemini 3.8 Flash has a single round: its lineage ended after two OpenRouter outages.
- Scores are for the agent's own pick of its best round; the ring marker shows the judge's favourite round.
Runs and provenance
Computed on 2026-09-28 from results.json and judge-raw.json. Task pelican-bicycle-svg. Every run is listed on 02 · Runs.
| Lineage | Cohort | Collection | Runs | Trajectories |
|---|---|---|---|---|
| claude-code · Opus 5.5 | v2 | jettygrowthteam | 5 | 5361e30a ba932f3e 35c47f4e 2e046a89 0e41c0a0 |
| claude-code · Sonnet 5 | v2 | jettygrowthteam | 5 | ea416761 b77ea720 6e98127a 240351f2 1925c665 |
| hermes · Jev Router | v2 | jettygrowthteam | 5 | 1a125d51 7a63ffb0 ded99127 3ba61c00 ef67915d |
| opencode · Gemini 3.8 Flash | v2 | jettygrowthteam | 1 | 225e57cd |
| opencode · GPT-5.6 Terra | v2 | jettygrowthteam | 5 | ccbdd583 699fefde ce858eaf 9fe62cbb de112455 |
| claude-code · Opus 4.7 | v1 | jettyio | 10 | c3121188 cb2a7f37 d4a107dc 8ec2e90c 4160dcf1 4848f8a3 df18c780 20f46c4d 4897cbca 38634df1 |
| claude-code · Sonnet 4.6 | v1 | jettyio | 11 | 7ddfcf08 414e644d b169371d 063dfed9 7ddc3c19 39febb11 ddf779bd 79fdee62 5f980543 2043d7e3 3a4c3357 |
| gemini-cli · Gemini 3.5 Flash | v1 | jettyio | 10 | f5cd6a31 c990e7c4 5d360721 4ed5de35 a652e4e3 eca90d64 f08684e3 ce2de30f 4e122677 24091968 |
| hermes · Gemini 3.5 Flash | v1 | jettyio | 10 | c118fb80 2128ffc1 2405db86 822a3bca 60502506 2c2a74e8 c165de92 d391e018 e6d4548b 3e21bf05 |
| hermes · Fusion router | v1 | jettyio | 3 | 2d0c10c4 b38f9569 cb138790 |
| hermes · GLM 5.1 | v1 | jettyio | 10 | c1f2da04 4b324d80 f443b217 7ce1468b ab8cb01d 9fd19804 83e8efe2 21b9b8c1 9f9e50be fff60669 |
| hermes · Sonnet 4.6 | v1 | jettyio | 10 | 697b57d4 acb2ba11 99a389e4 28f94566 8969a1a0 ebc95c4b fdf47c73 25323cb8 5b9cf6ae 34f10b3e |
| opencode · Gemini 3.5 Flash | v1 | jettyio | 10 | 67836d67 660d2a66 5cd01e9e b130c9c2 d94ff2e1 4a378be2 e3059c87 70898b5c b56c0c03 2a46de05 |
| opencode · Fusion router | v1 | jettyio | 3 | 40df25cb bfa3612b 9776d557 |