evals
Run evals on Jetty

02 — Runs · v1 · jettyio · auto-evolved runbook (weakest-axis targeting)

hermes
openrouter/google/gemini-3.5-flash

Same hermes agent, Sonnet swapped for Flash: +5 self-score points from the model alone.

Judge, best round
28.2/40
round v9 (agent's pick)
Self-score
39.5/40
inflation +11.3
Runs
10
not recorded (v1)
hermes · Gemini 3.5 Flash best round
best round v9 · e6d4548b

Self-score vs judge, by axis

Best round v9, 0–10 per axis

agent self-scoreexternal judge

Judge: The bicycle is well formed, with spoked wheels, chain, crank, fenders and a fish crate, set against an attractive seaside backdrop. The pelican's anatomy is confused: the huge orange pouch reads as a detached sheet, the head is tiny and tilted back, the body is a jumble of white blobs with an odd grey tail, and the feet only loosely meet the pedals.

Score by round

Out of 40. Each round starts from the previous round's best SVG. Dashed line: the 28/40 pass threshold.

self-scorejudgepass ≥ 28

10 drawings, oldest first.

Click a drawing to open it in the head-to-head viewer. The outlined card is the agent's best round. Verdicts are derived (code checks on the stored SVG and judge ≥ 28/40); see Runs.

hermes · Gemini 3.5 Flash round v1
v1 passjudge 28.5 · self 39 c118fb80runbook
hermes · Gemini 3.5 Flash round v2
v2 passjudge 28.2 · self 38 2128ffc1runbook
hermes · Gemini 3.5 Flash round v3
v3 failjudge 26.5 · self 37 2405db86runbook
hermes · Gemini 3.5 Flash round v4
v4 passjudge 28.7 · self 39 822a3bcarunbook
hermes · Gemini 3.5 Flash round v5
v5 passjudge 30 · self 38.8 60502506runbook
hermes · Gemini 3.5 Flash round v6
v6 failjudge 26.8 · self 38 2c2a74e8runbook
hermes · Gemini 3.5 Flash round v7
v7 failjudge 27 · self 38.5 c165de92runbook
hermes · Gemini 3.5 Flash round v8
v8 failjudge 26.3 · self 38 d391e018runbook
hermes · Gemini 3.5 Flash round v9
v9 · best passjudge 28.2 · self 39.5 e6d4548brunbook
hermes · Gemini 3.5 Flash round v10
v10 failjudge 26.5 · self 38.5 3e21bf05runbook

Scores per round

RoundPelicanBicycleCompositionPolishSelf totalJudge totalVerdictJudge spreadCostMinutesTrajectory
v17self 107.5self 9.56.8self 107.2self 9.53928.5pass1––c118fb80
v27self 97.8self 106.3self 97self 103828.2pass1––2128ffc1
v36.3self 9.57.3self 96.3self 9.56.5self 9.53726.5fail0––2405db86
v47self 107.7self 106.8self 9.57.2self 9.53928.7pass1.5––822a3bca
v57self 9.58.3self 9.57.2self 107.5self 9.838.830pass0––60502506
v66.3self 9.57.5self 9.56.3self 9.56.7self 9.53826.8fail0.5––2c2a74e8
v75.8self 9.87.8self 9.56.7self 9.56.7self 9.738.527fail2.5––c165de92
v85.5self 98self 106.2self 96.7self 103826.3fail1––d391e018
v96self 9.58self 106.8self 107.3self 1039.528.2pass2––e6d4548b
v105.3self 9.88self 9.56.3self 9.56.8self 9.738.526.5fail1––3e21bf05

Ready to stop guessing if your outputs are good?

Get Started Book a 15-minute walkthrough
Connect your agent to Jetty →