evals
Create your eval

01 — Author

Tell Jetty
the job to be done.

Everything the agents did came from one Markdown runbook: the objective, the constraints, a four-axis rubric and a self-critique loop. Here is how that runbook works, how the hill climb rewrote it each round, and the external judge we added on top.

A pelican, riding a bicycle, in SVG.

Hand-write a pure-XML SVG of a pelican riding a bicycle. Both subjects must be unmistakable, and the pelican has to be riding the bike (on the seat, feet at the pedals, wing tips on the bars) rather than floating next to it. The agent draws, renders the SVG to PNG with rsvg-convert, looks at the PNG, scores it and redraws, then keeps its best round.

SVG constraints

  • Pure SVG elements only: no <image> tags, no base64 or external rasters
  • Well-formed XML that renders in any modern browser
  • viewBox about 0 0 800 600, file size under 50 KB

The rubric

AxisQuestionScale
PelicanWould a stranger immediately say “that's a pelican”?0–10
BicycleTwo spoked wheels, a real frame, handlebars, seat, pedals?0–10
CompositionIs the pelican clearly riding the bike?0–10
PolishLine quality, colour, balance0–10

Total out of 40. The agent scores itself with this rubric; the external judge uses the same four axes.

The runbook

A runbook is one Markdown file that gives an agent its job, its bar for done and its checks. Jetty runs it in a sandbox and computes a verdict from the checks. This one has these parts:

  1. Objective: hand-write the best SVG of a pelican riding a bicycle, within the SVG constraints above.
  2. Parameters: the results directory, the number of draw-and-critique rounds (3) and the pass threshold (28/40).
  3. Steps: draw, render the SVG to PNG, look at it, score it on the four-axis rubric and redraw; then keep the best round.
  4. Code checks: the SVG parses, stays under 50 KB, embeds no raster images, and every round has its SVG and PNG.
  5. Checklist: the bird reads as a pelican, the bicycle is complete, the pelican is actually riding, and the scores agree.
  6. Validation report: validation_report.json in Jetty's evals-v2 shape, with the score as the judge check; Jetty computes the verdict.

Fork and run copies this runbook, unchanged, into your Jetty collection and launches its first run with the inputs you choose; Jetty asks for an OpenRouter key if your collection has none.

What the current version evaluates on every run

KindIdWhat it checks
code checkoutputs-existEvery required output file exists and is non-empty
code checksvg-parsesfinal.svg is well-formed XML with an <svg> root
code checksvg-size-under-50kbfinal.svg is under 50 KB
code checkno-image-tagsfinal.svg has no <image> elements or embedded rasters
code checkpng-renderedfinal.png is a real PNG at least 400 px wide
code checkrounds-recordedEvery scored round has its SVG and PNG
checklistpelican-recognizablethe bird reads as a pelican at a glance (long beak with a visible throat pouch)
checklistbicycle-completetwo spoked wheels, a connected frame, handlebars, seat and pedals are all present
checklistactually-ridingthe pelican sits on the seat with wing tips on the handlebars and feet at the pedals
checklistscores-match-reportreport.md, scores.json and final.svg agree on the per-round scores and the best round
judgeself-scoreThe agent's own 4-axis score of its best round, out of 40; passes at ≥ {{pass_threshold}} (default 28)

How the hill climb rewrites the runbook.

Each hill-climb round is one Jetty run. Between rounds a small orchestrator (hill_climb.py) rewrites the runbook: it embeds the previous round's best SVG as the round-1 seed, reads the per-round scores from report.md, and rewrites the description to target the lowest self-scored axis. v1 ran 10 rounds per agent (3 for the Fusion sweeps); v2 ran 5.

One lineage was edited by hand instead: Jon read each claude-code + Sonnet 4.6 trajectory and edited the runbook himself, eleven times. Three of those edits:

Big leap · v2 → v3 · self 32 → 34 · judge 25.2 → 24.5
Embed the baseline
v2
v2 · before
v3
v3 · after

The edit: Use the v2 SVG verbatim as the round-1 input. Don't start from scratch.

What happened: Round 1 jumped from 28 to 33 self-score points just by skipping the cold start. The single highest-leverage edit in the sequence.

Lesson: When a runbook can carry a working artifact as a seed, it should.

Peak · v4 → v5 · self 36 → 37 · judge 24.5 → 26.2
Coordinate-precise asks
v4
v4 · before
v5
v5 · after

The edit: Close the 13 px right-wing-to-grip gap. Drop the left-wing tips to y ≈ 230.

What happened: Composition hit 10/10 on the self-score for the first time. Sonnet's peak self-score, never beaten later.

Lesson: At the top of the curve, the asks become measurements.

Cautionary · v5 → v6 · self 37 → 36 · judge 26.2 → 26.5
Scope creep cost a point
v5
v5 · before
v6
v6 · after

The edit: Add motion lines, wind tufts and a sun. Extend the right-wing covert lines to the wrist.

What happened: A richer scene, but composition slipped back to 9. The new elements added clutter the rubric couldn't reward.

Lesson: More isn't better when the rubric doesn't measure scene density.

One fixed yardstick for every drawing.

Judge model
Opus 5.5
anthropic/claude-opus-5.5 via OpenRouter
Sampling
3 × T=0
independent calls, averaged
Renderer
librsvg
rsvg-convert -w 800, white background
Judge spend
$3.07
all 108 drawings

The system prompt, rubric and model are fixed across all drawings. Raw responses, including token usage, are published in judge-raw.json. A single LLM judge has biases of its own: it is one fixed yardstick, not ground truth, and the judge model is also one of the v2 contestants.

The image below is a rasterized SVG that a coding agent hand-wrote for the prompt:
"Generate an SVG of a pelican riding a bicycle."

Score it on four axes, each an integer or half-integer from 0 to 10:

1. pelican - Pelican recognizability. Would a stranger immediately say "that's a pelican"? Look for the long beak with a throat pouch, pelican body shape, eye, wings, legs/feet. 10 = unmistakably a pelican; 5 = a generic bird; 0 = no bird.
2. bicycle - Bicycle recognizability and correctness. Two wheels (with spokes), a plausible frame (top tube, down tube, seat tube), handlebars, seat, pedals/crank. 10 = a correct, well-formed bicycle; 5 = recognizable but structurally wrong; 0 = no bicycle.
3. composition - Is the pelican clearly RIDING the bicycle? Body on the seat, feet on or near the pedals, wing tips on the handlebars, parts connected rather than floating, sensible scale. 10 = convincingly riding; 5 = next to or loosely attached; 0 = unrelated.
4. polish - Aesthetic polish: line quality, color, balance, absence of rendering glitches, clutter or stray shapes. 10 = professional illustration quality; 5 = passable clip-art; 0 = broken.

Calibration: be strict. Reserve 9-10 for genuinely excellent work with no visible defects. Most competent attempts land between 5 and 8. Judge only what is visible in the image, not what was intended.

Reply with ONLY a JSON object, no prose outside it:
{"pelican": <0-10>, "bicycle": <0-10>, "composition": <0-10>, "polish": <0-10>, "notes": "<one or two sentences naming the main strengths and defects>"}

Why this prompt.

Simon Willison has run the pelican prompt against nearly every major model release since 2024. He picked it on purpose:

They shouldn't be able to draw anything at all. But they can generate code… and SVG is code.

Simon Willison, June 2025

Most importantly: pelicans can't ride bicycles. They're the wrong shape!

Simon Willison

Drawing a pelican on a bicycle in SVG makes a text model reason in code about shape, anatomy and composition all at once, and because pelicans can't ride bicycles the result can't be copied from training data. Simon's pelican posts ↗

Ready to stop guessing if your outputs are good?

Create your eval Book a 15-minute walkthrough
Connect your agent to Jetty →