Tell Jetty
the job to be done.

---
name: pelican-bicycle-svg
version: "2.0.0"
evals_version: 2
description: Hand-write the best possible SVG of a pelican riding a bicycle, refining it over several self-critique rounds, and write an evals-v2 validation report.
agent: claude-code
model: anthropic/claude-sonnet-5
model_provider: openrouter
snapshot: python312-uv
evaluation: programmatic
primary_outputs:
  - final.svg
  - final.png
  - report.md
secrets:
  OPENROUTER_API_KEY:
    env: OPENROUTER_API_KEY
    description: "OpenRouter key (collection environment variable) used by the agent's model calls"
    required: true
---

Pelican Riding a Bicycle: SVG Runbook

Objective

Produce the highest-quality hand-written SVG of a pelican riding a bicycle and deliver it, a PNG render, the per-round scores and an evals-v2 validation report to {{results_dir}}.

Both subjects must be unmistakable, and the composition must be coherent: the pelican is riding the bicycle (on the seat, feet at the pedals, wing tips on the handlebars), not floating next to it. You draw, render the SVG to PNG, look at the PNG, score it on a four-axis rubric and redraw, for up to {{max_rounds}} rounds, then keep the best round.

Hard constraints on every SVG you write:

• Pure SVG XML: <path>, <circle>, <ellipse>, <rect>, <polygon>, <line>, <g>, <defs>, gradients and so on
• No <image> elements and no embedded rasters (data:image/..., base64, external bitmaps)
• Well-formed XML that renders in any modern browser
• viewBox approximately 0 0 800 600 (landscape)
• File size under 50 KB

REQUIRED OUTPUT FILES

• {{results_dir}}/final.svg: the best round's SVG
• {{results_dir}}/final.png: final.svg rendered with rsvg-convert -w 800
• {{results_dir}}/rounds/v<N>.svg and rounds/v<N>.png: every round you attempted
• {{results_dir}}/scores.json: your rubric scores per round and the best round (format in step 4)
• {{results_dir}}/report.md: per-round score table and notes (format in step 4)
• {{results_dir}}/validation_report.json: evals-v2 report, written in the last section

Parameters

Dependencies

• python312-uv snapshot (Python 3.12 for the checks and the report)
• An SVG rasterizer: rsvg-convert from librsvg2-bin (installed in step 1 if missing)
• OPENROUTER_API_KEY collection environment variable for the agent's own model calls

Steps

1. Setup

mkdir -p {{results_dir}}/rounds && cd {{results_dir}}
which rsvg-convert || (apt-get update && apt-get install -y librsvg2-bin) || true
which rsvg-convert

2. Round 1: first draft

Write your best first attempt to {{results_dir}}/rounds/v1.svg. Aim to depict:

• Pelican: long beak with throat pouch (the iconic silhouette), body, eye, wings, legs and feet
• Bicycle: two wheels with spokes, frame (top tube, down tube, seat tube), handlebars, seat, pedals
• Riding: body on the seat, feet on or near the pedals, wing tips on the handlebars

Render it:

cd {{results_dir}} && rsvg-convert -w 800 rounds/v1.svg -o rounds/v1.png

3. Critique and refine

For each round R from 1 to {{max_rounds}}:

1. Look at rounds/v${R}.png (open it as an image; do not score from the SVG source).
2. Score it on four axes, 0 to 10 each:
   Pelican: would a stranger immediately say "that's a pelican"?
   Bicycle: would they say "that's a bicycle"?
   Composition: is the pelican clearly riding the bike?
   Polish: line quality, colour, balance
3. If the total is 36/40 or more, or R equals {{max_rounds}}, stop the loop.
4. Otherwise name the lowest-scoring axis, write down two or three concrete fixes, write rounds/v$((R+1)).svg applying them (keep what worked) and render rounds/v$((R+1)).png.

Score honestly: the benchmark re-scores every drawing with an external vision judge, and agents typically rate themselves several points above it. Score what the PNG shows, not what you intended to draw.

4. Finalize

Pick the round with the highest total (break ties in favour of the later round) and copy it:

cd {{results_dir}} && cp rounds/vBEST.svg final.svg && cp rounds/vBEST.png final.png

Write {{results_dir}}/scores.json:

{"best_round": 2, "rounds": [
  {"round": 1, "pelican": 7, "bicycle": 8, "composition": 6, "polish": 7, "total": 28},
  {"round": 2, "pelican": 8, "bicycle": 8, "composition": 7, "polish": 8, "total": 31}]}

Write {{results_dir}}/report.md with a ## Per-round scores table (Round, Pelican, Bicycle, Composition, Polish, Total), the best round, and short What worked, What didn't work and SVG technique notes sections (viewBox, shape count, file size).

Code Checks

Each check is a deterministic command; a failing check fails the run. The report script in the last section runs the same checks, so run these for your own verification first.

outputs-exist — Every required output file exists and is non-empty

cd {{results_dir}} && for f in final.svg final.png scores.json report.md; do test -s "$f" || { echo "missing $f"; exit 1; }; done

svg-parses — final.svg is well-formed XML with an <svg> root

python3 -c "import xml.etree.ElementTree as E; r=E.parse('{{results_dir}}/final.svg').getroot(); assert r.tag.split('}')[-1]=='svg', r.tag"

svg-size-under-50kb — final.svg is under 50 KB

test "$(wc -c < {{results_dir}}/final.svg)" -lt 51200

no-image-tags — final.svg has no <image> elements or embedded rasters

python3 -c "import xml.etree.ElementTree as E; s=open('{{results_dir}}/final.svg').read(); r=E.fromstring(s); assert not [e for e in r.iter() if e.tag.split('}')[-1]=='image'] and 'data:image' not in s and 'base64' not in s"

png-rendered — final.png is a real PNG at least 400 px wide

python3 -c "import struct; b=open('{{results_dir}}/final.png','rb').read(24); assert b[:8]==b'\x89PNG\r\n\x1a\n', 'not a PNG'; w=struct.unpack('>I', b[16:20])[0]; assert w>=400, w"

rounds-recorded — Every scored round has its SVG and PNG

python3 -c "import json,os; d=json.load(open('{{results_dir}}/scores.json')); ns=[r['round'] for r in d['rounds']]; assert ns and d['best_round'] in ns; miss=[n for n in ns for e in ('svg','png') if not os.path.getsize('{{results_dir}}/rounds/v%d.%s'%(n,e))]; assert not miss, miss"

Checklist

Confirm each item by looking at final.png; a failed item fails the run.

• pelican-recognizable: the bird reads as a pelican at a glance (long beak with a visible throat pouch)
• bicycle-complete: two spoked wheels, a connected frame, handlebars, seat and pedals are all present
• actually-riding: the pelican sits on the seat with wing tips on the handlebars and feet at the pedals
• scores-match-report: report.md, scores.json and final.svg agree on the per-round scores and the best round

Write Validation Report

Write {{results_dir}}/validation_report.json in the evals-v2 shape by running the script below. It runs every code check above, records one step entry per step, one checklist entry per item you pass in (status, a colon, then a one-line reason) and your best-round self-score as a judge check (score out of 40, threshold {{pass_threshold}}). Jetty computes the verdict from these checks. Use fail: <reason> for anything that did not hold, and set ITERATIONS to the number of rounds you drew.

cd {{results_dir}} && ITERATIONS=3 \
STEPS='{"setup": "pass", "draft": "pass", "refine": "pass", "finalize": "pass"}' \
CHECKLIST='{"pelican-recognizable": "pass: <why>", "bicycle-complete": "pass: <why>", "actually-riding": "pass: <why>", "scores-match-report": "pass: <why>"}' \
python3 - <<'PY'
import json, os, struct, xml.etree.ElementTree as E
R = "{{results_dir}}"; T = "{{pass_threshold}}"; T = float(T) if T.replace(".", "", 1).isdigit() else 28.0
checks = []
def add(kind, cid, name, ok, msg, **kw):
    checks.append({"kind": kind, "id": cid, "name": name, "status": "pass" if ok else "fail", "message": str(msg)[:500], **kw})
def run(cid, name, fn):
    try:
        add("code_check", cid, name, True, fn() or "ok")
    except Exception as e:
        add("code_check", cid, name, False, repr(e))
p = lambda f: os.path.join(R, f)
def outputs():
    miss = [f for f in ("final.svg", "final.png", "scores.json", "report.md") if not os.path.exists(p(f)) or not os.path.getsize(p(f))]
    assert not miss, "missing: " + ", ".join(miss)
def parses():
    r = E.parse(p("final.svg")).getroot(); assert r.tag.split("}")[-1] == "svg", r.tag
def size():
    n = os.path.getsize(p("final.svg")); assert n < 51200, n; return f"{n} bytes"
def noimg():
    s = open(p("final.svg")).read(); r = E.fromstring(s)
    assert not [e for e in r.iter() if e.tag.split("}")[-1] == "image"] and "data:image" not in s and "base64" not in s
def png():
    b = open(p("final.png"), "rb").read(24); assert b[:8] == b"\x89PNG\r\n\x1a\n", "not a PNG"
    w = struct.unpack(">I", b[16:20])[0]; assert w >= 400, w; return f"{w} px wide"
def rounds():
    d = json.load(open(p("scores.json"))); ns = [r["round"] for r in d["rounds"]]
    assert ns and d["best_round"] in ns, "best_round not among rounds"
    miss = [f"v{n}.{e}" for n in ns for e in ("svg", "png") if not os.path.exists(p(f"rounds/v{n}.{e}"))]
    assert not miss, miss; return f"{len(ns)} rounds"
run("outputs-exist", "Every required output file exists and is non-empty", outputs)
run("svg-parses", "final.svg is well-formed XML with an <svg> root", parses)
run("svg-size-under-50kb", "final.svg is under 50 KB", size)
run("no-image-tags", "final.svg has no <image> elements or embedded rasters", noimg)
run("png-rendered", "final.png is a real PNG at least 400 px wide", png)
run("rounds-recorded", "Every scored round has its SVG and PNG", rounds)
for sid, st in json.loads(os.environ["STEPS"]).items():
    add("step", sid, sid, st.startswith("pass"), st)
for cid, st in json.loads(os.environ["CHECKLIST"]).items():
    ok = st.strip().lower().startswith("pass"); add("checklist", cid, cid, ok, st.split(":", 1)[-1].strip())
try:
    d = json.load(open(p("scores.json"))); best = next(r for r in d["rounds"] if r["round"] == d["best_round"])
    add("judge", "self-score", "Agent self-score of the best round (4 axes x 0-10)", best["total"] >= T,
        f"round {d['best_round']}: " + ", ".join(f"{k} {best[k]}" for k in ("pelican", "bicycle", "composition", "polish")),
        score=float(best["total"]), max_score=40.0, threshold=T)
except Exception as e:
    checks.append({"kind": "judge", "id": "self-score", "name": "Agent self-score of the best round", "status": "error", "message": repr(e)})
gating = [c for c in checks if c["kind"] != "step"]
passed = all(c["status"] == "pass" for c in gating)
rep = {"version": 2, "verdict": "pass" if passed else "fail", "overall_passed": passed,
       "iterations": int(os.environ.get("ITERATIONS", "1")), "checks": checks}
json.dump(rep, open(p("validation_report.json"), "w"), indent=2)
for c in checks: print(f"{c['status']:5} {c['kind']:10} {c['id']}: {c['message']}")
print("verdict:", rep["verdict"])
PY

Do not hand-edit the file and do not invent a different format: it must stay {"version": 2, "verdict", "overall_passed", "iterations", "checks": [...]}, with each check carrying kind (step, code_check, checklist or judge), id, name, status (pass, fail, skipped or error) and message, plus score, max_score and threshold on the judge.

Tips

• One round costs roughly $0.50 to $2.50 depending on the model; most of it is the agent's own reasoning.
• Seeding round 1 with a previous run's final.svg (paste it into the runbook) is the single biggest improvement we measured in hand-edited climbs: it skips the cold start.
• Coordinate-precise fixes ("close the 13 px gap between the right wing tip and the grip") move scores more than scene-level additions (sun, motion lines, scenery), which tend to add clutter.
• The self-score is a weak signal across models: at evaljetty.com/pelicans it correlates only loosely with an external judge. Treat it as a steering aid for your own climb, not as a ranking.

Changelog

• 2.0.0: Evals v2 shape (Parameters table, Code Checks, Checklist, Write Validation Report with the self-score as a judge check at {{pass_threshold}}/40); scores.json output; honest-scoring note.
• 1.x: The runbook used for every run published at evaljetty.com/pelicans (v1 cohort May 2026, v2 cohort Sep 2026): draw, render, self-score and redraw for max_rounds, then final.svg, final.png and report.md.

---
name: pelican-bicycle-svg
version: 2
description: Generate the best possible hand-written SVG depicting a pelican riding a bicycle, with iterative self-refinement.
agent: claude-code
model: anthropic/claude-sonnet-5
model_provider: openrouter
parameters:
  max_rounds:
    type: integer
    default: 3
    description: Number of refinement rounds (each round = critique previous + write improved)
  results_dir:
    type: string
    default: /app/results
secrets:
  openrouter:
    env: OPENROUTER_API_KEY
    description: OpenRouter key (collection environment variable) used by the agent's model calls
evaluation:
  pattern: self-judge
---

mkdir -p {{results_dir}}/rounds
cd {{results_dir}}

which rsvg-convert || which inkscape || which convert

apt-get update && apt-get install -y librsvg2-bin || true

cd {{results_dir}}
rsvg-convert -w 800 rounds/v1.svg -o rounds/v1.png

cp {{results_dir}}/rounds/vBEST.svg {{results_dir}}/final.svg
cp {{results_dir}}/rounds/vBEST.png {{results_dir}}/final.png

# Pelican-Bicycle SVG Report

## Per-round scores
| Round | Pelican | Bicycle | Composition | Polish | Total |
|-------|---------|---------|-------------|--------|-------|
| 1 | ? | ? | ? | ? | ? |
| 2 | ? | ? | ? | ? | ? |
| 3 | ? | ? | ? | ? | ? |

## Best round: vN (Total: X/40)

## What worked
- ...

## What didn't work
- ...

## SVG technique notes
- viewBox used:
- Total path / shape count:
- File size: