evals
Run evals on Jetty
Run it yourself · one runbook · one API key

Draw your own pelican on Jetty

Everything on this site came from one runbook and a small orchestrator. Here's all of it. You need a Jetty collection and an OpenRouter key; a single round costs roughly $0.50 to $2.50 depending on the model.

Download the runbook (.md) Task workflow JSON Get a Jetty account

  1. The runbook

    Plain markdown with YAML frontmatter. It tells the agent to draw, render with rsvg-convert, look at the PNG, score it on four axes and redraw, three times, then write final.svg, final.png and report.md.

    ---
name: pelican-bicycle-svg
version: 2
description: Generate the best possible hand-written SVG depicting a pelican riding a bicycle, with iterative self-refinement.
agent: claude-code
model: anthropic/claude-sonnet-5
model_provider: openrouter
parameters:
  max_rounds:
    type: integer
    default: 3
    description: Number of refinement rounds (each round = critique previous + write improved)
  results_dir:
    type: string
    default: /app/results
secrets:
  openrouter:
    env: OPENROUTER_API_KEY
    description: OpenRouter key (collection environment variable) used by the agent's model calls
evaluation:
  pattern: self-judge
---

# Pelican Riding a Bicycle — SVG Runbook

## Mission

Produce the highest-quality hand-written SVG depicting **a pelican riding a bicycle**. Both subjects must be unmistakably recognizable, the composition coherent (pelican IS interacting with the bicycle, not floating next to it), and the file should be pure XML SVG (no `<image>` tags, no base64 data, no external rasters).

## Hard constraints

- Pure SVG XML — use `<path>`, `<circle>`, `<ellipse>`, `<rect>`, `<polygon>`, `<line>`, `<g>`, `<defs>`, `<linearGradient>`, etc.
- **No** `<image>` elements with external or base64 data
- Must validate as well-formed XML and render in any modern browser
- viewBox approximately `0 0 800 600` (landscape) — adjust if you have a specific reason
- Total file size under 50KB

## INPUTS

- `max_rounds` = **3** (override only if explicitly told otherwise)
- `results_dir` = `/app/results`

## Steps

### 1. Setup

```bash
mkdir -p {{results_dir}}/rounds
cd {{results_dir}}
```

Check that `rsvg-convert` or an SVG rasterizer is available:

```bash
which rsvg-convert || which inkscape || which convert
```

If none are available, install one:

```bash
apt-get update && apt-get install -y librsvg2-bin || true
```

### 2. Round 1 — first draft

Write your best first attempt to `{{results_dir}}/rounds/v1.svg`. Aim to depict:

- **Pelican**: long beak with throat pouch (the iconic pelican silhouette), body, eye, wings, legs/feet
- **Bicycle**: two wheels with spokes, frame (top tube + down tube + seat tube), handlebars, seat, pedals
- **Riding**: pelican's body is on the seat, feet on or near pedals, "hands" (wing tips) on handlebars

Render it:

```bash
cd {{results_dir}}
rsvg-convert -w 800 rounds/v1.svg -o rounds/v1.png
```

### 3. Self-critique + refinement loop

For each round R from 1 to `max_rounds - 1`:

1. **Inspect** `rounds/v${R}.png` visually (read it as an image).
2. **Score** on four axes, 0–10 each:
   - **Pelican recognizability** — would a stranger immediately say "that's a pelican"?
   - **Bicycle recognizability** — would they say "that's a bicycle"?
   - **Composition / riding** — is the pelican clearly riding the bike?
   - **Aesthetic polish** — line quality, color, balance
3. **Identify the lowest-scoring axis** and write down 2-3 concrete fixes.
4. **Write** `rounds/v$((R+1)).svg` applying those fixes. Keep what worked; rewrite what didn't.
5. **Render** `rounds/v$((R+1)).png`.
6. If the total (sum of 4 axes) is ≥ 36/40 — early-exit the loop.

### 4. Finalize

After the loop, pick the round with the highest total score (break ties by preferring later rounds since they had more iteration).

```bash
cp {{results_dir}}/rounds/vBEST.svg {{results_dir}}/final.svg
cp {{results_dir}}/rounds/vBEST.png {{results_dir}}/final.png
```

Write `{{results_dir}}/report.md` containing:

```markdown
# Pelican-Bicycle SVG Report

## Per-round scores
| Round | Pelican | Bicycle | Composition | Polish | Total |
|-------|---------|---------|-------------|--------|-------|
| 1 | ? | ? | ? | ? | ? |
| 2 | ? | ? | ? | ? | ? |
| 3 | ? | ? | ? | ? | ? |

## Best round: vN (Total: X/40)

## What worked
- ...

## What didn't work
- ...

## SVG technique notes
- viewBox used:
- Total path / shape count:
- File size:
```

## Final checklist

- [ ] `{{results_dir}}/final.svg` exists, is valid XML, renders in a browser, has no `<image>` tags
- [ ] `{{results_dir}}/final.png` exists (rasterized version)
- [ ] `{{results_dir}}/rounds/v*.svg` and `v*.png` exist for every round attempted
- [ ] `{{results_dir}}/report.md` exists with per-round score table and a notes section
- [ ] File size of `final.svg` is under 50KB

  2. The Jetty task

    A single runbook step. Agent, model, provider, snapshot, instruction and template variables are all read from init_params, so one task can run any agent/model pair. The runbook itself is inlined as instruction (elided here).

    {
  "init_params": {
    "agent": "claude-code",
    "model": "anthropic/claude-sonnet-5",
    "model_provider": "openrouter",
    "snapshot": "python312-uv",
    "instruction": "<contents of pelican-bicycle-svg.runbook.md>",
    "vars": {
      "prompt": "Execute the runbook end-to-end.",
      "results_dir": "/app/results",
      "max_rounds": 3
    },
    "file_paths": []
  },
  "steps": [
    "run"
  ],
  "step_configs": {
    "run": {
      "activity": "runbook",
      "agent_path": "init_params.agent",
      "model_path": "init_params.model",
      "model_provider_path": "init_params.model_provider",
      "snapshot_path": "init_params.snapshot",
      "instruction_path": "init_params.instruction",
      "template_variables_path": "init_params.vars",
      "files_path": "init_params.file_paths",
      "cpus": 4,
      "memory": "8G",
      "timeout_sec": 1800,
      "network_enabled": true
    }
  }
}

  3. Set the key and create the task

    The collection needs one secret, OPENROUTER_API_KEY, stored as a collection environment variable. Secrets never go in init_params.

    export JETTY_TOKEN=mlc_...            # your Jetty API key
export COLLECTION=your-collection   # a collection you own

# 1. Store your OpenRouter key as a collection environment variable (sent via stdin, not argv)
printf '{"environment_variables": {"OPENROUTER_API_KEY": "%s"}}' "$OPENROUTER_API_KEY" | \
  curl -s -X PATCH "https://flows-api.jetty.io/api/v1/collections/$COLLECTION/environment" \
    -H "Authorization: Bearer $JETTY_TOKEN" -H "Content-Type: application/json" --data-binary @-

# 2. Create the task from the workflow JSON (the runbook is embedded as init_params.instruction)
curl -sO https://evaljetty.com/pelicans/run/task-workflow.json
jq '{name: "pelican-bicycle-svg", description: "Pelican riding a bicycle, SVG", workflow: .}' task-workflow.json | \
  curl -s -X POST "https://flows-api.jetty.io/api/v1/tasks/$COLLECTION" \
    -H "Authorization: Bearer $JETTY_TOKEN" -H "Content-Type: application/json" --data-binary @-

  4. Run it

    Launch with a multipart init_params field. Top-level keys you pass replace the task defaults; keys you omit (snapshot, file_paths) keep the task's values.

    # 3. Run it. Anything in init_params overrides the task defaults (top-level keys are merged)
curl -s -X POST "https://flows-api.jetty.io/api/v1/run/$COLLECTION/pelican-bicycle-svg" \
  -H "Authorization: Bearer $JETTY_TOKEN" \
  -F 'init_params={"agent": "claude-code", "model": "anthropic/claude-sonnet-5", "model_provider": "openrouter",
                   "vars": {"prompt": "Execute the runbook end-to-end. You are headless: never stop to ask questions.",
                            "results_dir": "/app/results", "max_rounds": 3}}'
# => {"trajectory_id": "…", "workflow_id": "…"}

# Other agent/model pairs we ran (all model_provider=openrouter):
#   claude-code  anthropic/claude-opus-5.5
#   opencode     openrouter/google/gemini-3.8-flash
#   opencode     openrouter/openai/gpt-5.6-terra
#   hermes       openrouter/typesafe/jev-router

  5. Collect the drawing

    # 4. Poll until status is completed, then download everything (final.svg, final.png, report.md, rounds/)
curl -s "https://flows-api.jetty.io/api/v1/db/trajectory/$COLLECTION/pelican-bicycle-svg/$TRAJ" \
  -H "Authorization: Bearer $JETTY_TOKEN" | jq '.status, .steps.run.outputs.usage'
curl -s -o pelican.zip "https://flows-api.jetty.io/api/v1/trajectory/$COLLECTION/pelican-bicycle-svg/$TRAJ/download" \
  -H "Authorization: Bearer $JETTY_TOKEN"
# or open https://jetty.io/$COLLECTION/pelican-bicycle-svg/$TRAJ

  6. Hill-climb it

    The orchestrator we used for v2: each round embeds the previous best SVG into the runbook and targets the weakest self-scored axis. It stops launching runs past a spend budget.

    # 5. Optional: hill-climb (5 rounds per agent, agents in parallel, $60 spend guard)
curl -sO https://evaljetty.com/pelicans/run/hill_climb.py
curl -so runbook.md https://evaljetty.com/pelicans/run/pelican-bicycle-svg.runbook.md
# edit COLLECTION at the top of hill_climb.py, then:
JETTY_TOKEN=$JETTY_TOKEN python3 hill_climb.py --agent all --rounds 5 --budget 60

Want this kind of eval for your own agents?

Every run on this site is a Jetty runbook: versioned instructions, a pinned sandbox, and a trajectory you can inspect. Point Jetty at your task and get the same receipts.

Get started with Jetty, free Book a 15-minute walkthrough