--- title: "Runs: every pelican drawing | Pelican benchmark" url: https://evaljetty.com/pelicans/runs/ description: "All 98 pelican-on-a-bicycle runs on Jetty: agent and model, round, self-score, external judge score, derived verdict, cost and trajectory link." updated: 2026-09-28 publisher: Jetty (https://jetty.io) --- 02 — Runs · 98 runs · 14 lineages # Every run, with its evaluation. Each row is one Jetty trajectory: one hill-climb round, with three internal self-critique passes. v1 runs live in the jettyio collection, v2 runs in jettygrowthteam. [All runs](https://evaljetty.com/pelicans/runs/#all-runs) [Launch runs from the API](https://evaljetty.com/pelicans/runs/#launch) Runs 98 77 v1 · 21 v2 Derived pass rate 52/98 code checks pass and judge ≥ 28/40 Metered spend $9.12 10 claude-code runs; 11 v2 runs not metered Median time per run 5.5 min 32 runs with a recorded duration ## 14 lineages, *one gallery each.* [![](https://evaljetty.com/pelicans/img/v2-claude-opus55/v3.svg) v2 · 5 runs · judge 32.8 claude-code · Opus 5.5 anthropic/claude-opus-5.5](https://evaljetty.com/pelicans/runs/agents/v2-claude-opus55.html)[![](https://evaljetty.com/pelicans/img/v2-claude-sonnet5/v4.svg) v2 · 5 runs · judge 25.3 claude-code · Sonnet 5 anthropic/claude-sonnet-5](https://evaljetty.com/pelicans/runs/agents/v2-claude-sonnet5.html)[![](https://evaljetty.com/pelicans/img/v2-opencode-gpt56/v1.svg) v2 · 5 runs · judge 28 opencode · GPT-5.6 Terra openai/gpt-5.6-terra](https://evaljetty.com/pelicans/runs/agents/v2-opencode-gpt56.html)[![](https://evaljetty.com/pelicans/img/v2-opencode-gemini38/v1.svg) v2 · 1 run · judge 32.8 opencode · Gemini 3.8 Flash google/gemini-3.8-flash](https://evaljetty.com/pelicans/runs/agents/v2-opencode-gemini38.html)[![](https://evaljetty.com/pelicans/img/v2-hermes-jev/v4.svg) v2 · 5 runs · judge 31.7 hermes · Jev Router typesafe/jev-router](https://evaljetty.com/pelicans/runs/agents/v2-hermes-jev.html)[![](https://evaljetty.com/pelicans/img/gemini-cli/v1.svg) v1 · 10 runs · judge 29.8 gemini-cli · Gemini 3.5 Flash gemini-3.5-flash](https://evaljetty.com/pelicans/runs/agents/gemini-cli.html)[![](https://evaljetty.com/pelicans/img/opencode/v2.svg) v1 · 10 runs · judge 29.3 opencode · Gemini 3.5 Flash google/gemini-3.5-flash](https://evaljetty.com/pelicans/runs/agents/opencode.html)[![](https://evaljetty.com/pelicans/img/hermes-fusion/v1.svg) v1 · 3 runs · judge 28.8 hermes · Fusion router openrouter/fusion](https://evaljetty.com/pelicans/runs/agents/hermes-fusion.html)[![](https://evaljetty.com/pelicans/img/claude-opus/v3.svg) v1 · 10 runs · judge 28.7 claude-code · Opus 4.7 claude-opus-4-7](https://evaljetty.com/pelicans/runs/agents/claude-opus.html)[![](https://evaljetty.com/pelicans/img/opencode-fusion/v2.svg) v1 · 3 runs · judge 28.3 opencode · Fusion router openrouter/fusion](https://evaljetty.com/pelicans/runs/agents/opencode-fusion.html)[![](https://evaljetty.com/pelicans/img/hermes-flash/v9.svg) v1 · 10 runs · judge 28.2 hermes · Gemini 3.5 Flash openrouter/google/gemini-3.5-flash](https://evaljetty.com/pelicans/runs/agents/hermes-flash.html)[![](https://evaljetty.com/pelicans/img/hermes-glm51/v2.svg) v1 · 10 runs · judge 27.3 hermes · GLM 5.1 openrouter/z-ai/glm-5.1](https://evaljetty.com/pelicans/runs/agents/hermes-glm51.html)[![](https://evaljetty.com/pelicans/img/claude-sonnet/v5.svg) v1 · 11 runs · judge 26.2 claude-code · Sonnet 4.6 claude-sonnet-4-6](https://evaljetty.com/pelicans/runs/agents/claude-sonnet.html)[![](https://evaljetty.com/pelicans/img/hermes-sonnet/v5.svg) v1 · 10 runs · judge 24 hermes · Sonnet 4.6 openrouter/anthropic/claude-sonnet-4.6](https://evaljetty.com/pelicans/runs/agents/hermes-sonnet.html) ## Scores, verdicts, *cost and trajectory.* **About the verdict:** these runs used runbook v1, which wrote no validation report, so Jetty computed no verdict for them. The verdict here is *derived* the way the v2 runbook would score it, with the external judge in place of the self-score: **pass** when every replayable code check passes on the stored SVG (svg-parses, svg-size-under-50kb, no-image-tags, png-rendered) and the judge total is at least 28/40. New runs of the v2 runbook get Jetty's own verdict. Cost is Jetty's metered estimate; opencode and hermes runs reported no token usage, so they are marked not metered rather than zero. Lineageallclaude-code · Opus 5.5 (v2)claude-code · Sonnet 5 (v2)opencode · GPT-5.6 Terra (v2)opencode · Gemini 3.8 Flash (v2)hermes · Jev Router (v2)gemini-cli · Gemini 3.5 Flash (v1)opencode · Gemini 3.5 Flash (v1)hermes · Fusion router (v1)claude-code · Opus 4.7 (v1)opencode · Fusion router (v1)hermes · Gemini 3.5 Flash (v1)hermes · GLM 5.1 (v1)claude-code · Sonnet 4.6 (v1)hermes · Sonnet 4.6 (v1) Cohortallv2v1 Verdictallpassfail | Trajectory | Agent · model | Cohort | Round | Self /40 | Judge /40 | Verdict | Cost | Minutes | | --- | --- | --- | --- | --- | --- | --- | --- | --- | | 5361e30a | [claude-code · Opus 5.5](https://evaljetty.com/pelicans/runs/agents/v2-claude-opus55.html)anthropic/claude-opus-5.5 | v2 · Sep 2026 | v1 | 35 | **32.7** | pass | $0.87 | 4.3 | | ba932f3e | [claude-code · Opus 5.5](https://evaljetty.com/pelicans/runs/agents/v2-claude-opus55.html)anthropic/claude-opus-5.5 | v2 · Sep 2026 | v2 | 35 | **33** | pass | $0.88 | 4.6 | | 35c47f4e | [claude-code · Opus 5.5](https://evaljetty.com/pelicans/runs/agents/v2-claude-opus55.html)anthropic/claude-opus-5.5 | v2 · Sep 2026 | v3 agent's pick | 36 | **32.8** | pass | $0.83 | 4.8 | | 2e046a89 | [claude-code · Opus 5.5](https://evaljetty.com/pelicans/runs/agents/v2-claude-opus55.html)anthropic/claude-opus-5.5 | v2 · Sep 2026 | v4 | 36 | **33** | pass | $0.87 | 4.5 | | 0e41c0a0 | [claude-code · Opus 5.5](https://evaljetty.com/pelicans/runs/agents/v2-claude-opus55.html)anthropic/claude-opus-5.5 | v2 · Sep 2026 | v5 | 35 | **33.3** | pass | $1.10 | 6.4 | | ea416761 | [claude-code · Sonnet 5](https://evaljetty.com/pelicans/runs/agents/v2-claude-sonnet5.html)anthropic/claude-sonnet-5 | v2 · Sep 2026 | v1 | 28 | **26.5** | fail | $0.80 | 5.7 | | b77ea720 | [claude-code · Sonnet 5](https://evaljetty.com/pelicans/runs/agents/v2-claude-sonnet5.html)anthropic/claude-sonnet-5 | v2 · Sep 2026 | v2 | 34 | **27.3** | fail | $0.96 | 5.9 | | 6e98127a | [claude-code · Sonnet 5](https://evaljetty.com/pelicans/runs/agents/v2-claude-sonnet5.html)anthropic/claude-sonnet-5 | v2 · Sep 2026 | v3 | 34 | **27** | fail | $0.74 | 5.6 | | 240351f2 | [claude-code · Sonnet 5](https://evaljetty.com/pelicans/runs/agents/v2-claude-sonnet5.html)anthropic/claude-sonnet-5 | v2 · Sep 2026 | v4 agent's pick | 36 | **25.3** | fail | $0.81 | 6.4 | | 1925c665 | [claude-code · Sonnet 5](https://evaljetty.com/pelicans/runs/agents/v2-claude-sonnet5.html)anthropic/claude-sonnet-5 | v2 · Sep 2026 | v5 | 36 | **26.2** | fail | $1.26 | 11.8 | | ccbdd583 | [opencode · GPT-5.6 Terra](https://evaljetty.com/pelicans/runs/agents/v2-opencode-gpt56.html)openai/gpt-5.6-terra | v2 · Sep 2026 | v1 agent's pick | 38 | **28** | pass | not metered | 1.4 | | 699fefde | [opencode · GPT-5.6 Terra](https://evaljetty.com/pelicans/runs/agents/v2-opencode-gpt56.html)openai/gpt-5.6-terra | v2 · Sep 2026 | v2 | 37.2 | **27.5** | fail | not metered | 1.7 | | ce858eaf | [opencode · GPT-5.6 Terra](https://evaljetty.com/pelicans/runs/agents/v2-opencode-gpt56.html)openai/gpt-5.6-terra | v2 · Sep 2026 | v3 | 38 | **27.2** | fail | not metered | 1.5 | | 9fe62cbb | [opencode · GPT-5.6 Terra](https://evaljetty.com/pelicans/runs/agents/v2-opencode-gpt56.html)openai/gpt-5.6-terra | v2 · Sep 2026 | v4 | 37.5 | **27.3** | fail | not metered | 1.4 | | de112455 | [opencode · GPT-5.6 Terra](https://evaljetty.com/pelicans/runs/agents/v2-opencode-gpt56.html)openai/gpt-5.6-terra | v2 · Sep 2026 | v5 | 38 | **29.8** | pass | not metered | 1.5 | | 225e57cd | [opencode · Gemini 3.8 Flash](https://evaljetty.com/pelicans/runs/agents/v2-opencode-gemini38.html)google/gemini-3.8-flash | v2 · Sep 2026 | v1 agent's pick | 38.5 | **32.8** | pass | not metered | 11.7 | | 1a125d51 | [hermes · Jev Router](https://evaljetty.com/pelicans/runs/agents/v2-hermes-jev.html)typesafe/jev-router | v2 · Sep 2026 | v1 | 36 | **29.8** | pass | not metered | 2.2 | | 7a63ffb0 | [hermes · Jev Router](https://evaljetty.com/pelicans/runs/agents/v2-hermes-jev.html)typesafe/jev-router | v2 · Sep 2026 | v2 | 35 | **31.5** | pass | not metered | 1.6 | | ded99127 | [hermes · Jev Router](https://evaljetty.com/pelicans/runs/agents/v2-hermes-jev.html)typesafe/jev-router | v2 · Sep 2026 | v3 | 34.5 | **30.7** | pass | not metered | 2.6 | | 3ba61c00 | [hermes · Jev Router](https://evaljetty.com/pelicans/runs/agents/v2-hermes-jev.html)typesafe/jev-router | v2 · Sep 2026 | v4 agent's pick | 37 | **31.7** | pass | not metered | 2 | | ef67915d | [hermes · Jev Router](https://evaljetty.com/pelicans/runs/agents/v2-hermes-jev.html)typesafe/jev-router | v2 · Sep 2026 | v5 | 36 | **30.8** | pass | not metered | 3.1 | | f5cd6a31 | [gemini-cli · Gemini 3.5 Flash](https://evaljetty.com/pelicans/runs/agents/gemini-cli.html)gemini-3.5-flash | v1 · May 2026 | v1 agent's pick | 40 | **29.8** | pass | – | – | | c990e7c4 | [gemini-cli · Gemini 3.5 Flash](https://evaljetty.com/pelicans/runs/agents/gemini-cli.html)gemini-3.5-flash | v1 · May 2026 | v2 | 38.5 | **31** | pass | – | – | | 5d360721 | [gemini-cli · Gemini 3.5 Flash](https://evaljetty.com/pelicans/runs/agents/gemini-cli.html)gemini-3.5-flash | v1 · May 2026 | v3 | 38.4 | **29.3** | pass | – | – | | 4ed5de35 | [gemini-cli · Gemini 3.5 Flash](https://evaljetty.com/pelicans/runs/agents/gemini-cli.html)gemini-3.5-flash | v1 · May 2026 | v4 | 40 | **31.7** | pass | – | – | | a652e4e3 | [gemini-cli · Gemini 3.5 Flash](https://evaljetty.com/pelicans/runs/agents/gemini-cli.html)gemini-3.5-flash | v1 · May 2026 | v5 | 39.5 | **32.3** | pass | – | – | | eca90d64 | [gemini-cli · Gemini 3.5 Flash](https://evaljetty.com/pelicans/runs/agents/gemini-cli.html)gemini-3.5-flash | v1 · May 2026 | v6 | 39.5 | **30.5** | pass | – | – | | f08684e3 | [gemini-cli · Gemini 3.5 Flash](https://evaljetty.com/pelicans/runs/agents/gemini-cli.html)gemini-3.5-flash | v1 · May 2026 | v7 | 38.7 | **31** | pass | – | – | | ce2de30f | [gemini-cli · Gemini 3.5 Flash](https://evaljetty.com/pelicans/runs/agents/gemini-cli.html)gemini-3.5-flash | v1 · May 2026 | v8 | 40 | **31.7** | pass | – | – | | 4e122677 | [gemini-cli · Gemini 3.5 Flash](https://evaljetty.com/pelicans/runs/agents/gemini-cli.html)gemini-3.5-flash | v1 · May 2026 | v9 | 37.5 | **32** | pass | – | – | | 24091968 | [gemini-cli · Gemini 3.5 Flash](https://evaljetty.com/pelicans/runs/agents/gemini-cli.html)gemini-3.5-flash | v1 · May 2026 | v10 | 39.2 | **31.2** | pass | – | – | | 67836d67 | [opencode · Gemini 3.5 Flash](https://evaljetty.com/pelicans/runs/agents/opencode.html)google/gemini-3.5-flash | v1 · May 2026 | v1 | 39.5 | **28.2** | pass | – | – | | 660d2a66 | [opencode · Gemini 3.5 Flash](https://evaljetty.com/pelicans/runs/agents/opencode.html)google/gemini-3.5-flash | v1 · May 2026 | v2 agent's pick | 40 | **29.3** | pass | – | – | | 5cd01e9e | [opencode · Gemini 3.5 Flash](https://evaljetty.com/pelicans/runs/agents/opencode.html)google/gemini-3.5-flash | v1 · May 2026 | v3 | 39 | **28.7** | pass | – | – | | b130c9c2 | [opencode · Gemini 3.5 Flash](https://evaljetty.com/pelicans/runs/agents/opencode.html)google/gemini-3.5-flash | v1 · May 2026 | v4 | 40 | **29** | pass | – | – | | d94ff2e1 | [opencode · Gemini 3.5 Flash](https://evaljetty.com/pelicans/runs/agents/opencode.html)google/gemini-3.5-flash | v1 · May 2026 | v5 | 38.6 | **29** | pass | – | – | | 4a378be2 | [opencode · Gemini 3.5 Flash](https://evaljetty.com/pelicans/runs/agents/opencode.html)google/gemini-3.5-flash | v1 · May 2026 | v6 | 38.7 | **30.2** | pass | – | – | | e3059c87 | [opencode · Gemini 3.5 Flash](https://evaljetty.com/pelicans/runs/agents/opencode.html)google/gemini-3.5-flash | v1 · May 2026 | v7 | 40 | **30.2** | pass | – | – | | 70898b5c | [opencode · Gemini 3.5 Flash](https://evaljetty.com/pelicans/runs/agents/opencode.html)google/gemini-3.5-flash | v1 · May 2026 | v8 | 38 | **31.5** | pass | – | – | | b56c0c03 | [opencode · Gemini 3.5 Flash](https://evaljetty.com/pelicans/runs/agents/opencode.html)google/gemini-3.5-flash | v1 · May 2026 | v9 | 39 | **31.5** | pass | – | – | | 2a46de05 | [opencode · Gemini 3.5 Flash](https://evaljetty.com/pelicans/runs/agents/opencode.html)google/gemini-3.5-flash | v1 · May 2026 | v10 | 40 | **26.8** | fail | – | – | | 2d0c10c4 | [hermes · Fusion router](https://evaljetty.com/pelicans/runs/agents/hermes-fusion.html)openrouter/fusion | v1 · May 2026 | v1 agent's pick | 34 | **28.8** | pass | – | – | | b38f9569 | [hermes · Fusion router](https://evaljetty.com/pelicans/runs/agents/hermes-fusion.html)openrouter/fusion | v1 · May 2026 | v2 | 34 | **27.7** | fail | – | – | | cb138790 | [hermes · Fusion router](https://evaljetty.com/pelicans/runs/agents/hermes-fusion.html)openrouter/fusion | v1 · May 2026 | v3 | 34 | **29** | pass | – | – | | c3121188 | [claude-code · Opus 4.7](https://evaljetty.com/pelicans/runs/agents/claude-opus.html)claude-opus-4-7 | v1 · May 2026 | v1 | 33 | **29** | pass | – | – | | cb2a7f37 | [claude-code · Opus 4.7](https://evaljetty.com/pelicans/runs/agents/claude-opus.html)claude-opus-4-7 | v1 · May 2026 | v2 | 35 | **29.3** | pass | – | – | | d4a107dc | [claude-code · Opus 4.7](https://evaljetty.com/pelicans/runs/agents/claude-opus.html)claude-opus-4-7 | v1 · May 2026 | v3 agent's pick | 36 | **28.7** | pass | – | – | | 8ec2e90c | [claude-code · Opus 4.7](https://evaljetty.com/pelicans/runs/agents/claude-opus.html)claude-opus-4-7 | v1 · May 2026 | v4 | 32 | **29.7** | pass | – | – | | 4160dcf1 | [claude-code · Opus 4.7](https://evaljetty.com/pelicans/runs/agents/claude-opus.html)claude-opus-4-7 | v1 · May 2026 | v5 | 34.5 | **28** | pass | – | – | | 4848f8a3 | [claude-code · Opus 4.7](https://evaljetty.com/pelicans/runs/agents/claude-opus.html)claude-opus-4-7 | v1 · May 2026 | v6 | 36 | **27.3** | fail | – | – | | df18c780 | [claude-code · Opus 4.7](https://evaljetty.com/pelicans/runs/agents/claude-opus.html)claude-opus-4-7 | v1 · May 2026 | v7 | 33 | **27.5** | fail | – | – | | 20f46c4d | [claude-code · Opus 4.7](https://evaljetty.com/pelicans/runs/agents/claude-opus.html)claude-opus-4-7 | v1 · May 2026 | v8 | 33 | **27** | fail | – | – | | 4897cbca | [claude-code · Opus 4.7](https://evaljetty.com/pelicans/runs/agents/claude-opus.html)claude-opus-4-7 | v1 · May 2026 | v9 | 33.5 | **28** | pass | – | – | | 38634df1 | [claude-code · Opus 4.7](https://evaljetty.com/pelicans/runs/agents/claude-opus.html)claude-opus-4-7 | v1 · May 2026 | v10 | 34 | **27.7** | fail | – | – | | 40df25cb | [opencode · Fusion router](https://evaljetty.com/pelicans/runs/agents/opencode-fusion.html)openrouter/fusion | v1 · May 2026 | v1 | 34 | **29.3** | pass | – | – | | bfa3612b | [opencode · Fusion router](https://evaljetty.com/pelicans/runs/agents/opencode-fusion.html)openrouter/fusion | v1 · May 2026 | v2 agent's pick | 36 | **28.3** | pass | – | – | | 9776d557 | [opencode · Fusion router](https://evaljetty.com/pelicans/runs/agents/opencode-fusion.html)openrouter/fusion | v1 · May 2026 | v3 | 34 | **28** | pass | – | – | | c118fb80 | [hermes · Gemini 3.5 Flash](https://evaljetty.com/pelicans/runs/agents/hermes-flash.html)openrouter/google/gemini-3.5-flash | v1 · May 2026 | v1 | 39 | **28.5** | pass | – | – | | 2128ffc1 | [hermes · Gemini 3.5 Flash](https://evaljetty.com/pelicans/runs/agents/hermes-flash.html)openrouter/google/gemini-3.5-flash | v1 · May 2026 | v2 | 38 | **28.2** | pass | – | – | | 2405db86 | [hermes · Gemini 3.5 Flash](https://evaljetty.com/pelicans/runs/agents/hermes-flash.html)openrouter/google/gemini-3.5-flash | v1 · May 2026 | v3 | 37 | **26.5** | fail | – | – | | 822a3bca | [hermes · Gemini 3.5 Flash](https://evaljetty.com/pelicans/runs/agents/hermes-flash.html)openrouter/google/gemini-3.5-flash | v1 · May 2026 | v4 | 39 | **28.7** | pass | – | – | | 60502506 | [hermes · Gemini 3.5 Flash](https://evaljetty.com/pelicans/runs/agents/hermes-flash.html)openrouter/google/gemini-3.5-flash | v1 · May 2026 | v5 | 38.8 | **30** | pass | – | – | | 2c2a74e8 | [hermes · Gemini 3.5 Flash](https://evaljetty.com/pelicans/runs/agents/hermes-flash.html)openrouter/google/gemini-3.5-flash | v1 · May 2026 | v6 | 38 | **26.8** | fail | – | – | | c165de92 | [hermes · Gemini 3.5 Flash](https://evaljetty.com/pelicans/runs/agents/hermes-flash.html)openrouter/google/gemini-3.5-flash | v1 · May 2026 | v7 | 38.5 | **27** | fail | – | – | | d391e018 | [hermes · Gemini 3.5 Flash](https://evaljetty.com/pelicans/runs/agents/hermes-flash.html)openrouter/google/gemini-3.5-flash | v1 · May 2026 | v8 | 38 | **26.3** | fail | – | – | | e6d4548b | [hermes · Gemini 3.5 Flash](https://evaljetty.com/pelicans/runs/agents/hermes-flash.html)openrouter/google/gemini-3.5-flash | v1 · May 2026 | v9 agent's pick | 39.5 | **28.2** | pass | – | – | | 3e21bf05 | [hermes · Gemini 3.5 Flash](https://evaljetty.com/pelicans/runs/agents/hermes-flash.html)openrouter/google/gemini-3.5-flash | v1 · May 2026 | v10 | 38.5 | **26.5** | fail | – | – | | c1f2da04 | [hermes · GLM 5.1](https://evaljetty.com/pelicans/runs/agents/hermes-glm51.html)openrouter/z-ai/glm-5.1 | v1 · May 2026 | v1 | 32 | **28.7** | pass | – | – | | 4b324d80 | [hermes · GLM 5.1](https://evaljetty.com/pelicans/runs/agents/hermes-glm51.html)openrouter/z-ai/glm-5.1 | v1 · May 2026 | v2 agent's pick | 36.5 | **27.3** | fail | – | – | | f443b217 | [hermes · GLM 5.1](https://evaljetty.com/pelicans/runs/agents/hermes-glm51.html)openrouter/z-ai/glm-5.1 | v1 · May 2026 | v3 | 36 | **28.8** | pass | – | – | | 7ce1468b | [hermes · GLM 5.1](https://evaljetty.com/pelicans/runs/agents/hermes-glm51.html)openrouter/z-ai/glm-5.1 | v1 · May 2026 | v4 | 33 | **28.7** | pass | – | – | | ab8cb01d | [hermes · GLM 5.1](https://evaljetty.com/pelicans/runs/agents/hermes-glm51.html)openrouter/z-ai/glm-5.1 | v1 · May 2026 | v5 | 33 | **27.7** | fail | – | – | | 9fd19804 | [hermes · GLM 5.1](https://evaljetty.com/pelicans/runs/agents/hermes-glm51.html)openrouter/z-ai/glm-5.1 | v1 · May 2026 | v6 | 34.5 | **25.7** | fail | – | – | | 83e8efe2 | [hermes · GLM 5.1](https://evaljetty.com/pelicans/runs/agents/hermes-glm51.html)openrouter/z-ai/glm-5.1 | v1 · May 2026 | v7 | 36 | **25** | fail | – | – | | 21b9b8c1 | [hermes · GLM 5.1](https://evaljetty.com/pelicans/runs/agents/hermes-glm51.html)openrouter/z-ai/glm-5.1 | v1 · May 2026 | v8 | 34 | **24.7** | fail | – | – | | 9f9e50be | [hermes · GLM 5.1](https://evaljetty.com/pelicans/runs/agents/hermes-glm51.html)openrouter/z-ai/glm-5.1 | v1 · May 2026 | v9 | 33 | **24.2** | fail | – | – | | fff60669 | [hermes · GLM 5.1](https://evaljetty.com/pelicans/runs/agents/hermes-glm51.html)openrouter/z-ai/glm-5.1 | v1 · May 2026 | v10 | 36.5 | **25.7** | fail | – | – | | 7ddfcf08 | [claude-code · Sonnet 4.6](https://evaljetty.com/pelicans/runs/agents/claude-sonnet.html)claude-sonnet-4-6 | v1 · May 2026 | v1 | 31 | **26.5** | fail | – | 7 | | 414e644d | [claude-code · Sonnet 4.6](https://evaljetty.com/pelicans/runs/agents/claude-sonnet.html)claude-sonnet-4-6 | v1 · May 2026 | v2 | 32 | **25.2** | fail | – | 12.6 | | b169371d | [claude-code · Sonnet 4.6](https://evaljetty.com/pelicans/runs/agents/claude-sonnet.html)claude-sonnet-4-6 | v1 · May 2026 | v3 | 34 | **24.5** | fail | – | 10.3 | | 063dfed9 | [claude-code · Sonnet 4.6](https://evaljetty.com/pelicans/runs/agents/claude-sonnet.html)claude-sonnet-4-6 | v1 · May 2026 | v4 | 36 | **24.5** | fail | – | 7 | | 7ddc3c19 | [claude-code · Sonnet 4.6](https://evaljetty.com/pelicans/runs/agents/claude-sonnet.html)claude-sonnet-4-6 | v1 · May 2026 | v5 agent's pick | 37 | **26.2** | fail | – | 7.8 | | 39febb11 | [claude-code · Sonnet 4.6](https://evaljetty.com/pelicans/runs/agents/claude-sonnet.html)claude-sonnet-4-6 | v1 · May 2026 | v6 | 36 | **26.5** | fail | – | 5.4 | | ddf779bd | [claude-code · Sonnet 4.6](https://evaljetty.com/pelicans/runs/agents/claude-sonnet.html)claude-sonnet-4-6 | v1 · May 2026 | v7 | 34 | **27.7** | fail | – | 13.8 | | 79fdee62 | [claude-code · Sonnet 4.6](https://evaljetty.com/pelicans/runs/agents/claude-sonnet.html)claude-sonnet-4-6 | v1 · May 2026 | v8 | 34 | **26.3** | fail | – | 5.3 | | 5f980543 | [claude-code · Sonnet 4.6](https://evaljetty.com/pelicans/runs/agents/claude-sonnet.html)claude-sonnet-4-6 | v1 · May 2026 | v9 | 33 | **25.8** | fail | – | 7.3 | | 2043d7e3 | [claude-code · Sonnet 4.6](https://evaljetty.com/pelicans/runs/agents/claude-sonnet.html)claude-sonnet-4-6 | v1 · May 2026 | v10 | 35 | **28** | pass | – | 6.7 | | 3a4c3357 | [claude-code · Sonnet 4.6](https://evaljetty.com/pelicans/runs/agents/claude-sonnet.html)claude-sonnet-4-6 | v1 · May 2026 | v11 | 35 | **25.3** | fail | – | 9.5 | | 697b57d4 | [hermes · Sonnet 4.6](https://evaljetty.com/pelicans/runs/agents/hermes-sonnet.html)openrouter/anthropic/claude-sonnet-4.6 | v1 · May 2026 | v1 | 32 | **26.5** | fail | – | – | | acb2ba11 | [hermes · Sonnet 4.6](https://evaljetty.com/pelicans/runs/agents/hermes-sonnet.html)openrouter/anthropic/claude-sonnet-4.6 | v1 · May 2026 | v2 | 34 | **27** | fail | – | – | | 99a389e4 | [hermes · Sonnet 4.6](https://evaljetty.com/pelicans/runs/agents/hermes-sonnet.html)openrouter/anthropic/claude-sonnet-4.6 | v1 · May 2026 | v3 | 34 | **24** | fail | – | – | | 28f94566 | [hermes · Sonnet 4.6](https://evaljetty.com/pelicans/runs/agents/hermes-sonnet.html)openrouter/anthropic/claude-sonnet-4.6 | v1 · May 2026 | v4 | 34 | **25** | fail | – | – | | 8969a1a0 | [hermes · Sonnet 4.6](https://evaljetty.com/pelicans/runs/agents/hermes-sonnet.html)openrouter/anthropic/claude-sonnet-4.6 | v1 · May 2026 | v5 agent's pick | 36 | **24** | fail | – | – | | ebc95c4b | [hermes · Sonnet 4.6](https://evaljetty.com/pelicans/runs/agents/hermes-sonnet.html)openrouter/anthropic/claude-sonnet-4.6 | v1 · May 2026 | v6 | 36 | **22.2** | fail | – | – | | fdf47c73 | [hermes · Sonnet 4.6](https://evaljetty.com/pelicans/runs/agents/hermes-sonnet.html)openrouter/anthropic/claude-sonnet-4.6 | v1 · May 2026 | v7 | 36 | **19.2** | fail | – | – | | 25323cb8 | [hermes · Sonnet 4.6](https://evaljetty.com/pelicans/runs/agents/hermes-sonnet.html)openrouter/anthropic/claude-sonnet-4.6 | v1 · May 2026 | v8 | 36 | **18** | fail | – | – | | 5b9cf6ae | [hermes · Sonnet 4.6](https://evaljetty.com/pelicans/runs/agents/hermes-sonnet.html)openrouter/anthropic/claude-sonnet-4.6 | v1 · May 2026 | v9 | 36 | **18.7** | fail | – | – | | 34f10b3e | [hermes · Sonnet 4.6](https://evaljetty.com/pelicans/runs/agents/hermes-sonnet.html)openrouter/anthropic/claude-sonnet-4.6 | v1 · May 2026 | v10 | 36 | **19.8** | fail | – | – | ## Launch runs *from the API.* Once the task exists in your collection (see [Author · run it on your Jetty](https://evaljetty.com/pelicans/author.html#run-it)), a run is one multipart POST. Top-level keys you pass replace the task defaults; keys you omit (snapshot, file\_paths, runbook\_evals\_version) keep the task's values. 1. ### Run it ``` # Launch a run. Anything in init_params overrides the task defaults (top-level keys are replaced) curl -s -X POST "https://flows-api.jetty.io/api/v1/run/$COLLECTION/pelican-bicycle-svg" \ -H "Authorization: Bearer $JETTY_TOKEN" \ -F 'init_params={"agent": "claude-code", "model": "anthropic/claude-sonnet-5", "model_provider": "openrouter", "vars": {"prompt": "Execute the runbook end-to-end. You are headless: never stop to ask questions.", "results_dir": "/app/results", "max_rounds": 3, "pass_threshold": 28}}' # => {"trajectory_id": "…", "workflow_id": "…"} # Other agent/model pairs we ran (all model_provider=openrouter): # claude-code anthropic/claude-opus-5.5 # opencode openrouter/google/gemini-3.8-flash # opencode openrouter/openai/gpt-5.6-terra # hermes openrouter/typesafe/jev-router ``` 2. ### Read the verdict and collect the drawing ``` # Poll until status is completed, then read Jetty's evaluation and download the outputs curl -s "https://flows-api.jetty.io/api/v1/db/trajectory/$COLLECTION/pelican-bicycle-svg/$TRAJ" \ -H "Authorization: Bearer $JETTY_TOKEN" | jq '.status, .attributes.evaluation.verdict, .steps.run.outputs.usage' curl -s -o pelican.zip "https://flows-api.jetty.io/api/v1/trajectory/$COLLECTION/pelican-bicycle-svg/$TRAJ/download" \ -H "Authorization: Bearer $JETTY_TOKEN" # final.svg, final.png, scores.json, report.md, validation_report.json, rounds/ # or open https://jetty.io/$COLLECTION/pelican-bicycle-svg/$TRAJ ``` --- Source: https://evaljetty.com/pelicans/runs/ · Evals by Jetty · Run your own evals: https://jetty.io/?utm_source=evaljetty&utm_medium=llms