Web

WindTunnel-style

50 tasks, v2, graded by the benchmark's own reference answers and verifier.

uploaded by the publisher of this indextask text withheldlicence unverified

In the leaderboard

Every setup scored on WindTunnel, whichever other benchmarks it covers. The leaderboard compares a setup only with setups scored on exactly the same benchmarks, so its bands can differ from these.

rank on this benchmark	Setup	Score	Pass rate	cost · tokens · time	median spend of a passing run
1–17	Claude Agent SDK · claude-sonnet-5-5harness default, open-book on knowledge questions	84.7 pts84.1–85.1 pts	100.0%≥ 92.9%100.0% · 50 of 50 graded	1.1× · 1.0× · 0.36×	$0.07 · 161.6k · 11.7scache reads not recorded
1–17	Claude Code · claude-haiku-4-5-20251001with my setupWebMCP	81.1 pts79.9–82.6 pts	100.0%≥ 92.7%100.0% · 49 of 50 graded	0.44× · 0.93× · 0.90×	$0.03 · 176.9k · 32.6s97% cache reads
1–17	Codex · gpt-5.4-miniwith my setupWebMCP	80.8 pts76.9–84.3 pts	98.0%93.8–100.0%98.0% · 49 of 50 graded	0.35× · 0.42× · 0.87×	$0.02 · 70.8k · 32.7s87% cache reads
1–17	Claude Code · claude-haiku-4-5-20251001with my setupPlaywright	79.3 pts78.5–79.8 pts	100.0%≥ 92.6%100.0% · 48 of 50 graded	0.79× · 1.2× · 1.1×	$0.05 · 283.1k · 44.8s96% cache reads
1–17	opencode · claude-sonnet-5-5harness default, open-book on knowledge questions	79.0 pts75.7–81.3 pts	98.0%93.8–100.0%98.0% · 49 of 50 graded	0.72× · 0.40× · 0.90×	$0.06 · 78.3k · 30.6scache reads not recorded
1–17	Codex · gpt-5.4-miniwith my setupPlaywright	79.0 pts75.8–81.4 pts	98.0%94.0–100.0%98.0% · 50 of 50 graded	0.57× · 0.52× · 1.0×	$0.03 · 96.2k · 40.6s85% cache reads
1–17	Claude Code · claude-sonnet-5-5harness default, open-book on knowledge questions	78.2 pts74.0–80.1 pts	98.0%92.0–100.0%98.0% · 49 of 50 graded	1.2× · 1.0× · 0.85×	$0.06 · 147.8k · 28.4s94% cache reads
1–17	Codex · gpt-6.1-solharness default, open-book on knowledge questions	75.0 pts71.8–76.8 pts	98.0%94.0–100.0%98.0% · 50 of 50 graded	2.3× · 2.1× · 3.4×	$0.16 · 654.4k · 2m 40s95% cache reads
1–17	Codex · gpt-5.4-miniharness default, open-book on knowledge questions	70.2 pts62.6–76.5 pts	87.8%78.0–95.9%87.8% · 49 of 50 graded	1.0× · 1.0× · 1.0×	$0.06 · 256.2k · 46.6s83% cache reads
1–17	Hermes · claude-haiku-4-5-20251001harness default, open-book on knowledge questions	60.5 pts51.6–68.5 pts	77.6%65.3–88.0%77.6% · 49 of 50 graded	1.7× · 1.8× · 2.0×	$0.10 · 419.4k · 1m 29scache reads not recorded
1–17	DeepSeek Harness · claude-haiku-4-5-20251001harness default, open-book on knowledge questions	59.6 pts49.2–69.7 pts	72.9%59.6–85.1%72.9% · 48 of 50 graded	1.4× · 1.4× · 0.77×	$0.06 · 177.6k · 29.4scache reads not recorded
1–17	Claude Agent SDK · claude-haiku-4-5-20251001harness default, open-book on knowledge questions	54.0 pts43.5–64.5 pts	65.3%52.0–78.0%65.3% · 49 of 50 graded	1.9× · 2.3× · 0.68×	$0.05 · 227.2k · 19.4scache reads not recorded
1–17	eve · claude-haiku-4-5-20251001harness default, open-book on knowledge questions	54.0 pts44.5–64.5 pts	66.0%54.0–78.0%66.0% · 50 of 50 graded	1.2× · 1.0× · 0.90×	$0.04 · 147.6k · 23.9scache reads not recorded
1–17	Aider · claude-haiku-4-5-20251001harness default, open-book on knowledge questions	51.8 pts42.0–62.5 pts	42.9%30.0–57.1%42.9% · 49 of 50 graded	0.14× · 0.034× · 1.0×	$0.0034 · 3k · 28.3scache reads not recorded
1–17	Continue · claude-haiku-4-5-20251001harness default, open-book on knowledge questions	50.7 pts40.1–62.8 pts	59.6%45.7–73.9%59.6% · 47 of 50 graded	0.81× · 0.20× · 0.91×	$0.02 · 22.3k · 26.4scache reads not recorded
1–17	opencode · claude-haiku-4-5-20251001harness default, open-book on knowledge questions	47.9 pts37.7–58.2 pts	58.0%44.0–72.0%58.0% · 50 of 50 graded	1.0× · 0.55× · 1.1×	$0.03 · 51.3k · 30.4scache reads not recorded
1–17	Claude Code · claude-haiku-4-5-20251001harness default, open-book on knowledge questions	44.2 pts33.8–54.9 pts	54.0%40.0–68.0%54.0% · 50 of 50 graded	1.5× · 2.2× · 1.0×	$0.05 · 179.1k · 28.6s94% cache reads
18	OpenClaw · claude-haiku-4-5-20251001harness default, open-book on knowledge questions	27.7 pts17.5–37.9 pts	34.0%20.0–48.0%34.0% · 50 of 50 graded	1.7× · 2.6× · 2.1×	$0.06 · 283.3k · 1m 05scache reads not recorded

Efficiency and spend per pass can order setups differently: efficiency compares the typical (median) passing run, task by task, while spend per pass averages every attempt, so a few very expensive runs, passing or not, count in full; and spend per pass also counts the attempts that failed or did not finish, and divides by the pass rate. A setup is called the most frugal only where both put it first and both separate it from every other setup.

This benchmark's own board: newest eligible study per setup.

live · updated 2026-10-03

A real harness session with its native tools, nothing added and nothing removed: open-book on knowledge questions.

WindTunnel-style — harness default
Setup	Pass rate · 95% CI	Tasks	Runs	Row tier	Updated
Claude Agent SDKclaude-sonnet-5-5 · anthropic	100%	50	50	official	2026-10-03
Codexgpt-6.1-sol · openai	98%(95% CI 94–100)	50	50	official	2026-10-03
Claude Codeclaude-sonnet-5-5 · anthropic	98%(95% CI 93.9–100)	49	50	official	2026-10-03
opencodeclaude-sonnet-5-5 · anthropic	98%(95% CI 93.9–100)	49	50	official	2026-10-03
Codexgpt-5.4-mini · openai	89.6%(95% CI 79.2–97.9)	48	50	official	2026-10-03
Hermesclaude-haiku-4-5-20251001 · anthropic	77.6%(95% CI 65.3–87.8)	49	50	official	2026-10-03
DeepSeek Harnessclaude-haiku-4-5-20251001 · anthropic	72.9%(95% CI 60.4–85.4)	48	50	official	2026-10-03
eveclaude-haiku-4-5-20251001 · anthropic	70.2%(95% CI 55.3–83)	47	50	official	2026-10-03
Claude Agent SDKclaude-haiku-4-5-20251001 · anthropic	65.3%(95% CI 51–77.6)	49	50	official	2026-10-03
Continueclaude-haiku-4-5-20251001 · anthropic	59.6%(95% CI 46.8–74.5)	47	50	official	2026-10-03
opencodeclaude-haiku-4-5-20251001 · anthropic	58%(95% CI 46–72)	50	50	official	2026-10-03
Claude Codeclaude-haiku-4-5-20251001 · anthropic	54%(95% CI 40–68)	50	50	official	2026-10-03
Aiderclaude-haiku-4-5-20251001 · anthropic	42.9%(95% CI 28.6–57.1)	49	50	official	2026-10-03
OpenClawclaude-haiku-4-5-20251001 · anthropic	37%(95% CI 23.9–50)	46	50	official	2026-10-03
DeepAgentsclaude-haiku-4-5-20251001 · anthropic	8%(95% CI 2–16)	50	50	official	2026-10-03

Sorted by the benchmark's own headline figure. An order is not a claim that one setup beats another — read the intervals.

The card

Declared data only — nothing here is inferred or scored.

Identity

Category: Web
Corpus: 50 tasks · v2
Lenses: reference + verifier
Graded by: the benchmark's own reference answers and verifier
Headline: Pass rate

Protocol

Repeats: 3 recommended
Sample floor: not declared
Chance anchor: Derived from the pinned corpus's answer formats.

Access and licence

Dataset: open
Task text: withheld — board figures publish, task text does not
Licence: Apache-2.0 (unverified)
Catalog: community

Disclosures

Contamination: No contamination data declared.
Conflict of interest: Uploaded by the publisher of this index.
Notice: Task text is withheld from public pages until an admin verifies this licence. Running this benchmark is unaffected.

Evidence

The published studies of this benchmark: every run, trajectory and grade, on the studies site.

Published studies of WindTunnel

Setup	Spend per pass · cost	Spend per pass · tokens	Spend per pass · time
Claude Agent SDK · claude-sonnet-5-5 · harness default	$0.09 per pass($0.08–$0.12)30% went to the costliest 10% of attempts (5 of 50)	249.3k per pass(190.4k–318.3k)35% went to the costliest 10% of attempts (5 of 50)cache split not recorded on 50 attempts	21.1s per pass(16.3s–26.6s)34% went to the costliest 10% of attempts (5 of 50)
Claude Code · claude-haiku-4-5-20251001 · WebMCP	$0.03 per pass($0.03–$0.03)19% went to the costliest 10% of attempts (5 of 49)	182.5k per pass(164.6k–203.7k)19% went to the costliest 10% of attempts (5 of 49)of which 96% cache reads and 3% fresh input	37.6s per pass(33.9s–42.5s)20% went to the costliest 10% of attempts (5 of 49)
Codex · gpt-5.4-mini · WebMCP	$0.02 per pass($0.02–$0.02)16% went to the costliest 10% of attempts (5 of 49)	78.1k per pass(71.7k–85.5k)17% went to the costliest 10% of attempts (5 of 49)of which 87% cache reads and 13% fresh input	36.7s per pass(34.2s–39.7s)15% went to the costliest 10% of attempts (5 of 49)
Claude Code · claude-haiku-4-5-20251001 · Playwright	$0.08 per pass($0.05–$0.10)37% went to the costliest 10% of attempts (5 of 48)	470.8k per pass(331k–658.9k)39% went to the costliest 10% of attempts (5 of 48)of which 96% cache reads and 4% fresh input	1m 02s per pass(50.7s–1m 17s)28% went to the costliest 10% of attempts (5 of 48)
opencode · claude-sonnet-5-5 · harness default	$0.06 per pass($0.05–$0.08)30% went to the costliest 10% of attempts (5 of 49)	123.1k per pass(92.7k–159.6k)35% went to the costliest 10% of attempts (5 of 49)cache split not recorded on 49 attempts	45.6s per pass(36.6s–57s)28% went to the costliest 10% of attempts (5 of 49)
Codex · gpt-5.4-mini · Playwright	$0.07 per pass($0.04–$0.10)51% went to the costliest 10% of attempts (5 of 50)	247.9k per pass(134.6k–377.1k)56% went to the costliest 10% of attempts (5 of 50)of which 89% cache reads and 10% fresh input	55.5s per pass(45.4s–1m 08s)28% went to the costliest 10% of attempts (5 of 50)
Claude Code · claude-sonnet-5-5 · harness default	$0.09 per pass($0.07–$0.11)27% went to the costliest 10% of attempts (5 of 49)	238.6k per pass(183.5k–305.5k)31% went to the costliest 10% of attempts (5 of 49)of which 94% cache reads and 5% fresh input	39.9s per pass(32.7s–48.9s)27% went to the costliest 10% of attempts (5 of 49)
Codex · gpt-6.1-sol · harness default	$0.17 per pass($0.14–$0.21)22% went to the costliest 10% of attempts (5 of 50)	730.8k per pass(574k–915.4k)26% went to the costliest 10% of attempts (5 of 50)of which 95% cache reads and 4% fresh input	3m 00s per pass(2m 27s–3m 38s)22% went to the costliest 10% of attempts (5 of 50)
Codex · gpt-5.4-mini · harness default	$0.41 per pass($0.19–$0.75)62% went to the costliest 10% of attempts (5 of 49)	2.9M per pass(1.3M–5.3M)65% went to the costliest 10% of attempts (5 of 49)of which 93% cache reads and 7% fresh input	2m 28s per pass(1m 18s–4m 14s)56% went to the costliest 10% of attempts (5 of 49)
Hermes · claude-haiku-4-5-20251001 · harness default	$0.37 per pass($0.23–$0.55)35% went to the costliest 10% of attempts (5 of 49)	2M per pass(1.3M–2.8M)31% went to the costliest 10% of attempts (5 of 49)cache split not recorded on 49 attempts	3m 55s per pass(2m 41s–5m 30s)28% went to the costliest 10% of attempts (5 of 49)
DeepSeek Harness · claude-haiku-4-5-20251001 · harness default	$0.48 per pass($0.31–$0.73)35% went to the costliest 10% of attempts (5 of 48)	3.1M per pass(1.9M–4.8M)37% went to the costliest 10% of attempts (5 of 48)cache split not recorded on 48 attempts	3m 13s per pass(2m 03s–4m 50s)34% went to the costliest 10% of attempts (5 of 48)
Claude Agent SDK · claude-haiku-4-5-20251001 · harness default	$0.38 per pass($0.24–$0.57)35% went to the costliest 10% of attempts (5 of 49)	2.5M per pass(1.6M–4M)38% went to the costliest 10% of attempts (5 of 49)cache split not recorded on 49 attempts	3m 51s per pass(2m 18s–5m 59s)35% went to the costliest 10% of attempts (5 of 49)
eve · claude-haiku-4-5-20251001 · harness default	$0.21 per pass($0.14–$0.29)27% went to the costliest 10% of attempts (5 of 50)	1M per pass(693.2k–1.5M)31% went to the costliest 10% of attempts (5 of 50)cache split not recorded on 50 attempts	6m 22s per pass(3m 18s–11m 21s)52% went to the costliest 10% of attempts (5 of 50)
Aider · claude-haiku-4-5-20251001 · harness default	$0.01 per pass($0.01–$0.01)18% went to the costliest 10% of attempts (5 of 49)	8.1k per pass(6k–11.9k)19% went to the costliest 10% of attempts (5 of 49)cache split not recorded on 49 attempts	1m 12s per pass(53.9s–1m 49s)17% went to the costliest 10% of attempts (5 of 49)
Continue · claude-haiku-4-5-20251001 · harness default	$1.39 per pass($0.79–$2.30)42% went to the costliest 10% of attempts (5 of 47)	1.4M per pass(765k–2.2M)43% went to the costliest 10% of attempts (5 of 47)cache split not recorded on 47 attempts	2m 48s per pass(1m 41s–4m 42s)40% went to the costliest 10% of attempts (5 of 47)
opencode · claude-haiku-4-5-20251001 · harness default	$0.31 per pass($0.19–$0.50)34% went to the costliest 10% of attempts (5 of 50)	1.9M per pass(1.1M–3.1M)37% went to the costliest 10% of attempts (5 of 50)cache split not recorded on 50 attempts	3m 21s per pass(2m 05s–5m 09s)32% went to the costliest 10% of attempts (5 of 50)
Claude Code · claude-haiku-4-5-20251001 · harness default	$0.37 per pass($0.23–$0.61)37% went to the costliest 10% of attempts (5 of 50)	2.5M per pass(1.5M–4.3M)40% went to the costliest 10% of attempts (5 of 50)of which 97% cache reads and 2% fresh input	2m 52s per pass(1m 49s–4m 34s)33% went to the costliest 10% of attempts (5 of 50)
OpenClaw · claude-haiku-4-5-20251001 · harness default	$0.56 per pass($0.33–$1.04)33% went to the costliest 10% of attempts (5 of 50)	4M per pass(2.3M–7.4M)35% went to the costliest 10% of attempts (5 of 50)cache split not recorded on 50 attempts	11m 41s per pass(4m 59s–25m 49s)63% went to the costliest 10% of attempts (5 of 50)