Code

HumanEval-style

164 tasks, v1, graded by the benchmark's own execution tests.

In the leaderboard

Every setup scored on HumanEval, whichever other benchmarks it covers. The leaderboard compares a setup only with setups scored on exactly the same benchmarks, so its bands can differ from these.

rank on this benchmark	Setup	Score	Pass rate	cost · tokens · time	median spend of a passing run
1–15	DeepAgents · claude-haiku-4-5-20251001harness default, open-book on knowledge questions	86.4 pts83.2–89.3 pts	92.7%88.4–96.3%92.7% · 164 of 164 graded	1.0× · 0.38× · 0.11×	$0.01 · 4.3k · 1.7scache reads not recorded
1–15	Continue · claude-haiku-4-5-20251001harness default, open-book on knowledge questions	85.0 pts81.8–87.8 pts	93.3%89.0–96.9%93.3% · 163 of 164 graded	0.63× · 0.23× · 1.0×	$0.0033 · 2.5k · 14.3scache reads not recorded
1–15	eve · claude-haiku-4-5-20251001harness default, open-book on knowledge questions	84.6 pts82.1–86.7 pts	95.7%92.7–98.8%95.7% · 164 of 164 graded	1.0× · 0.38× · 0.21×	$0.01 · 4.2k · 3.4scache reads not recorded
1–15	Aider · claude-haiku-4-5-20251001harness default, open-book on knowledge questions	83.0 pts79.6–86.1 pts	92.1%87.8–96.3%92.1% · 164 of 164 graded	0.66× · 0.25× · 1.2×	$0.0035 · 2.8k · 18.3scache reads not recorded
1–15	opencode · claude-haiku-4-5-20251001harness default, open-book on knowledge questions	81.8 pts78.6–84.3 pts	93.9%89.6–97.5%93.9% · 163 of 164 graded	0.45× · 0.75× · 1.1×	$0.0024 · 8.3k · 18.2scache reads not recorded
1–15	Codex · gpt-5.4-miniharness default, open-book on knowledge questions	81.6 pts78.4–83.9 pts	93.9%90.2–97.6%93.9% · 164 of 164 graded	0.44× · 0.97× · 1.0×	$0.0025 · 10.7k · 14.2s89% cache reads
1–15	DeepSeek Harness · claude-haiku-4-5-20251001harness default, open-book on knowledge questions	81.0 pts77.6–83.9 pts	92.0%87.7–95.7%92.0% · 163 of 164 graded	0.55× · 1.0× · 0.26×	$0.0031 · 11.1k · 4scache reads not recorded
1–15	opencode · claude-sonnet-5-5harness default, open-book on knowledge questions	80.4 pts78.6–81.8 pts	98.2%95.7–100.0%98.2% · 164 of 164 graded	0.89× · 0.99× · 1.1×	$0.0048 · 11k · 18.3scache reads not recorded
1–15	Claude Agent SDK · claude-sonnet-5-5harness default, open-book on knowledge questions	80.3 pts78.5–81.7 pts	98.2%95.7–100.0%98.2% · 164 of 164 graded	2.8× · 2.6× · 0.19×	$0.02 · 28.9k · 3scache reads not recorded
1–15	Hermes · claude-haiku-4-5-20251001harness default, open-book on knowledge questions	78.4 pts75.1–81.2 pts	92.7%88.4–96.3%92.7% · 164 of 164 graded	0.54× · 1.2× · 1.8×	$0.0028 · 13.2k · 28.3scache reads not recorded
1–15	OpenClaw · claude-haiku-4-5-20251001harness default, open-book on knowledge questions	76.6 pts74.7–78.0 pts	98.1%95.7–100.0%98.1% · 162 of 164 graded	1.7× · 3.4× · 3.1×	$0.01 · 37.9k · 49.1scache reads not recorded
1–15	Codex · gpt-6.1-solharness default, open-book on knowledge questions	76.6 pts74.4–78.1 pts	97.5%94.5–99.4%97.5% · 163 of 164 graded	4.5× · 1.1× · 1.0×	$0.02 · 12k · 16.3s0% cache reads
1–15	Claude Agent SDK · claude-haiku-4-5-20251001harness default, open-book on knowledge questions	76.6 pts74.1–78.5 pts	96.3%93.3–98.8%96.3% · 164 of 164 graded	2.0× · 2.7× · 0.49×	$0.01 · 30.4k · 7.8scache reads not recorded
1–15	Claude Code · claude-sonnet-5-5harness default, open-book on knowledge questions	76.5 pts74.7–77.9 pts	98.2%95.7–100.0%98.2% · 164 of 164 graded	2.9× · 2.6× · 1.0×	$0.02 · 29.3k · 14.5s88% cache reads
1–15	Claude Code · claude-haiku-4-5-20251001harness default, open-book on knowledge questions	75.4 pts73.0–77.3 pts	96.3%93.3–98.8%96.3% · 164 of 164 graded	2.0× · 3.2× · 1.2×	$0.01 · 35.2k · 18.3s89% cache reads

Efficiency and spend per pass can order setups differently: efficiency compares the typical (median) passing run, task by task, while spend per pass averages every attempt, so a few very expensive runs, passing or not, count in full; and spend per pass also counts the attempts that failed or did not finish, and divides by the pass rate. A setup is called the most frugal only where both put it first and both separate it from every other setup.

This benchmark's own board: newest eligible study per setup.

live · updated 2026-10-04

A real harness session with its native tools, nothing added and nothing removed: open-book on knowledge questions.

HumanEval-style — harness default
Setup	Pass rate · 95% CI	Tasks	Runs	Row tier	Updated
Claude Agent SDKclaude-sonnet-5-5 · anthropic	98.2%(95% CI 95.7–100)	164	164	official	2026-10-04
Claude Codeclaude-sonnet-5-5 · anthropic	98.2%(95% CI 95.7–100)	164	164	official	2026-10-04
opencodeclaude-sonnet-5-5 · anthropic	98.2%(95% CI 96.3–100)	164	164	official	2026-10-04
OpenClawclaude-haiku-4-5-20251001 · anthropic	98.2%(95% CI 96.3–100)	162	164	official	2026-10-04
Codexgpt-6.1-sol · openai	97.6%(95% CI 95.1–99.4)	163	164	official	2026-10-04
Claude Agent SDKclaude-haiku-4-5-20251001 · anthropic	96.3%(95% CI 93.3–98.8)	164	164	official	2026-10-04
Claude Codeclaude-haiku-4-5-20251001 · anthropic	96.3%(95% CI 93.3–98.8)	164	164	official	2026-10-04
eveclaude-haiku-4-5-20251001 · anthropic	95.7%(95% CI 92.7–98.2)	164	164	official	2026-10-04
Codexgpt-5.4-mini · openai	93.9%(95% CI 89.6–97.6)	164	164	official	2026-10-04
opencodeclaude-haiku-4-5-20251001 · anthropic	93.9%(95% CI 90.2–97.6)	163	164	official	2026-10-04
Continueclaude-haiku-4-5-20251001 · anthropic	93.3%(95% CI 89.6–96.3)	163	164	official	2026-10-04
DeepAgentsclaude-haiku-4-5-20251001 · anthropic	92.7%(95% CI 88.4–96.3)	164	164	official	2026-10-04
Hermesclaude-haiku-4-5-20251001 · anthropic	92.7%(95% CI 88.4–96.3)	164	164	official	2026-10-04
Aiderclaude-haiku-4-5-20251001 · anthropic	92.1%(95% CI 87.8–95.7)	164	164	official	2026-10-04
DeepSeek Harnessclaude-haiku-4-5-20251001 · anthropic	92%(95% CI 87.7–96.3)	163	164	official	2026-10-04

Sorted by the benchmark's own headline figure. An order is not a claim that one setup beats another — read the intervals.

The card

Declared data only — nothing here is inferred or scored.

Identity

Category: Code
Corpus: 164 tasks · v1
Lenses: execution
Graded by: the benchmark's own execution tests
Headline: Pass rate

Protocol

Repeats: 1 recommended
Sample floor: 50 tasks
Chance anchor: Derived from the pinned corpus's answer formats.

Access and licence

Dataset: open
Task text: available
Licence: MIT
Catalog: featured

Disclosures

Contamination: No contamination data declared.
Conflict of interest: None declared.

Evidence

The published studies of this benchmark: every run, trajectory and grade, on the studies site.

Published studies of HumanEval

Setup	Spend per pass · cost	Spend per pass · tokens	Spend per pass · time
DeepAgents · claude-haiku-4-5-20251001 · harness default	$0.01 per pass($0.01–$0.01)15% went to the costliest 10% of attempts (17 of 164)	4.7k per pass(4.5k–4.9k)11% went to the costliest 10% of attempts (17 of 164)cache split not recorded on 164 attempts	2.1s per pass(1.9s–2.2s)21% went to the costliest 10% of attempts (17 of 164)
Continue · claude-haiku-4-5-20251001 · harness default	$0.0038 per pass($0.0036–$0.0040)15% went to the costliest 10% of attempts (17 of 163)	2.8k per pass(2.6k–2.9k)13% went to the costliest 10% of attempts (17 of 163)cache split not recorded on 163 attempts	16.4s per pass(15.7s–17.2s)12% went to the costliest 10% of attempts (17 of 163)
eve · claude-haiku-4-5-20251001 · harness default	$0.01 per pass($0.01–$0.01)19% went to the costliest 10% of attempts (17 of 164)	5.1k per pass(4.8k–5.5k)20% went to the costliest 10% of attempts (17 of 164)cache split not recorded on 164 attempts	4.1s per pass(3.8s–4.4s)20% went to the costliest 10% of attempts (17 of 164)
Aider · claude-haiku-4-5-20251001 · harness default	$0.0039 per pass($0.0037–$0.0041)15% went to the costliest 10% of attempts (17 of 164)	3k per pass(2.9k–3.2k)12% went to the costliest 10% of attempts (17 of 164)cache split not recorded on 164 attempts	21.1s per pass(19.8s–22.7s)15% went to the costliest 10% of attempts (17 of 164)
opencode · claude-haiku-4-5-20251001 · harness default	$0.0028 per pass($0.0026–$0.0030)17% went to the costliest 10% of attempts (17 of 163)	9.1k per pass(8.7k–9.6k)12% went to the costliest 10% of attempts (17 of 163)cache split not recorded on 163 attempts	19.1s per pass(18.3s–20s)13% went to the costliest 10% of attempts (17 of 163)
Codex · gpt-5.4-mini · harness default	$0.01 per pass($0.0044–$0.01)22% went to the costliest 10% of attempts (17 of 163)1 attempt's cost was not recorded	11.5k per pass(11.1k–12k)11% went to the costliest 10% of attempts (17 of 163)1 attempt's tokens was not recordedof which 58% cache reads and 39% fresh input	16.4s per pass(15.6s–17.5s)15% went to the costliest 10% of attempts (17 of 164)
DeepSeek Harness · claude-haiku-4-5-20251001 · harness default	$0.0037 per pass($0.0034–$0.0041)19% went to the costliest 10% of attempts (17 of 163)	12.6k per pass(11.8k–13.6k)14% went to the costliest 10% of attempts (17 of 163)cache split not recorded on 163 attempts	5s per pass(4.6s–5.4s)19% went to the costliest 10% of attempts (17 of 163)
opencode · claude-sonnet-5-5 · harness default	$0.01 per pass($0.01–$0.01)21% went to the costliest 10% of attempts (17 of 164)	11.2k per pass(11k–11.5k)11% went to the costliest 10% of attempts (17 of 164)cache split not recorded on 164 attempts	18.6s per pass(18.1s–19.1s)12% went to the costliest 10% of attempts (17 of 164)
Claude Agent SDK · claude-sonnet-5-5 · harness default	$0.02 per pass($0.02–$0.02)18% went to the costliest 10% of attempts (17 of 164)	29.5k per pass(28.9k–30.2k)10% went to the costliest 10% of attempts (17 of 164)cache split not recorded on 164 attempts	3.2s per pass(3.1s–3.3s)15% went to the costliest 10% of attempts (17 of 164)
Hermes · claude-haiku-4-5-20251001 · harness default	$0.0033 per pass($0.0031–$0.0035)16% went to the costliest 10% of attempts (17 of 164)	14.3k per pass(13.7k–15k)11% went to the costliest 10% of attempts (17 of 164)cache split not recorded on 164 attempts	32.3s per pass(29.6s–35.6s)18% went to the costliest 10% of attempts (17 of 164)
OpenClaw · claude-haiku-4-5-20251001 · harness default	$0.01 per pass($0.01–$0.01)21% went to the costliest 10% of attempts (17 of 162)	38.8k per pass(38.1k–39.9k)11% went to the costliest 10% of attempts (17 of 162)cache split not recorded on 162 attempts	53s per pass(51.2s–55.2s)15% went to the costliest 10% of attempts (17 of 162)
Codex · gpt-6.1-sol · harness default	$0.03 per pass($0.02–$0.03)11% went to the costliest 10% of attempts (17 of 162)1 attempt's cost was not recorded	12.3k per pass(12k–12.7k)11% went to the costliest 10% of attempts (17 of 162)1 attempt's tokens was not recordedof which 0% cache reads and 99% fresh input	17.3s per pass(16.7s–18s)14% went to the costliest 10% of attempts (17 of 163)
Claude Agent SDK · claude-haiku-4-5-20251001 · harness default	$0.01 per pass($0.01–$0.01)24% went to the costliest 10% of attempts (17 of 164)	31.9k per pass(30.9k–33.1k)11% went to the costliest 10% of attempts (17 of 164)cache split not recorded on 164 attempts	10.4s per pass(8.9s–12.2s)31% went to the costliest 10% of attempts (17 of 164)
Claude Code · claude-sonnet-5-5 · harness default	$0.02 per pass($0.02–$0.02)14% went to the costliest 10% of attempts (17 of 164)	30k per pass(29.4k–30.8k)11% went to the costliest 10% of attempts (17 of 164)of which 87% cache reads and 12% fresh input	15.9s per pass(15.4s–16.4s)13% went to the costliest 10% of attempts (17 of 164)
Claude Code · claude-haiku-4-5-20251001 · harness default	$0.01 per pass($0.01–$0.01)23% went to the costliest 10% of attempts (17 of 164)	37k per pass(35.9k–38.4k)12% went to the costliest 10% of attempts (17 of 164)of which 88% cache reads and 9% fresh input	22.7s per pass(21s–24.8s)22% went to the costliest 10% of attempts (17 of 164)