Knowledge & reasoning

GPQA Diamond-style

198 tasks, v1, graded by the benchmark's own reference answers.

gated dataset

In the leaderboard

Every setup scored on GPQA Diamond, whichever other benchmarks it covers. The leaderboard compares a setup only with setups scored on exactly the same benchmarks, so its bands can differ from these.

rank on this benchmark	Setup	Score	Pass rate	Above chance	cost · tokens · time	median spend of a passing run
1–15	opencode · claude-sonnet-5-5harness default, open-book on knowledge questions	78.1 pts73.8–81.9 pts	90.9%86.9–94.9%90.9% · 198 of 198 graded	87.9%82.5–93.3%	0.38× · 0.53× · 0.99×	$0.01 · 11.2k · 20.3scache reads not recorded
1–15	Claude Agent SDK · claude-sonnet-5-5harness default, open-book on knowledge questions	77.5 pts73.0–81.1 pts	90.4%85.9–94.4%90.4% · 198 of 198 graded	87.2%81.1–92.6%	1.0× · 1.4× · 0.26×	$0.02 · 29.4k · 5.7scache reads not recorded
1–15	Codex · gpt-6.1-solharness default, open-book on knowledge questions	77.1 pts73.9–79.7 pts	96.0%92.9–98.5%96.0% · 198 of 198 graded	94.6%90.6–98.0%	1.6× · 0.72× · 0.93×	$0.03 · 16.1k · 20.3s0% cache reads
1–15	Claude Code · claude-sonnet-5-5harness default, open-book on knowledge questions	73.5 pts69.5–77.0 pts	92.4%88.4–96.0%92.4% · 198 of 198 graded	89.9%84.5–94.6%	1.0× · 1.4× · 0.89×	$0.02 · 29.7k · 18.3s86% cache reads
1–15	Codex · gpt-5.4-miniharness default, open-book on knowledge questions	64.9 pts59.1–71.3 pts	78.8%72.7–84.3%78.8% · 198 of 198 graded	71.7%63.6–79.1%	0.42× · 0.67× · 0.88×	$0.01 · 11.1k · 18.2s80% cache reads
1–15	Aider · claude-haiku-4-5-20251001harness default, open-book on knowledge questions	54.1 pts47.1–60.5 pts	60.6%54.0–67.2%60.6% · 198 of 198 graded	47.5%38.7–56.2%	0.33× · 0.18× · 1.0×	$0.01 · 3.1k · 22.3scache reads not recorded
1–15	Continue · claude-haiku-4-5-20251001harness default, open-book on knowledge questions	52.4 pts45.6–59.3 pts	64.6%57.9–71.5%64.6% · 195 of 198 graded	52.8%43.8–62.0%	0.60× · 0.25× · 1.0×	$0.01 · 3.4k · 22.3scache reads not recorded
1–15	Hermes · claude-haiku-4-5-20251001harness default, open-book on knowledge questions	51.2 pts44.4–57.1 pts	66.7%59.6–72.7%66.7% · 198 of 198 graded	55.6%46.1–63.6%	0.40× · 1.0× · 1.6×	$0.01 · 13.9k · 32.5scache reads not recorded
1–15	Claude Code · claude-haiku-4-5-20251001harness default, open-book on knowledge questions	49.2 pts43.1–54.9 pts	70.7%64.6–76.3%70.7% · 198 of 198 graded	60.9%52.9–68.4%	1.7× · 2.1× · 1.9×	$0.03 · 38.7k · 42.6s81% cache reads
1–15	DeepAgents · claude-haiku-4-5-20251001harness default, open-book on knowledge questions	49.1 pts40.9–56.7 pts	66.2%58.6–72.2%66.2% · 198 of 198 graded	54.9%44.8–63.0%	1.2× · 0.76× · 0.58×	$0.02 · 11.4k · 11.7scache reads not recorded
1–15	OpenClaw · claude-haiku-4-5-20251001harness default, open-book on knowledge questions	47.2 pts40.7–53.2 pts	68.2%61.6–74.2%68.2% · 198 of 198 graded	57.6%48.8–65.7%	1.1× · 2.0× · 2.9×	$0.02 · 39.7k · 1m 05scache reads not recorded
1–15	DeepSeek Harness · claude-haiku-4-5-20251001harness default, open-book on knowledge questions	46.4 pts39.2–52.9 pts	67.2%60.1–73.7%67.2% · 198 of 198 graded	56.2%46.8–65.0%	1.5× · 2.5× · 1.2×	$0.03 · 55.3k · 30.7scache reads not recorded
1–15	eve · claude-haiku-4-5-20251001harness default, open-book on knowledge questions	46.1 pts39.1–52.2 pts	65.2%58.1–71.2%65.2% · 198 of 198 graded	53.5%44.1–61.6%	1.4× · 0.99× · 0.88×	$0.02 · 17.8k · 19.4scache reads not recorded
1–15	Claude Agent SDK · claude-haiku-4-5-20251001harness default, open-book on knowledge questions	45.6 pts38.6–51.6 pts	66.7%59.6–72.7%66.7% · 198 of 198 graded	55.6%46.1–63.6%	1.7× · 1.9× · 1.4×	$0.03 · 34.5k · 33.5scache reads not recorded
1–15	opencode · claude-haiku-4-5-20251001harness default, open-book on knowledge questions	43.3 pts36.1–49.4 pts	55.1%48.0–61.6%55.1% · 198 of 198 graded	40.1%30.6–48.8%	0.32× · 0.56× · 1.0×	$0.01 · 8.9k · 22.3scache reads not recorded

Efficiency and spend per pass can order setups differently: efficiency compares the typical (median) passing run, task by task, while spend per pass averages every attempt, so a few very expensive runs, passing or not, count in full; and spend per pass also counts the attempts that failed or did not finish, and divides by the pass rate. A setup is called the most frugal only where both put it first and both separate it from every other setup.

This benchmark's own board: newest eligible study per setup.

live · updated 2026-10-04

A real harness session with its native tools, nothing added and nothing removed: open-book on knowledge questions.

GPQA Diamond-style — harness default
Setup	Pass rate · 95% CI	Tasks	Runs	Row tier	Updated
Codexgpt-6.1-sol · openai	96%(95% CI 92.9–98.5)	198	198	official	2026-10-04
Claude Codeclaude-sonnet-5-5 · anthropic	92.4%(95% CI 88.9–96)	198	198	official	2026-10-04
opencodeclaude-sonnet-5-5 · anthropic	90.9%(95% CI 86.9–95)	198	198	official	2026-10-04
Claude Agent SDKclaude-sonnet-5-5 · anthropic	90.4%(95% CI 86.4–94.4)	198	198	official	2026-10-04
Codexgpt-5.4-mini · openai	78.8%(95% CI 72.7–83.8)	198	198	official	2026-10-04
Claude Codeclaude-haiku-4-5-20251001 · anthropic	70.7%(95% CI 64.7–77.3)	198	198	official	2026-10-04
OpenClawclaude-haiku-4-5-20251001 · anthropic	68.2%(95% CI 61.6–74.2)	198	198	official	2026-10-04
DeepSeek Harnessclaude-haiku-4-5-20251001 · anthropic	67.2%(95% CI 60.6–74.2)	198	198	official	2026-10-04
Claude Agent SDKclaude-haiku-4-5-20251001 · anthropic	67%(95% CI 60.4–73.6)	197	198	official	2026-10-04
DeepAgentsclaude-haiku-4-5-20251001 · anthropic	66.8%(95% CI 60.7–73.5)	196	198	official	2026-10-04
Hermesclaude-haiku-4-5-20251001 · anthropic	66.7%(95% CI 60.1–72.7)	198	198	official	2026-10-04
eveclaude-haiku-4-5-20251001 · anthropic	65.2%(95% CI 58.1–71.7)	198	198	official	2026-10-04
Continueclaude-haiku-4-5-20251001 · anthropic	64.6%(95% CI 57.4–71.3)	195	198	official	2026-10-04
Aiderclaude-haiku-4-5-20251001 · anthropic	60.6%(95% CI 54–67.2)	198	198	official	2026-10-04
opencodeclaude-haiku-4-5-20251001 · anthropic	55.1%(95% CI 48.5–62.1)	198	198	official	2026-10-04

Sorted by the benchmark's own headline figure. An order is not a claim that one setup beats another — read the intervals.

The card

Declared data only — nothing here is inferred or scored.

Identity

Category: Knowledge & reasoning
Corpus: 198 tasks · v1
Lenses: reference
Graded by: the benchmark's own reference answers
Headline: Pass rate

Protocol

Repeats: 1 recommended
Sample floor: 100 tasks
Chance anchor: Derived from the pinned corpus's answer formats.

Access and licence

Dataset: gated
Task text: available
Licence: CC-BY-4.0
Catalog: featured

Disclosures

Contamination: No contamination data declared.
Conflict of interest: None declared.

Evidence

The published studies of this benchmark: every run, trajectory and grade, on the studies site.

Published studies of GPQA Diamond

Setup	Spend per pass · cost	Spend per pass · tokens	Spend per pass · time
opencode · claude-sonnet-5-5 · harness default	$0.01 per pass($0.01–$0.01)25% went to the costliest 10% of attempts (20 of 198)	13.2k per pass(12.1k–14.7k)16% went to the costliest 10% of attempts (20 of 198)cache split not recorded on 198 attempts	24.4s per pass(22.9s–26.1s)17% went to the costliest 10% of attempts (20 of 198)
Claude Agent SDK · claude-sonnet-5-5 · harness default	$0.03 per pass($0.02–$0.03)29% went to the costliest 10% of attempts (20 of 198)	35.4k per pass(32.2k–39.6k)17% went to the costliest 10% of attempts (20 of 198)cache split not recorded on 198 attempts	9.7s per pass(8.1s–11.9s)37% went to the costliest 10% of attempts (20 of 198)
Codex · gpt-6.1-sol · harness default	$0.03 per pass($0.03–$0.04)19% went to the costliest 10% of attempts (20 of 197)1 attempt's cost was not recorded	17.4k per pass(15.6k–20k)22% went to the costliest 10% of attempts (20 of 197)1 attempt's tokens was not recordedof which 6% cache reads and 94% fresh input	23.9s per pass(21.9s–26.4s)21% went to the costliest 10% of attempts (20 of 198)
Claude Code · claude-sonnet-5-5 · harness default	$0.03 per pass($0.02–$0.03)23% went to the costliest 10% of attempts (20 of 197)1 attempt's cost was not recorded	33.9k per pass(31.8k–36.8k)15% went to the costliest 10% of attempts (20 of 197)1 attempt's tokens was not recordedof which 86% cache reads and 12% fresh input	28.5s per pass(21.8s–40.5s)36% went to the costliest 10% of attempts (20 of 198)
Codex · gpt-5.4-mini · harness default	$0.02 per pass($0.02–$0.03)44% went to the costliest 10% of attempts (20 of 198)	21.2k per pass(17k–27.6k)34% went to the costliest 10% of attempts (20 of 198)of which 53% cache reads and 32% fresh input	35.2s per pass(30.1s–42s)31% went to the costliest 10% of attempts (20 of 198)
Aider · claude-haiku-4-5-20251001 · harness default	$0.01 per pass($0.01–$0.01)25% went to the costliest 10% of attempts (20 of 198)	5.8k per pass(4.9k–6.8k)20% went to the costliest 10% of attempts (20 of 198)cache split not recorded on 198 attempts	38.6s per pass(34.3s–44.2s)16% went to the costliest 10% of attempts (20 of 198)
Continue · claude-haiku-4-5-20251001 · harness default	$0.03 per pass($0.02–$0.04)50% went to the costliest 10% of attempts (20 of 195)	19.3k per pass(13.7k–27.1k)55% went to the costliest 10% of attempts (20 of 195)cache split not recorded on 195 attempts	45.4s per pass(39.6s–53.7s)28% went to the costliest 10% of attempts (20 of 195)
Hermes · claude-haiku-4-5-20251001 · harness default	$0.02 per pass($0.01–$0.02)49% went to the costliest 10% of attempts (20 of 198)	40.3k per pass(33.5k–49.6k)44% went to the costliest 10% of attempts (20 of 198)cache split not recorded on 198 attempts	1m 02s per pass(54.6s–1m 12s)25% went to the costliest 10% of attempts (20 of 198)
Claude Code · claude-haiku-4-5-20251001 · harness default	$0.06 per pass($0.05–$0.07)27% went to the costliest 10% of attempts (20 of 198)	64k per pass(56.9k–73.1k)20% went to the costliest 10% of attempts (20 of 198)of which 77% cache reads and 7% fresh input	1m 43s per pass(1m 28s–1m 59s)27% went to the costliest 10% of attempts (20 of 198)
DeepAgents · claude-haiku-4-5-20251001 · harness default	$0.07 per pass($0.05–$0.09)49% went to the costliest 10% of attempts (20 of 196)2 attempts' cost were not recorded	105.2k per pass(72.2k–149.7k)69% went to the costliest 10% of attempts (20 of 196)2 attempts' tokens were not recordedcache split not recorded on 196 attempts	1m 04s per pass(47.1s–1m 24s)58% went to the costliest 10% of attempts (20 of 198)
OpenClaw · claude-haiku-4-5-20251001 · harness default	$0.04 per pass($0.04–$0.05)30% went to the costliest 10% of attempts (20 of 198)	70.7k per pass(59.6k–86.6k)24% went to the costliest 10% of attempts (20 of 198)cache split not recorded on 198 attempts	2m 04s per pass(1m 49s–2m 22s)23% went to the costliest 10% of attempts (20 of 198)
DeepSeek Harness · claude-haiku-4-5-20251001 · harness default	$0.07 per pass($0.05–$0.08)34% went to the costliest 10% of attempts (20 of 198)	137.9k per pass(111.4k–173.2k)41% went to the costliest 10% of attempts (20 of 198)cache split not recorded on 198 attempts	1m 15s per pass(1m 03s–1m 30s)33% went to the costliest 10% of attempts (20 of 198)
eve · claude-haiku-4-5-20251001 · harness default	$0.06 per pass($0.05–$0.07)39% went to the costliest 10% of attempts (20 of 198)	56.6k per pass(43.7k–73.6k)47% went to the costliest 10% of attempts (20 of 198)cache split not recorded on 198 attempts	59.9s per pass(48s–1m 15s)45% went to the costliest 10% of attempts (20 of 198)
Claude Agent SDK · claude-haiku-4-5-20251001 · harness default	$0.07 per pass($0.06–$0.09)29% went to the costliest 10% of attempts (20 of 197)1 attempt's cost was not recorded	60.3k per pass(53.7k–68.8k)19% went to the costliest 10% of attempts (20 of 197)1 attempt's tokens was not recordedcache split not recorded on 197 attempts	1m 49s per pass(1m 29s–2m 12s)35% went to the costliest 10% of attempts (20 of 198)
opencode · claude-haiku-4-5-20251001 · harness default	$0.01 per pass($0.01–$0.02)51% went to the costliest 10% of attempts (20 of 198)	25.6k per pass(20.5k–32.5k)44% went to the costliest 10% of attempts (20 of 198)cache split not recorded on 198 attempts	46.1s per pass(40.2s–53.3s)23% went to the costliest 10% of attempts (20 of 198)