Skip to content
kalanu
Theme
AIME 2025, GPQA Diamond and HumanEval only · 1 setup · 3 of 6 compared benchmarks

DeepAgents · claude-haiku-4-5-20251001 · harness default

anthropic

harness default, open-book on knowledge questions

Its score and figures

Every figure the leaderboard's table prints for this setup.

Its score and figures
rank in group–
Score59.8 pts54.6–65.6 pts
Pass rate70.7%64.5–77.1%70.7% · 605 of 605 graded
Above chance67.0%60.5–73.8%
Efficiency38.433.8–47.1
cost1.3×
tokens0.95×
time0.45×
Coverage3/6
Runs605

50.2 + 2.5 + 2.5 + 4.6 = 59.8 (outcome + cost + tokens + time)

  • pass rate on each benchmark's own grader
  • cost
  • tokens
  • time
  • not measured
First run
2026-10-02
Last run
2026-10-03
did not finish (scored as a fail)
7
Tasks with no grade
0

On each benchmark

Each benchmark it ran: its status, pass rate, spend and spend per pass. The studies behind them are its evidence, below.

WindTunnel

too few passes to price

WindTunnel: 8.0% · 50 of 50 graded · too few passes to price

cost
95.5% of this setup's cost here went to runs that did not pass or were not graded; efficiency does not price it$0.50 per pass($0.20–$2.09)47% went to the costliest 10% of attempts (5 of 50)
tokens
96.0% of this setup's tokens here went to runs that did not pass or were not graded; efficiency does not price it941.4k per pass(334k–4.1M)59% went to the costliest 10% of attempts (5 of 50)cache split not recorded on 50 attempts
time
96.4% of this setup's time here went to runs that did not pass or were not graded; efficiency does not price it4m 59s per pass(1m 41s–20m 47s)59% went to the costliest 10% of attempts (5 of 50)

benchme — intent tasks

too few passes to price

benchme — intent tasks: 0.6% · 163 of 163 graded · too few passes to price

cost
99.0% of this setup's cost here went to runs that did not pass or were not graded; efficiency does not price it$6.86 per passinterval not quotable61% went to the costliest 10% of attempts (17 of 163)
tokens
99.1% of this setup's tokens here went to runs that did not pass or were not graded; efficiency does not price it15.8M per passinterval not quotable77% went to the costliest 10% of attempts (17 of 163)cache split not recorded on 163 attempts
time
99.2% of this setup's time here went to runs that did not pass or were not graded; efficiency does not price it75m 38s per passinterval not quotable70% went to the costliest 10% of attempts (17 of 163)

AIME 2025

44.0 pts

AIME 2025: 53.3% · 30 of 30 graded · scored

cost
1.9× the typical spend per passing run44.8% of this setup's cost here went to runs that did not pass or were not graded; efficiency does not price it (5 such runs' cost were not recorded)$0.31 per pass($0.18–$0.52)32% went to the costliest 10% of attempts (3 of 25)5 attempts' cost were not recorded
tokens
2.9× the typical spend per passing run41.5% of this setup's tokens here went to runs that did not pass or were not graded; efficiency does not price it (5 such runs' tokens were not recorded)588.4k per pass(322.4k–1M)40% went to the costliest 10% of attempts (3 of 25)5 attempts' tokens were not recordedcache split not recorded on 25 attempts
time
1.5× the typical spend per passing run68.5% of this setup's time here went to runs that did not pass or were not graded; efficiency does not price it6m 02s per pass(3m 29s–10m 36s)25% went to the costliest 10% of attempts (3 of 30)

GPQA Diamond

49.1 pts

GPQA Diamond: 66.2% · 198 of 198 graded · scored

cost
1.2× the typical spend per passing run33.9% of this setup's cost here went to runs that did not pass or were not graded; efficiency does not price it (2 such runs' cost were not recorded)$0.07 per pass($0.05–$0.09)49% went to the costliest 10% of attempts (20 of 196)2 attempts' cost were not recorded
tokens
0.76× the typical spend per passing run30.5% of this setup's tokens here went to runs that did not pass or were not graded; efficiency does not price it (2 such runs' tokens were not recorded)105.2k per pass(72.2k–149.7k)69% went to the costliest 10% of attempts (20 of 196)2 attempts' tokens were not recordedcache split not recorded on 196 attempts
time
0.58× the typical spend per passing run44.0% of this setup's time here went to runs that did not pass or were not graded; efficiency does not price it1m 04s per pass(47.1s–1m 24s)58% went to the costliest 10% of attempts (20 of 198)

HumanEval

86.4 pts

HumanEval: 92.7% · 164 of 164 graded · scored

cost
1.0× the typical spend per passing run7.9% of this setup's cost here went to runs that did not pass or were not graded; efficiency does not price it$0.01 per pass($0.01–$0.01)15% went to the costliest 10% of attempts (17 of 164)
tokens
0.38× the typical spend per passing run7.5% of this setup's tokens here went to runs that did not pass or were not graded; efficiency does not price it4.7k per pass(4.5k–4.9k)11% went to the costliest 10% of attempts (17 of 164)cache split not recorded on 164 attempts
time
0.11× the typical spend per passing run9.0% of this setup's time here went to runs that did not pass or were not graded; efficiency does not price it2.1s per pass(1.9s–2.2s)21% went to the costliest 10% of attempts (17 of 164)

Efficiency and spend per pass can order setups differently: efficiency compares the typical (median) passing run, task by task, while spend per pass averages every attempt, so a few very expensive runs, passing or not, count in full; and spend per pass also counts the attempts that failed or did not finish, and divides by the pass rate. A setup is called the most frugal only where both put it first and both separate it from every other setup.

Evidence

Every published study its runs come from.