Skip to content
kalanu
Theme
AIME 2025, benchme — intent tasks, GPQA Diamond and 2 more · 11 setups · 5 of 6 compared benchmarks

Claude Agent SDK · claude-haiku-4-5-20251001 · harness default

anthropic

harness default, open-book on knowledge questions

Its score and figures

Every figure the leaderboard's table prints for this setup.

Its score and figures
rank in group4–10
Score63.8 pts60.1–67.4 pts
Pass rate79.5%75.1–84.0%79.5% · 602 of 605 graded
Above chance77.3%72.6–81.8%
Efficiency23.320.8–26.5
cost1.5×
tokens1.8×
time0.79×
Coverage5/6
Runs605

58.0 + 2.0 + 0.9 + 2.9 = 63.8 (outcome + cost + tokens + time)

  • pass rate on each benchmark's own grader
  • cost
  • tokens
  • time
  • not measured
First run
2026-10-02
Last run
2026-10-03
did not finish (scored as a fail)
3
Tasks with no grade
3

On each benchmark

Each benchmark it ran: its status, pass rate, spend and spend per pass. The studies behind them are its evidence, below.

WindTunnel

54.0 pts

WindTunnel: 65.3% · 49 of 50 graded · scored

cost
1.9× the typical spend per passing run52.3% of this setup's cost here went to runs that did not pass or were not graded; efficiency does not price it$0.38 per pass($0.24–$0.57)35% went to the costliest 10% of attempts (5 of 49)
tokens
2.3× the typical spend per passing run54.5% of this setup's tokens here went to runs that did not pass or were not graded; efficiency does not price it2.5M per pass(1.6M–4M)38% went to the costliest 10% of attempts (5 of 49)cache split not recorded on 49 attempts
time
0.68× the typical spend per passing run67.8% of this setup's time here went to runs that did not pass or were not graded; efficiency does not price it3m 51s per pass(2m 18s–5m 59s)35% went to the costliest 10% of attempts (5 of 49)

benchme — intent tasks

77.6 pts

benchme — intent tasks: 92.5% · 161 of 163 graded · scored

cost
1.1× the typical spend per passing run5.1% of this setup's cost here went to runs that did not pass or were not graded; efficiency does not price it$0.04 per pass($0.04–$0.04)18% went to the costliest 10% of attempts (17 of 161)
tokens
1.8× the typical spend per passing run4.4% of this setup's tokens here went to runs that did not pass or were not graded; efficiency does not price it195.1k per pass(185.3k–205.6k)19% went to the costliest 10% of attempts (17 of 161)cache split not recorded on 161 attempts
time
0.62× the typical spend per passing run5.9% of this setup's time here went to runs that did not pass or were not graded; efficiency does not price it22.6s per pass(19.8s–26.6s)29% went to the costliest 10% of attempts (17 of 161)

AIME 2025

65.0 pts

AIME 2025: 76.7% · 30 of 30 graded · scored

cost
1.1× the typical spend per passing run38.9% of this setup's cost here went to runs that did not pass or were not graded; efficiency does not price it (2 such runs' cost were not recorded)$0.16 per pass($0.10–$0.25)33% went to the costliest 10% of attempts (3 of 28)2 attempts' cost were not recorded
tokens
0.98× the typical spend per passing run43.2% of this setup's tokens here went to runs that did not pass or were not graded; efficiency does not price it (2 such runs' tokens were not recorded)155.3k per pass(72.6k–295.5k)54% went to the costliest 10% of attempts (3 of 28)2 attempts' tokens were not recordedcache split not recorded on 28 attempts
time
1.0× the typical spend per passing run52.4% of this setup's time here went to runs that did not pass or were not graded; efficiency does not price it3m 23s per pass(2m 03s–5m 26s)30% went to the costliest 10% of attempts (3 of 30)

GPQA Diamond

45.6 pts

GPQA Diamond: 66.7% · 198 of 198 graded · scored

cost
1.7× the typical spend per passing run47.6% of this setup's cost here went to runs that did not pass or were not graded; efficiency does not price it (1 such run's cost was not recorded)$0.07 per pass($0.06–$0.09)29% went to the costliest 10% of attempts (20 of 197)1 attempt's cost was not recorded
tokens
1.9× the typical spend per passing run36.1% of this setup's tokens here went to runs that did not pass or were not graded; efficiency does not price it (1 such run's tokens was not recorded)60.3k per pass(53.7k–68.8k)19% went to the costliest 10% of attempts (20 of 197)1 attempt's tokens was not recordedcache split not recorded on 197 attempts
time
1.4× the typical spend per passing run52.0% of this setup's time here went to runs that did not pass or were not graded; efficiency does not price it1m 49s per pass(1m 29s–2m 12s)35% went to the costliest 10% of attempts (20 of 198)

HumanEval

76.6 pts

HumanEval: 96.3% · 164 of 164 graded · scored

cost
2.0× the typical spend per passing run6.8% of this setup's cost here went to runs that did not pass or were not graded; efficiency does not price it$0.01 per pass($0.01–$0.01)24% went to the costliest 10% of attempts (17 of 164)
tokens
2.7× the typical spend per passing run3.9% of this setup's tokens here went to runs that did not pass or were not graded; efficiency does not price it31.9k per pass(30.9k–33.1k)11% went to the costliest 10% of attempts (17 of 164)cache split not recorded on 164 attempts
time
0.49× the typical spend per passing run8.6% of this setup's time here went to runs that did not pass or were not graded; efficiency does not price it10.4s per pass(8.9s–12.2s)31% went to the costliest 10% of attempts (17 of 164)

Efficiency and spend per pass can order setups differently: efficiency compares the typical (median) passing run, task by task, while spend per pass averages every attempt, so a few very expensive runs, passing or not, count in full; and spend per pass also counts the attempts that failed or did not finish, and divides by the pass rate. A setup is called the most frugal only where both put it first and both separate it from every other setup.

Evidence

Every published study its runs come from.