Skip to content
kalanu
Theme
AIME 2025, benchme — intent tasks, GPQA Diamond and 2 more · 11 setups · 5 of 6 compared benchmarks

Claude Code · claude-haiku-4-5-20251001 · harness default

anthropic

harness default, open-book on knowledge questions

Its score and figures

Every figure the leaderboard's table prints for this setup.

Its score and figures
rank in group4–10
Score61.3 pts58.1–64.5 pts
Pass rate77.9%74.0–81.8%77.9% · 604 of 605 graded
Above chance76.0%71.9–80.1%
Efficiency17.415.9–19.3
cost1.6×
tokens2.0×
time1.3×
Coverage5/6
Runs725

57.0 + 1.8 + 0.8 + 1.7 = 61.3 (outcome + cost + tokens + time)

  • pass rate on each benchmark's own grader
  • cost
  • tokens
  • time
  • not measured
First run
2026-09-29
Last run
2026-10-03
did not finish (scored as a fail)
1
Tasks with no grade
1

On each benchmark

Each benchmark it ran: its status, pass rate, spend and spend per pass. The studies behind them are its evidence, below.

WindTunnel

44.2 pts

WindTunnel: 54.0% · 50 of 50 graded · scored

cost
1.5× the typical spend per passing run67.4% of this setup's cost here went to runs that did not pass or were not graded; efficiency does not price it$0.37 per pass($0.23–$0.61)37% went to the costliest 10% of attempts (5 of 50)
tokens
2.2× the typical spend per passing run70.4% of this setup's tokens here went to runs that did not pass or were not graded; efficiency does not price it2.5M per pass(1.5M–4.3M)40% went to the costliest 10% of attempts (5 of 50)of which 97% cache reads and 2% fresh input
time
1.0× the typical spend per passing run66.5% of this setup's time here went to runs that did not pass or were not graded; efficiency does not price it2m 52s per pass(1m 49s–4m 34s)33% went to the costliest 10% of attempts (5 of 50)

benchme — intent tasks

69.5 pts

benchme — intent tasks: 85.2% · 162 of 163 graded · scored

cost
1.2× the typical spend per passing run5.5% of this setup's cost here went to runs that did not pass or were not graded; efficiency does not price it$0.04 per pass($0.04–$0.05)19% went to the costliest 10% of attempts (17 of 162)
tokens
2.1× the typical spend per passing run3.8% of this setup's tokens here went to runs that did not pass or were not graded; efficiency does not price it232.9k per pass(219.7k–248k)21% went to the costliest 10% of attempts (17 of 162)of which 95% cache reads and 4% fresh input
time
1.3× the typical spend per passing run14.1% of this setup's time here went to runs that did not pass or were not graded; efficiency does not price it43.5s per pass(40.6s–47s)17% went to the costliest 10% of attempts (17 of 162)

AIME 2025

68.3 pts

AIME 2025: 83.3% · 30 of 30 graded · scored

cost
1.5× the typical spend per passing run26.9% of this setup's cost here went to runs that did not pass or were not graded; efficiency does not price it (1 such run's cost was not recorded)$0.16 per pass($0.11–$0.24)35% went to the costliest 10% of attempts (15 of 149)1 attempt's cost was not recorded
tokens
1.1× the typical spend per passing run30.7% of this setup's tokens here went to runs that did not pass or were not graded; efficiency does not price it (1 such run's tokens was not recorded)185.4k per pass(92.6k–327.1k)62% went to the costliest 10% of attempts (15 of 149)1 attempt's tokens was not recordedof which 81% cache reads and 4% fresh input
time
1.4× the typical spend per passing run33.3% of this setup's time here went to runs that did not pass or were not graded; efficiency does not price it3m 46s per pass(2m 25s–5m 39s)38% went to the costliest 10% of attempts (15 of 150)

GPQA Diamond

49.2 pts

GPQA Diamond: 70.7% · 198 of 198 graded · scored

cost
1.7× the typical spend per passing run38.7% of this setup's cost here went to runs that did not pass or were not graded; efficiency does not price it$0.06 per pass($0.05–$0.07)27% went to the costliest 10% of attempts (20 of 198)
tokens
2.1× the typical spend per passing run32.7% of this setup's tokens here went to runs that did not pass or were not graded; efficiency does not price it64k per pass(56.9k–73.1k)20% went to the costliest 10% of attempts (20 of 198)of which 77% cache reads and 7% fresh input
time
1.9× the typical spend per passing run40.1% of this setup's time here went to runs that did not pass or were not graded; efficiency does not price it1m 43s per pass(1m 28s–1m 59s)27% went to the costliest 10% of attempts (20 of 198)

HumanEval

75.4 pts

HumanEval: 96.3% · 164 of 164 graded · scored

cost
2.0× the typical spend per passing run5.0% of this setup's cost here went to runs that did not pass or were not graded; efficiency does not price it$0.01 per pass($0.01–$0.01)23% went to the costliest 10% of attempts (17 of 164)
tokens
3.2× the typical spend per passing run3.7% of this setup's tokens here went to runs that did not pass or were not graded; efficiency does not price it37k per pass(35.9k–38.4k)12% went to the costliest 10% of attempts (17 of 164)of which 88% cache reads and 9% fresh input
time
1.2× the typical spend per passing run4.6% of this setup's time here went to runs that did not pass or were not graded; efficiency does not price it22.7s per pass(21s–24.8s)22% went to the costliest 10% of attempts (17 of 164)

Efficiency and spend per pass can order setups differently: efficiency compares the typical (median) passing run, task by task, while spend per pass averages every attempt, so a few very expensive runs, passing or not, count in full; and spend per pass also counts the attempts that failed or did not finish, and divides by the pass rate. A setup is called the most frugal only where both put it first and both separate it from every other setup.

Evidence

Every published study its runs come from.