Skip to content
kalanu
Theme
AIME 2025, benchme — intent tasks, GPQA Diamond and 2 more · 11 setups · 5 of 6 compared benchmarks

DeepSeek Harness · claude-haiku-4-5-20251001 · harness default

anthropic

harness default, open-book on knowledge questions

Its score and figures

Every figure the leaderboard's table prints for this setup.

Its score and figures
rank in group4–10
Score69.1 pts65.8–72.1 pts
Pass rate83.8%79.8–87.4%83.8% · 600 of 605 graded
Above chance81.6%77.4–85.6%
Efficiency31.728.8–36.2
cost1.0×
tokens1.5×
time0.65×
Coverage5/6
Runs602

61.2 + 3.4 + 1.0 + 3.5 = 69.1 (outcome + cost + tokens + time)

  • pass rate on each benchmark's own grader
  • cost
  • tokens
  • time
  • not measured
First run
2026-10-02
Last run
2026-10-03
did not finish (scored as a fail)
1
Tasks with no grade
5

On each benchmark

Each benchmark it ran: its status, pass rate, spend and spend per pass. The studies behind them are its evidence, below.

WindTunnel

59.6 pts

WindTunnel: 72.9% · 48 of 50 graded · scored

cost
1.4× the typical spend per passing run51.5% of this setup's cost here went to runs that did not pass or were not graded; efficiency does not price it$0.48 per pass($0.31–$0.73)35% went to the costliest 10% of attempts (5 of 48)
tokens
1.4× the typical spend per passing run52.8% of this setup's tokens here went to runs that did not pass or were not graded; efficiency does not price it3.1M per pass(1.9M–4.8M)37% went to the costliest 10% of attempts (5 of 48)cache split not recorded on 48 attempts
time
0.77× the typical spend per passing run50.4% of this setup's time here went to runs that did not pass or were not graded; efficiency does not price it3m 13s per pass(2m 03s–4m 50s)34% went to the costliest 10% of attempts (5 of 48)

benchme — intent tasks

88.3 pts

benchme — intent tasks: 100.0% · 161 of 163 graded · scored

cost
0.63× the typical spend per passing run0.9% of this setup's cost here went to runs that did not pass or were not graded; efficiency does not price it$0.03 per pass($0.02–$0.03)26% went to the costliest 10% of attempts (17 of 161)
tokens
0.87× the typical spend per passing run0.9% of this setup's tokens here went to runs that did not pass or were not graded; efficiency does not price it121.5k per pass(106.6k–137.4k)28% went to the costliest 10% of attempts (17 of 161)cache split not recorded on 161 attempts
time
0.44× the typical spend per passing run0.8% of this setup's time here went to runs that did not pass or were not graded; efficiency does not price it18.8s per pass(15.9s–22.6s)34% went to the costliest 10% of attempts (17 of 161)

AIME 2025

70.1 pts

AIME 2025: 86.7% · 30 of 30 graded · scored

cost
1.5× the typical spend per passing run19.7% of this setup's cost here went to runs that did not pass or were not graded; efficiency does not price it (1 such run's cost was not recorded)$0.15 per pass($0.10–$0.21)33% went to the costliest 10% of attempts (3 of 29)1 attempt's cost was not recorded
tokens
2.9× the typical spend per passing run21.2% of this setup's tokens here went to runs that did not pass or were not graded; efficiency does not price it (1 such run's tokens was not recorded)381.7k per pass(244.7k–572k)35% went to the costliest 10% of attempts (3 of 29)1 attempt's tokens was not recordedcache split not recorded on 29 attempts
time
1.1× the typical spend per passing run57.6% of this setup's time here went to runs that did not pass or were not graded; efficiency does not price it4m 11s per pass(1m 45s–7m 54s)56% went to the costliest 10% of attempts (3 of 30)

GPQA Diamond

46.4 pts

GPQA Diamond: 67.2% · 198 of 198 graded · scored

cost
1.5× the typical spend per passing run36.1% of this setup's cost here went to runs that did not pass or were not graded; efficiency does not price it$0.07 per pass($0.05–$0.08)34% went to the costliest 10% of attempts (20 of 198)
tokens
2.5× the typical spend per passing run33.7% of this setup's tokens here went to runs that did not pass or were not graded; efficiency does not price it137.9k per pass(111.4k–173.2k)41% went to the costliest 10% of attempts (20 of 198)cache split not recorded on 198 attempts
time
1.2× the typical spend per passing run37.7% of this setup's time here went to runs that did not pass or were not graded; efficiency does not price it1m 15s per pass(1m 03s–1m 30s)33% went to the costliest 10% of attempts (20 of 198)

HumanEval

81.0 pts

HumanEval: 92.0% · 163 of 164 graded · scored

cost
0.55× the typical spend per passing run9.6% of this setup's cost here went to runs that did not pass or were not graded; efficiency does not price it$0.0037 per pass($0.0034–$0.0041)19% went to the costliest 10% of attempts (17 of 163)
tokens
1.0× the typical spend per passing run8.4% of this setup's tokens here went to runs that did not pass or were not graded; efficiency does not price it12.6k per pass(11.8k–13.6k)14% went to the costliest 10% of attempts (17 of 163)cache split not recorded on 163 attempts
time
0.26× the typical spend per passing run9.1% of this setup's time here went to runs that did not pass or were not graded; efficiency does not price it5s per pass(4.6s–5.4s)19% went to the costliest 10% of attempts (17 of 163)

Efficiency and spend per pass can order setups differently: efficiency compares the typical (median) passing run, task by task, while spend per pass averages every attempt, so a few very expensive runs, passing or not, count in full; and spend per pass also counts the attempts that failed or did not finish, and divides by the pass rate. A setup is called the most frugal only where both put it first and both separate it from every other setup.

Evidence

Every published study its runs come from.