Skip to content
kalanu
Theme
GPQA Diamond, HumanEval and WindTunnel only · 1 setup · 3 of 6 compared benchmarks

Aider · claude-haiku-4-5-20251001 · harness default

anthropic

harness default, open-book on knowledge questions

Its score and figures

Every figure the leaderboard's table prints for this setup.

Its score and figures
rank in group–
Score63.0 pts58.5–67.5 pts
Pass rate65.2%59.8–70.8%65.2% · 602 of 605 graded
Above chance60.8%55.0–66.9%
Efficiency69.568.0–70.3
cost0.31×
tokens0.12×
time1.1×
Coverage3/6
Runs605

45.6 + 7.3 + 8.1 + 2.0 = 63.0 (outcome + cost + tokens + time)

  • pass rate on each benchmark's own grader
  • cost
  • tokens
  • time
  • not measured
First run
2026-10-02
Last run
2026-10-03
did not finish (scored as a fail)
2
Tasks with no grade
3

On each benchmark

Each benchmark it ran: its status, pass rate, spend and spend per pass. The studies behind them are its evidence, below.

WindTunnel

51.8 pts

WindTunnel: 42.9% · 49 of 50 graded · scored

cost
0.14× the typical spend per passing run61.8% of this setup's cost here went to runs that did not pass or were not graded; efficiency does not price it (1 such run's cost was not recorded)$0.01 per pass($0.01–$0.01)18% went to the costliest 10% of attempts (5 of 49)
tokens
0.034× the typical spend per passing run58.3% of this setup's tokens here went to runs that did not pass or were not graded; efficiency does not price it (1 such run's tokens was not recorded)8.1k per pass(6k–11.9k)19% went to the costliest 10% of attempts (5 of 49)cache split not recorded on 49 attempts
time
1.0× the typical spend per passing run61.8% of this setup's time here went to runs that did not pass or were not graded; efficiency does not price it1m 12s per pass(53.9s–1m 49s)17% went to the costliest 10% of attempts (5 of 49)

benchme — intent tasks

too few passes to price

benchme — intent tasks: 2.5% · 161 of 163 graded · too few passes to price

cost
98.1% of this setup's cost here went to runs that did not pass or were not graded; efficiency does not price it$0.16 per pass($0.08–$0.63)12% went to the costliest 10% of attempts (17 of 161)
tokens
97.7% of this setup's tokens here went to runs that did not pass or were not graded; efficiency does not price it116.8k per pass(58k–469.9k)11% went to the costliest 10% of attempts (17 of 161)cache split not recorded on 161 attempts
time
97.9% of this setup's time here went to runs that did not pass or were not graded; efficiency does not price it20m 48s per pass(9m 59s–85m 11s)20% went to the costliest 10% of attempts (17 of 161)

AIME 2025

too few passes to price

AIME 2025: 26.7% · 30 of 30 graded · too few passes to price

cost
83.9% of this setup's cost here went to runs that did not pass or were not graded; efficiency does not price it$0.05 per pass($0.03–$0.14)40% went to the costliest 10% of attempts (3 of 30)
tokens
79.5% of this setup's tokens here went to runs that did not pass or were not graded; efficiency does not price it18.3k per pass(10k–43.9k)28% went to the costliest 10% of attempts (3 of 30)cache split not recorded on 30 attempts
time
89.4% of this setup's time here went to runs that did not pass or were not graded; efficiency does not price it4m 07s per pass(1m 36s–12m 11s)54% went to the costliest 10% of attempts (3 of 30)

GPQA Diamond

54.1 pts

GPQA Diamond: 60.6% · 198 of 198 graded · scored

cost
0.33× the typical spend per passing run41.1% of this setup's cost here went to runs that did not pass or were not graded; efficiency does not price it$0.01 per pass($0.01–$0.01)25% went to the costliest 10% of attempts (20 of 198)
tokens
0.18× the typical spend per passing run39.4% of this setup's tokens here went to runs that did not pass or were not graded; efficiency does not price it5.8k per pass(4.9k–6.8k)20% went to the costliest 10% of attempts (20 of 198)cache split not recorded on 198 attempts
time
1.0× the typical spend per passing run40.6% of this setup's time here went to runs that did not pass or were not graded; efficiency does not price it38.6s per pass(34.3s–44.2s)16% went to the costliest 10% of attempts (20 of 198)

HumanEval

83.0 pts

HumanEval: 92.1% · 164 of 164 graded · scored

cost
0.66× the typical spend per passing run8.3% of this setup's cost here went to runs that did not pass or were not graded; efficiency does not price it$0.0039 per pass($0.0037–$0.0041)15% went to the costliest 10% of attempts (17 of 164)
tokens
0.25× the typical spend per passing run8.0% of this setup's tokens here went to runs that did not pass or were not graded; efficiency does not price it3k per pass(2.9k–3.2k)12% went to the costliest 10% of attempts (17 of 164)cache split not recorded on 164 attempts
time
1.2× the typical spend per passing run7.7% of this setup's time here went to runs that did not pass or were not graded; efficiency does not price it21.1s per pass(19.8s–22.7s)15% went to the costliest 10% of attempts (17 of 164)

Efficiency and spend per pass can order setups differently: efficiency compares the typical (median) passing run, task by task, while spend per pass averages every attempt, so a few very expensive runs, passing or not, count in full; and spend per pass also counts the attempts that failed or did not finish, and divides by the pass rate. A setup is called the most frugal only where both put it first and both separate it from every other setup.

Evidence

Every published study its runs come from.