Skip to content
kalanu
Theme
AIME 2025, benchme — intent tasks, GPQA Diamond and 2 more · 11 setups · 5 of 6 compared benchmarks

Continue · claude-haiku-4-5-20251001 · harness default

anthropic

harness default, open-book on knowledge questions

Its score and figures

Every figure the leaderboard's table prints for this setup.

Its score and figures
rank in group4–10
Score68.8 pts65.6–72.3 pts
Pass rate78.8%74.5–83.0%78.8% · 595 of 605 graded
Above chance76.5%72.0–80.8%
Efficiency46.042.2–51.6
cost0.76×
tokens0.28×
time0.98×
Coverage5/6
Runs601

57.3 + 3.8 + 5.4 + 2.3 = 68.8 (outcome + cost + tokens + time)

  • pass rate on each benchmark's own grader
  • cost
  • tokens
  • time
  • not measured
First run
2026-10-02
Last run
2026-10-03
did not finish (scored as a fail)
1
Tasks with no grade
10

On each benchmark

Each benchmark it ran: its status, pass rate, spend and spend per pass. The studies behind them are its evidence, below.

WindTunnel

50.7 pts

WindTunnel: 59.6% · 47 of 50 graded · scored

cost
0.81× the typical spend per passing run60.5% of this setup's cost here went to runs that did not pass or were not graded; efficiency does not price it$1.39 per pass($0.79–$2.30)42% went to the costliest 10% of attempts (5 of 47)
tokens
0.20× the typical spend per passing run60.4% of this setup's tokens here went to runs that did not pass or were not graded; efficiency does not price it1.4M per pass(765k–2.2M)43% went to the costliest 10% of attempts (5 of 47)cache split not recorded on 47 attempts
time
0.91× the typical spend per passing run65.8% of this setup's time here went to runs that did not pass or were not graded; efficiency does not price it2m 48s per pass(1m 41s–4m 42s)40% went to the costliest 10% of attempts (5 of 47)

benchme — intent tasks

83.3 pts

benchme — intent tasks: 90.0% · 160 of 163 graded · scored

cost
0.73× the typical spend per passing run7.8% of this setup's cost here went to runs that did not pass or were not graded; efficiency does not price it$0.05 per pass($0.04–$0.07)43% went to the costliest 10% of attempts (16 of 160)
tokens
0.19× the typical spend per passing run7.6% of this setup's tokens here went to runs that did not pass or were not graded; efficiency does not price it50k per pass(40.2k–61.6k)44% went to the costliest 10% of attempts (16 of 160)cache split not recorded on 160 attempts
time
1.0× the typical spend per passing run12.2% of this setup's time here went to runs that did not pass or were not graded; efficiency does not price it36.8s per pass(33.9s–40.2s)19% went to the costliest 10% of attempts (16 of 160)

AIME 2025

72.8 pts

AIME 2025: 86.7% · 30 of 30 graded · scored

cost
1.1× the typical spend per passing run6.4% of this setup's cost here went to runs that did not pass or were not graded; efficiency does not price it (1 such run's cost was not recorded)$0.16 per pass($0.09–$0.24)45% went to the costliest 10% of attempts (3 of 29)1 attempt's cost was not recorded
tokens
0.80× the typical spend per passing run5.6% of this setup's tokens here went to runs that did not pass or were not graded; efficiency does not price it (1 such run's tokens was not recorded)116.2k per pass(61.8k–185.1k)49% went to the costliest 10% of attempts (3 of 29)1 attempt's tokens was not recordedcache split not recorded on 29 attempts
time
1.0× the typical spend per passing run20.5% of this setup's time here went to runs that did not pass or were not graded; efficiency does not price it2m 13s per pass(1m 25s–3m 19s)36% went to the costliest 10% of attempts (3 of 30)

GPQA Diamond

52.4 pts

GPQA Diamond: 64.6% · 195 of 198 graded · scored

cost
0.60× the typical spend per passing run30.9% of this setup's cost here went to runs that did not pass or were not graded; efficiency does not price it (1 such run's cost was not recorded)$0.03 per pass($0.02–$0.04)50% went to the costliest 10% of attempts (20 of 195)
tokens
0.25× the typical spend per passing run29.2% of this setup's tokens here went to runs that did not pass or were not graded; efficiency does not price it (1 such run's tokens was not recorded)19.3k per pass(13.7k–27.1k)55% went to the costliest 10% of attempts (20 of 195)cache split not recorded on 195 attempts
time
1.0× the typical spend per passing run35.0% of this setup's time here went to runs that did not pass or were not graded; efficiency does not price it45.4s per pass(39.6s–53.7s)28% went to the costliest 10% of attempts (20 of 195)

HumanEval

85.0 pts

HumanEval: 93.3% · 163 of 164 graded · scored

cost
0.63× the typical spend per passing run7.3% of this setup's cost here went to runs that did not pass or were not graded; efficiency does not price it (1 such run's cost was not recorded)$0.0038 per pass($0.0036–$0.0040)15% went to the costliest 10% of attempts (17 of 163)
tokens
0.23× the typical spend per passing run6.9% of this setup's tokens here went to runs that did not pass or were not graded; efficiency does not price it (1 such run's tokens was not recorded)2.8k per pass(2.6k–2.9k)13% went to the costliest 10% of attempts (17 of 163)cache split not recorded on 163 attempts
time
1.0× the typical spend per passing run7.6% of this setup's time here went to runs that did not pass or were not graded; efficiency does not price it16.4s per pass(15.7s–17.2s)12% went to the costliest 10% of attempts (17 of 163)

Efficiency and spend per pass can order setups differently: efficiency compares the typical (median) passing run, task by task, while spend per pass averages every attempt, so a few very expensive runs, passing or not, count in full; and spend per pass also counts the attempts that failed or did not finish, and divides by the pass rate. A setup is called the most frugal only where both put it first and both separate it from every other setup.

Evidence

Every published study its runs come from.