Skip to content
kalanu
Theme
AIME 2025, benchme — intent tasks, GPQA Diamond and 2 more · 11 setups · 5 of 6 compared benchmarks

opencode · claude-haiku-4-5-20251001 · harness default

anthropic

harness default, open-book on knowledge questions

Its score and figures

Every figure the leaderboard's table prints for this setup.

Its score and figures
rank in group4–10
Score61.7 pts58.0–65.4 pts
Pass rate72.0%67.3–76.8%72.0% · 603 of 605 graded
Above chance69.0%64.2–73.9%
Efficiency39.837.6–43.0
cost0.57×
tokens0.69×
time1.1×
Coverage5/6
Runs605

51.7 + 5.8 + 2.1 + 2.1 = 61.7 (outcome + cost + tokens + time)

  • pass rate on each benchmark's own grader
  • cost
  • tokens
  • time
  • not measured
First run
2026-10-02
Last run
2026-10-03
did not finish (scored as a fail)
0
Tasks with no grade
2

On each benchmark

Each benchmark it ran: its status, pass rate, spend and spend per pass. The studies behind them are its evidence, below.

WindTunnel

47.9 pts

WindTunnel: 58.0% · 50 of 50 graded · scored

cost
1.0× the typical spend per passing run59.7% of this setup's cost here went to runs that did not pass or were not graded; efficiency does not price it$0.31 per pass($0.19–$0.50)34% went to the costliest 10% of attempts (5 of 50)
tokens
0.55× the typical spend per passing run61.4% of this setup's tokens here went to runs that did not pass or were not graded; efficiency does not price it1.9M per pass(1.1M–3.1M)37% went to the costliest 10% of attempts (5 of 50)cache split not recorded on 50 attempts
time
1.1× the typical spend per passing run57.3% of this setup's time here went to runs that did not pass or were not graded; efficiency does not price it3m 21s per pass(2m 05s–5m 09s)32% went to the costliest 10% of attempts (5 of 50)

benchme — intent tasks

68.7 pts

benchme — intent tasks: 72.8% · 162 of 163 graded · scored

cost
0.42× the typical spend per passing run21.6% of this setup's cost here went to runs that did not pass or were not graded; efficiency does not price it$0.02 per pass($0.02–$0.02)32% went to the costliest 10% of attempts (17 of 162)
tokens
0.49× the typical spend per passing run21.6% of this setup's tokens here went to runs that did not pass or were not graded; efficiency does not price it76.9k per pass(65.1k–91.5k)35% went to the costliest 10% of attempts (17 of 162)cache split not recorded on 162 attempts
time
1.1× the typical spend per passing run30.5% of this setup's time here went to runs that did not pass or were not graded; efficiency does not price it44.5s per pass(40.1s–50.1s)17% went to the costliest 10% of attempts (17 of 162)

AIME 2025

66.6 pts

AIME 2025: 80.0% · 30 of 30 graded · scored

cost
1.0× the typical spend per passing run23.8% of this setup's cost here went to runs that did not pass or were not graded; efficiency does not price it$0.13 per pass($0.07–$0.20)44% went to the costliest 10% of attempts (3 of 30)
tokens
1.4× the typical spend per passing run18.1% of this setup's tokens here went to runs that did not pass or were not graded; efficiency does not price it333k per pass(147.9k–574.5k)59% went to the costliest 10% of attempts (3 of 30)cache split not recorded on 30 attempts
time
1.2× the typical spend per passing run16.5% of this setup's time here went to runs that did not pass or were not graded; efficiency does not price it3m 58s per pass(1m 50s–6m 28s)55% went to the costliest 10% of attempts (3 of 30)

GPQA Diamond

43.3 pts

GPQA Diamond: 55.1% · 198 of 198 graded · scored

cost
0.32× the typical spend per passing run32.7% of this setup's cost here went to runs that did not pass or were not graded; efficiency does not price it$0.01 per pass($0.01–$0.02)51% went to the costliest 10% of attempts (20 of 198)
tokens
0.56× the typical spend per passing run37.0% of this setup's tokens here went to runs that did not pass or were not graded; efficiency does not price it25.6k per pass(20.5k–32.5k)44% went to the costliest 10% of attempts (20 of 198)cache split not recorded on 198 attempts
time
1.0× the typical spend per passing run40.9% of this setup's time here went to runs that did not pass or were not graded; efficiency does not price it46.1s per pass(40.2s–53.3s)23% went to the costliest 10% of attempts (20 of 198)

HumanEval

81.8 pts

HumanEval: 93.9% · 163 of 164 graded · scored

cost
0.45× the typical spend per passing run6.9% of this setup's cost here went to runs that did not pass or were not graded; efficiency does not price it (1 such run's cost was not recorded)$0.0028 per pass($0.0026–$0.0030)17% went to the costliest 10% of attempts (17 of 163)
tokens
0.75× the typical spend per passing run6.1% of this setup's tokens here went to runs that did not pass or were not graded; efficiency does not price it (1 such run's tokens was not recorded)9.1k per pass(8.7k–9.6k)12% went to the costliest 10% of attempts (17 of 163)cache split not recorded on 163 attempts
time
1.1× the typical spend per passing run15.1% of this setup's time here went to runs that did not pass or were not graded; efficiency does not price it19.1s per pass(18.3s–20s)13% went to the costliest 10% of attempts (17 of 163)

Efficiency and spend per pass can order setups differently: efficiency compares the typical (median) passing run, task by task, while spend per pass averages every attempt, so a few very expensive runs, passing or not, count in full; and spend per pass also counts the attempts that failed or did not finish, and divides by the pass rate. A setup is called the most frugal only where both put it first and both separate it from every other setup.

Evidence

Every published study its runs come from.