Skip to content
kalanu
Theme
AIME 2025, benchme — intent tasks, GPQA Diamond and 2 more · 11 setups · 5 of 6 compared benchmarks

OpenClaw · claude-haiku-4-5-20251001 · harness default

anthropic

harness default, open-book on knowledge questions

Its score and figures

Every figure the leaderboard's table prints for this setup.

Its score and figures
rank in group11
Score45.6 pts41.9–48.9 pts
Pass rate57.5%53.0–61.7%57.5% · 601 of 605 graded
Above chance55.4%50.7–59.8%
Efficiency16.014.6–17.4
cost1.3×
tokens2.2×
time2.4×
Coverage5/6
Runs604

41.6 + 2.2 + 0.8 + 1.0 = 45.6 (outcome + cost + tokens + time)

  • pass rate on each benchmark's own grader
  • cost
  • tokens
  • time
  • not measured
First run
2026-10-02
Last run
2026-10-03
did not finish (scored as a fail)
9
Tasks with no grade
4

On each benchmark

Each benchmark it ran: its status, pass rate, spend and spend per pass. The studies behind them are its evidence, below.

WindTunnel

27.7 pts

WindTunnel: 34.0% · 50 of 50 graded · scored

cost
1.7× the typical spend per passing run72.1% of this setup's cost here went to runs that did not pass or were not graded; efficiency does not price it$0.56 per pass($0.33–$1.04)33% went to the costliest 10% of attempts (5 of 50)
tokens
2.6× the typical spend per passing run74.0% of this setup's tokens here went to runs that did not pass or were not graded; efficiency does not price it4M per pass(2.3M–7.4M)35% went to the costliest 10% of attempts (5 of 50)cache split not recorded on 50 attempts
time
2.1× the typical spend per passing run88.3% of this setup's time here went to runs that did not pass or were not graded; efficiency does not price it11m 41s per pass(4m 59s–25m 49s)63% went to the costliest 10% of attempts (5 of 50)

benchme — intent tasks

17.1 pts

benchme — intent tasks: 17.4% · 161 of 163 graded · scored

cost
1.4× the typical spend per passing run72.0% of this setup's cost here went to runs that did not pass or were not graded; efficiency does not price it (1 such run's cost was not recorded)$0.15 per pass($0.12–$0.21)19% went to the costliest 10% of attempts (16 of 160)1 attempt's cost was not recorded
tokens
2.9× the typical spend per passing run72.6% of this setup's tokens here went to runs that did not pass or were not graded; efficiency does not price it (1 such run's tokens was not recorded)1M per pass(794.3k–1.4M)19% went to the costliest 10% of attempts (16 of 160)1 attempt's tokens was not recordedcache split not recorded on 160 attempts
time
2.4× the typical spend per passing run89.6% of this setup's time here went to runs that did not pass or were not graded; efficiency does not price it11m 01s per pass(6m 17s–18m 52s)55% went to the costliest 10% of attempts (17 of 161)

AIME 2025

59.3 pts

AIME 2025: 70.0% · 30 of 30 graded · scored

cost
0.90× the typical spend per passing run46.4% of this setup's cost here went to runs that did not pass or were not graded; efficiency does not price it$0.10 per pass($0.07–$0.16)25% went to the costliest 10% of attempts (3 of 30)
tokens
1.0× the typical spend per passing run52.6% of this setup's tokens here went to runs that did not pass or were not graded; efficiency does not price it122.6k per pass(61.2k–255.1k)50% went to the costliest 10% of attempts (3 of 30)cache split not recorded on 30 attempts
time
1.9× the typical spend per passing run44.7% of this setup's time here went to runs that did not pass or were not graded; efficiency does not price it3m 07s per pass(2m 11s–4m 39s)22% went to the costliest 10% of attempts (3 of 30)

GPQA Diamond

47.2 pts

GPQA Diamond: 68.2% · 198 of 198 graded · scored

cost
1.1× the typical spend per passing run42.8% of this setup's cost here went to runs that did not pass or were not graded; efficiency does not price it$0.04 per pass($0.04–$0.05)30% went to the costliest 10% of attempts (20 of 198)
tokens
2.0× the typical spend per passing run35.3% of this setup's tokens here went to runs that did not pass or were not graded; efficiency does not price it70.7k per pass(59.6k–86.6k)24% went to the costliest 10% of attempts (20 of 198)cache split not recorded on 198 attempts
time
2.9× the typical spend per passing run37.3% of this setup's time here went to runs that did not pass or were not graded; efficiency does not price it2m 04s per pass(1m 49s–2m 22s)23% went to the costliest 10% of attempts (20 of 198)

HumanEval

76.6 pts

HumanEval: 98.1% · 162 of 164 graded · scored

cost
1.7× the typical spend per passing run2.2% of this setup's cost here went to runs that did not pass or were not graded; efficiency does not price it (1 such run's cost was not recorded)$0.01 per pass($0.01–$0.01)21% went to the costliest 10% of attempts (17 of 162)
tokens
3.4× the typical spend per passing run1.9% of this setup's tokens here went to runs that did not pass or were not graded; efficiency does not price it (1 such run's tokens was not recorded)38.8k per pass(38.1k–39.9k)11% went to the costliest 10% of attempts (17 of 162)cache split not recorded on 162 attempts
time
3.1× the typical spend per passing run2.7% of this setup's time here went to runs that did not pass or were not graded; efficiency does not price it53s per pass(51.2s–55.2s)15% went to the costliest 10% of attempts (17 of 162)

Efficiency and spend per pass can order setups differently: efficiency compares the typical (median) passing run, task by task, while spend per pass averages every attempt, so a few very expensive runs, passing or not, count in full; and spend per pass also counts the attempts that failed or did not finish, and divides by the pass rate. A setup is called the most frugal only where both put it first and both separate it from every other setup.

Evidence

Every published study its runs come from.