Skip to content
kalanu
Theme
AIME 2025, benchme — intent tasks, GPQA Diamond and 2 more · 11 setups · 5 of 6 compared benchmarks

eve · claude-haiku-4-5-20251001 · harness default

anthropic

harness default, open-book on knowledge questions

Its score and figures

Every figure the leaderboard's table prints for this setup.

Its score and figures
rank in group4–10
Score68.6 pts65.2–71.8 pts
Pass rate81.9%77.7–85.7%81.9% · 604 of 605 graded
Above chance79.6%75.2–83.6%
Efficiency35.633.0–39.7
cost1.1×
tokens0.71×
time0.59×
Coverage5/6
Runs605

59.7 + 2.7 + 2.4 + 3.7 ≈ 68.6 (outcome + cost + tokens + time)

  • pass rate on each benchmark's own grader
  • cost
  • tokens
  • time
  • not measured
First run
2026-10-02
Last run
2026-10-03
did not finish (scored as a fail)
6
Tasks with no grade
1

On each benchmark

Each benchmark it ran: its status, pass rate, spend and spend per pass. The studies behind them are its evidence, below.

WindTunnel

54.0 pts

WindTunnel: 66.0% · 50 of 50 graded · scored

cost
1.2× the typical spend per passing run48.1% of this setup's cost here went to runs that did not pass or were not graded; efficiency does not price it$0.21 per pass($0.14–$0.29)27% went to the costliest 10% of attempts (5 of 50)
tokens
1.0× the typical spend per passing run50.2% of this setup's tokens here went to runs that did not pass or were not graded; efficiency does not price it1M per pass(693.2k–1.5M)31% went to the costliest 10% of attempts (5 of 50)cache split not recorded on 50 attempts
time
0.90× the typical spend per passing run76.3% of this setup's time here went to runs that did not pass or were not graded; efficiency does not price it6m 22s per pass(3m 18s–11m 21s)52% went to the costliest 10% of attempts (5 of 50)

benchme — intent tasks

88.0 pts

benchme — intent tasks: 99.4% · 162 of 163 graded · scored

cost
0.86× the typical spend per passing run2.6% of this setup's cost here went to runs that did not pass or were not graded; efficiency does not price it$0.03 per pass($0.03–$0.03)22% went to the costliest 10% of attempts (17 of 162)
tokens
0.46× the typical spend per passing run4.2% of this setup's tokens here went to runs that did not pass or were not graded; efficiency does not price it65.5k per pass(54.8k–79k)36% went to the costliest 10% of attempts (17 of 162)cache split not recorded on 162 attempts
time
0.44× the typical spend per passing run2.4% of this setup's time here went to runs that did not pass or were not graded; efficiency does not price it18s per pass(15s–22.7s)32% went to the costliest 10% of attempts (17 of 162)

AIME 2025

70.2 pts

AIME 2025: 83.3% · 30 of 30 graded · scored

cost
1.0× the typical spend per passing run16.9% of this setup's cost here went to runs that did not pass or were not graded; efficiency does not price it$0.09 per pass($0.07–$0.13)28% went to the costliest 10% of attempts (3 of 30)
tokens
1.0× the typical spend per passing run13.7% of this setup's tokens here went to runs that did not pass or were not graded; efficiency does not price it147.2k per pass(95.8k–209.5k)34% went to the costliest 10% of attempts (3 of 30)cache split not recorded on 30 attempts
time
0.95× the typical spend per passing run43.0% of this setup's time here went to runs that did not pass or were not graded; efficiency does not price it2m 37s per pass(1m 28s–4m 13s)37% went to the costliest 10% of attempts (3 of 30)

GPQA Diamond

46.1 pts

GPQA Diamond: 65.2% · 198 of 198 graded · scored

cost
1.4× the typical spend per passing run43.5% of this setup's cost here went to runs that did not pass or were not graded; efficiency does not price it$0.06 per pass($0.05–$0.07)39% went to the costliest 10% of attempts (20 of 198)
tokens
0.99× the typical spend per passing run47.5% of this setup's tokens here went to runs that did not pass or were not graded; efficiency does not price it56.6k per pass(43.7k–73.6k)47% went to the costliest 10% of attempts (20 of 198)cache split not recorded on 198 attempts
time
0.88× the typical spend per passing run45.9% of this setup's time here went to runs that did not pass or were not graded; efficiency does not price it59.9s per pass(48s–1m 15s)45% went to the costliest 10% of attempts (20 of 198)

HumanEval

84.6 pts

HumanEval: 95.7% · 164 of 164 graded · scored

cost
1.0× the typical spend per passing run4.7% of this setup's cost here went to runs that did not pass or were not graded; efficiency does not price it$0.01 per pass($0.01–$0.01)19% went to the costliest 10% of attempts (17 of 164)
tokens
0.38× the typical spend per passing run4.9% of this setup's tokens here went to runs that did not pass or were not graded; efficiency does not price it5.1k per pass(4.8k–5.5k)20% went to the costliest 10% of attempts (17 of 164)cache split not recorded on 164 attempts
time
0.21× the typical spend per passing run5.3% of this setup's time here went to runs that did not pass or were not graded; efficiency does not price it4.1s per pass(3.8s–4.4s)20% went to the costliest 10% of attempts (17 of 164)

Efficiency and spend per pass can order setups differently: efficiency compares the typical (median) passing run, task by task, while spend per pass averages every attempt, so a few very expensive runs, passing or not, count in full; and spend per pass also counts the attempts that failed or did not finish, and divides by the pass rate. A setup is called the most frugal only where both put it first and both separate it from every other setup.

Evidence

Every published study its runs come from.