Skip to content
kalanu
Theme
AIME 2025, benchme — intent tasks, GPQA Diamond and 2 more · 11 setups · 5 of 6 compared benchmarks

Hermes · claude-haiku-4-5-20251001 · harness default

anthropic

harness default, open-book on knowledge questions

Its score and figures

Every figure the leaderboard's table prints for this setup.

Its score and figures
rank in group4–10
Score65.4 pts61.8–68.7 pts
Pass rate80.7%76.2–85.1%80.7% · 602 of 605 graded
Above chance78.5%73.7–82.9%
Efficiency26.124.3–27.7
cost0.81×
tokens1.4×
time1.8×
Coverage5/6
Runs604

58.9 + 4.2 + 1.1 + 1.2 = 65.4 (outcome + cost + tokens + time)

  • pass rate on each benchmark's own grader
  • cost
  • tokens
  • time
  • not measured
First run
2026-10-02
Last run
2026-10-03
did not finish (scored as a fail)
3
Tasks with no grade
3

On each benchmark

Each benchmark it ran: its status, pass rate, spend and spend per pass. The studies behind them are its evidence, below.

WindTunnel

60.5 pts

WindTunnel: 77.6% · 49 of 50 graded · scored

cost
1.7× the typical spend per passing run49.3% of this setup's cost here went to runs that did not pass or were not graded; efficiency does not price it$0.37 per pass($0.23–$0.55)35% went to the costliest 10% of attempts (5 of 49)
tokens
1.8× the typical spend per passing run46.6% of this setup's tokens here went to runs that did not pass or were not graded; efficiency does not price it2M per pass(1.3M–2.8M)31% went to the costliest 10% of attempts (5 of 49)cache split not recorded on 49 attempts
time
2.0× the typical spend per passing run44.2% of this setup's time here went to runs that did not pass or were not graded; efficiency does not price it3m 55s per pass(2m 41s–5m 30s)28% went to the costliest 10% of attempts (5 of 49)

benchme — intent tasks

82.0 pts

benchme — intent tasks: 100.0% · 161 of 163 graded · scored

cost
0.85× the typical spend per passing run1.4% of this setup's cost here went to runs that did not pass or were not graded; efficiency does not price it$0.03 per pass($0.03–$0.03)21% went to the costliest 10% of attempts (17 of 161)
tokens
1.1× the typical spend per passing run1.6% of this setup's tokens here went to runs that did not pass or were not graded; efficiency does not price it130.5k per pass(121.5k–139.5k)21% went to the costliest 10% of attempts (17 of 161)cache split not recorded on 161 attempts
time
1.8× the typical spend per passing run1.3% of this setup's time here went to runs that did not pass or were not graded; efficiency does not price it54.4s per pass(52.4s–56.5s)16% went to the costliest 10% of attempts (17 of 161)

AIME 2025

55.0 pts

AIME 2025: 66.7% · 30 of 30 graded · scored

cost
1.1× the typical spend per passing run42.9% of this setup's cost here went to runs that did not pass or were not graded; efficiency does not price it (3 such runs' cost were not recorded)$0.14 per pass($0.08–$0.25)43% went to the costliest 10% of attempts (3 of 27)3 attempts' cost were not recorded
tokens
2.4× the typical spend per passing run46.0% of this setup's tokens here went to runs that did not pass or were not graded; efficiency does not price it (3 such runs' tokens were not recorded)376.2k per pass(194.1k–688.5k)48% went to the costliest 10% of attempts (3 of 27)3 attempts' tokens were not recordedcache split not recorded on 27 attempts
time
2.0× the typical spend per passing run54.2% of this setup's time here went to runs that did not pass or were not graded; efficiency does not price it4m 15s per pass(2m 41s–6m 53s)28% went to the costliest 10% of attempts (3 of 30)

GPQA Diamond

51.2 pts

GPQA Diamond: 66.7% · 198 of 198 graded · scored

cost
0.40× the typical spend per passing run34.8% of this setup's cost here went to runs that did not pass or were not graded; efficiency does not price it$0.02 per pass($0.01–$0.02)49% went to the costliest 10% of attempts (20 of 198)
tokens
1.0× the typical spend per passing run30.4% of this setup's tokens here went to runs that did not pass or were not graded; efficiency does not price it40.3k per pass(33.5k–49.6k)44% went to the costliest 10% of attempts (20 of 198)cache split not recorded on 198 attempts
time
1.6× the typical spend per passing run34.6% of this setup's time here went to runs that did not pass or were not graded; efficiency does not price it1m 02s per pass(54.6s–1m 12s)25% went to the costliest 10% of attempts (20 of 198)

HumanEval

78.4 pts

HumanEval: 92.7% · 164 of 164 graded · scored

cost
0.54× the typical spend per passing run8.9% of this setup's cost here went to runs that did not pass or were not graded; efficiency does not price it$0.0033 per pass($0.0031–$0.0035)16% went to the costliest 10% of attempts (17 of 164)
tokens
1.2× the typical spend per passing run7.4% of this setup's tokens here went to runs that did not pass or were not graded; efficiency does not price it14.3k per pass(13.7k–15k)11% went to the costliest 10% of attempts (17 of 164)cache split not recorded on 164 attempts
time
1.8× the typical spend per passing run7.3% of this setup's time here went to runs that did not pass or were not graded; efficiency does not price it32.3s per pass(29.6s–35.6s)18% went to the costliest 10% of attempts (17 of 164)

Efficiency and spend per pass can order setups differently: efficiency compares the typical (median) passing run, task by task, while spend per pass averages every attempt, so a few very expensive runs, passing or not, count in full; and spend per pass also counts the attempts that failed or did not finish, and divides by the pass rate. A setup is called the most frugal only where both put it first and both separate it from every other setup.

Evidence

Every published study its runs come from.