Skip to content
kalanu
Theme
AIME 2025, benchme — intent tasks, GPQA Diamond and 2 more · 11 setups · 5 of 6 compared benchmarks

Claude Code · claude-sonnet-5-5 · harness default

anthropic

harness default, open-book on knowledge questions

Its score and figures

Every figure the leaderboard's table prints for this setup.

Its score and figures
rank in group1–3
Score78.6 pts76.8–79.9 pts
Pass rate95.2%93.1–96.8%95.2% · 603 of 605 graded
Above chance94.7%92.5–96.4%
Efficiency30.527.6–32.1
cost1.2×
tokens1.2×
time0.74×
Coverage5/6
Runs605

71.0 + 2.7 + 1.4 + 3.6 ≈ 78.6 (outcome + cost + tokens + time)

  • pass rate on each benchmark's own grader
  • cost
  • tokens
  • time
  • not measured
First run
2026-10-02
Last run
2026-10-03
did not finish (scored as a fail)
0
Tasks with no grade
2

On each benchmark

Each benchmark it ran: its status, pass rate, spend and spend per pass. The studies behind them are its evidence, below.

WindTunnel

78.2 pts

WindTunnel: 98.0% · 49 of 50 graded · scored

cost
1.2× the typical spend per passing run10.2% of this setup's cost here went to runs that did not pass or were not graded; efficiency does not price it$0.09 per pass($0.07–$0.11)27% went to the costliest 10% of attempts (5 of 49)
tokens
1.0× the typical spend per passing run11.5% of this setup's tokens here went to runs that did not pass or were not graded; efficiency does not price it238.6k per pass(183.5k–305.5k)31% went to the costliest 10% of attempts (5 of 49)of which 94% cache reads and 5% fresh input
time
0.85× the typical spend per passing run7.2% of this setup's time here went to runs that did not pass or were not graded; efficiency does not price it39.9s per pass(32.7s–48.9s)27% went to the costliest 10% of attempts (5 of 49)

benchme — intent tasks

74.0 pts

benchme — intent tasks: 90.7% · 162 of 163 graded · scored

cost
1.7× the typical spend per passing run12.9% of this setup's cost here went to runs that did not pass or were not graded; efficiency does not price it$0.06 per pass($0.06–$0.07)18% went to the costliest 10% of attempts (17 of 162)
tokens
1.2× the typical spend per passing run10.2% of this setup's tokens here went to runs that did not pass or were not graded; efficiency does not price it152.6k per pass(142.4k–166.2k)18% went to the costliest 10% of attempts (17 of 162)of which 92% cache reads and 7% fresh input
time
1.0× the typical spend per passing run12.9% of this setup's time here went to runs that did not pass or were not graded; efficiency does not price it34.1s per pass(31.4s–37.1s)17% went to the costliest 10% of attempts (17 of 162)

AIME 2025

91.0 pts

AIME 2025: 96.7% · 30 of 30 graded · scored

cost
0.46× the typical spend per passing run3.6% of this setup's cost here went to runs that did not pass or were not graded; efficiency does not price it$0.03 per pass($0.02–$0.04)25% went to the costliest 10% of attempts (3 of 30)
tokens
0.60× the typical spend per passing run2.9% of this setup's tokens here went to runs that did not pass or were not graded; efficiency does not price it37.2k per pass(30.9k–46.8k)25% went to the costliest 10% of attempts (3 of 30)of which 83% cache reads and 14% fresh input
time
0.30× the typical spend per passing run3.8% of this setup's time here went to runs that did not pass or were not graded; efficiency does not price it22.3s per pass(19.5s–25.3s)17% went to the costliest 10% of attempts (3 of 30)

GPQA Diamond

73.5 pts

GPQA Diamond: 92.4% · 198 of 198 graded · scored

cost
1.0× the typical spend per passing run8.5% of this setup's cost here went to runs that did not pass or were not graded; efficiency does not price it$0.03 per pass($0.02–$0.03)23% went to the costliest 10% of attempts (20 of 197)1 attempt's cost was not recorded
tokens
1.4× the typical spend per passing run7.7% of this setup's tokens here went to runs that did not pass or were not graded; efficiency does not price it33.9k per pass(31.8k–36.8k)15% went to the costliest 10% of attempts (20 of 197)1 attempt's tokens was not recordedof which 86% cache reads and 12% fresh input
time
0.89× the typical spend per passing run24.3% of this setup's time here went to runs that did not pass or were not graded; efficiency does not price it28.5s per pass(21.8s–40.5s)36% went to the costliest 10% of attempts (20 of 198)

HumanEval

76.5 pts

HumanEval: 98.2% · 164 of 164 graded · scored

cost
2.9× the typical spend per passing run2.0% of this setup's cost here went to runs that did not pass or were not graded; efficiency does not price it$0.02 per pass($0.02–$0.02)14% went to the costliest 10% of attempts (17 of 164)
tokens
2.6× the typical spend per passing run1.8% of this setup's tokens here went to runs that did not pass or were not graded; efficiency does not price it30k per pass(29.4k–30.8k)11% went to the costliest 10% of attempts (17 of 164)of which 87% cache reads and 12% fresh input
time
1.0× the typical spend per passing run1.9% of this setup's time here went to runs that did not pass or were not graded; efficiency does not price it15.9s per pass(15.4s–16.4s)13% went to the costliest 10% of attempts (17 of 164)

Efficiency and spend per pass can order setups differently: efficiency compares the typical (median) passing run, task by task, while spend per pass averages every attempt, so a few very expensive runs, passing or not, count in full; and spend per pass also counts the attempts that failed or did not finish, and divides by the pass rate. A setup is called the most frugal only where both put it first and both separate it from every other setup.

Evidence

Every published study its runs come from.