Skip to content
kalanu
Theme
benchme — intent tasks, GPQA Diamond, HumanEval and WindTunnel only · 2 setups · 4 of 6 compared benchmarks

Claude Agent SDK · claude-sonnet-5-5 · harness default

anthropic

harness default, open-book on knowledge questions

Its score and figures

Every figure the leaderboard's table prints for this setup.

Its score and figures
rank in group1–2
Score79.4 pts77.7–80.7 pts
Pass rate93.3%91.4–95.0%93.3% · 575 of 575 graded
Above chance92.5%90.3–94.4%
Efficiency39.939.1–40.5
cost1.5×
tokens1.5×
time0.27×
Coverage4/6
Runs575

69.4 + 1.7 + 0.8 + 7.4 ≈ 79.4 (outcome + cost + tokens + time)

  • pass rate on each benchmark's own grader
  • cost
  • tokens
  • time
  • not measured
First run
2026-10-03
Last run
2026-10-04
did not finish (scored as a fail)
0
Tasks with no grade
0

On each benchmark

Each benchmark it ran: its status, pass rate, spend and spend per pass. The studies behind them are its evidence, below.

WindTunnel

84.7 pts

WindTunnel: 100.0% · 50 of 50 graded · scored

cost
1.1× the typical spend per passing run0.0% of this setup's cost here went to runs that did not pass or were not graded; efficiency does not price it$0.09 per pass($0.08–$0.12)30% went to the costliest 10% of attempts (5 of 50)
tokens
1.0× the typical spend per passing run0.0% of this setup's tokens here went to runs that did not pass or were not graded; efficiency does not price it249.3k per pass(190.4k–318.3k)35% went to the costliest 10% of attempts (5 of 50)cache split not recorded on 50 attempts
time
0.36× the typical spend per passing run0.0% of this setup's time here went to runs that did not pass or were not graded; efficiency does not price it21.1s per pass(16.3s–26.6s)34% went to the costliest 10% of attempts (5 of 50)

benchme — intent tasks

75.1 pts

benchme — intent tasks: 84.7% · 163 of 163 graded · scored

cost
1.8× the typical spend per passing run18.2% of this setup's cost here went to runs that did not pass or were not graded; efficiency does not price it$0.07 per pass($0.06–$0.08)17% went to the costliest 10% of attempts (17 of 163)
tokens
1.3× the typical spend per passing run16.9% of this setup's tokens here went to runs that did not pass or were not graded; efficiency does not price it162.3k per pass(149.3k–178.2k)17% went to the costliest 10% of attempts (17 of 163)cache split not recorded on 163 attempts
time
0.31× the typical spend per passing run21.5% of this setup's time here went to runs that did not pass or were not graded; efficiency does not price it12.1s per pass(10.9s–13.6s)19% went to the costliest 10% of attempts (17 of 163)

GPQA Diamond

77.5 pts

GPQA Diamond: 90.4% · 198 of 198 graded · scored

cost
1.0× the typical spend per passing run12.8% of this setup's cost here went to runs that did not pass or were not graded; efficiency does not price it$0.03 per pass($0.02–$0.03)29% went to the costliest 10% of attempts (20 of 198)
tokens
1.4× the typical spend per passing run9.2% of this setup's tokens here went to runs that did not pass or were not graded; efficiency does not price it35.4k per pass(32.2k–39.6k)17% went to the costliest 10% of attempts (20 of 198)cache split not recorded on 198 attempts
time
0.26× the typical spend per passing run20.1% of this setup's time here went to runs that did not pass or were not graded; efficiency does not price it9.7s per pass(8.1s–11.9s)37% went to the costliest 10% of attempts (20 of 198)

HumanEval

80.3 pts

HumanEval: 98.2% · 164 of 164 graded · scored

cost
2.8× the typical spend per passing run1.9% of this setup's cost here went to runs that did not pass or were not graded; efficiency does not price it$0.02 per pass($0.02–$0.02)18% went to the costliest 10% of attempts (17 of 164)
tokens
2.6× the typical spend per passing run1.8% of this setup's tokens here went to runs that did not pass or were not graded; efficiency does not price it29.5k per pass(28.9k–30.2k)10% went to the costliest 10% of attempts (17 of 164)cache split not recorded on 164 attempts
time
0.19× the typical spend per passing run2.2% of this setup's time here went to runs that did not pass or were not graded; efficiency does not price it3.2s per pass(3.1s–3.3s)15% went to the costliest 10% of attempts (17 of 164)

Efficiency and spend per pass can order setups differently: efficiency compares the typical (median) passing run, task by task, while spend per pass averages every attempt, so a few very expensive runs, passing or not, count in full; and spend per pass also counts the attempts that failed or did not finish, and divides by the pass rate. A setup is called the most frugal only where both put it first and both separate it from every other setup.

Evidence

Every published study its runs come from.