Skip to content
kalanu
Theme
benchme — intent tasks, GPQA Diamond, HumanEval and WindTunnel only · 2 setups · 4 of 6 compared benchmarks

opencode · claude-sonnet-5-5 · harness default

anthropic

harness default, open-book on knowledge questions

Its score and figures

Every figure the leaderboard's table prints for this setup.

Its score and figures
rank in group1–2
Score79.6 pts78.0–81.0 pts
Pass rate95.7%93.8–97.2%95.7% · 573 of 575 graded
Above chance94.9%92.8–96.7%
Efficiency33.732.4–35.3
cost0.69×
tokens0.58×
time1.0×
Coverage4/6
Runs573

71.2 + 4.1 + 2.1 + 2.2 = 79.6 (outcome + cost + tokens + time)

  • pass rate on each benchmark's own grader
  • cost
  • tokens
  • time
  • not measured
First run
2026-10-03
Last run
2026-10-04
did not finish (scored as a fail)
0
Tasks with no grade
2

On each benchmark

Each benchmark it ran: its status, pass rate, spend and spend per pass. The studies behind them are its evidence, below.

WindTunnel

79.0 pts

WindTunnel: 98.0% · 49 of 50 graded · scored

cost
0.72× the typical spend per passing run1.0% of this setup's cost here went to runs that did not pass or were not graded; efficiency does not price it$0.06 per pass($0.05–$0.08)30% went to the costliest 10% of attempts (5 of 49)
tokens
0.40× the typical spend per passing run1.2% of this setup's tokens here went to runs that did not pass or were not graded; efficiency does not price it123.1k per pass(92.7k–159.6k)35% went to the costliest 10% of attempts (5 of 49)cache split not recorded on 49 attempts
time
0.90× the typical spend per passing run2.2% of this setup's time here went to runs that did not pass or were not graded; efficiency does not price it45.6s per pass(36.6s–57s)28% went to the costliest 10% of attempts (5 of 49)

benchme — intent tasks

80.9 pts

benchme — intent tasks: 95.7% · 162 of 163 graded · scored

cost
0.95× the typical spend per passing run6.9% of this setup's cost here went to runs that did not pass or were not graded; efficiency does not price it$0.03 per pass($0.03–$0.04)20% went to the costliest 10% of attempts (17 of 162)
tokens
0.55× the typical spend per passing run6.0% of this setup's tokens here went to runs that did not pass or were not graded; efficiency does not price it64k per pass(59.6k–68.7k)19% went to the costliest 10% of attempts (17 of 162)cache split not recorded on 162 attempts
time
1.0× the typical spend per passing run6.1% of this setup's time here went to runs that did not pass or were not graded; efficiency does not price it34.6s per pass(32.1s–37.6s)17% went to the costliest 10% of attempts (17 of 162)

GPQA Diamond

78.1 pts

GPQA Diamond: 90.9% · 198 of 198 graded · scored

cost
0.38× the typical spend per passing run9.3% of this setup's cost here went to runs that did not pass or were not graded; efficiency does not price it$0.01 per pass($0.01–$0.01)25% went to the costliest 10% of attempts (20 of 198)
tokens
0.53× the typical spend per passing run8.7% of this setup's tokens here went to runs that did not pass or were not graded; efficiency does not price it13.2k per pass(12.1k–14.7k)16% went to the costliest 10% of attempts (20 of 198)cache split not recorded on 198 attempts
time
0.99× the typical spend per passing run10.4% of this setup's time here went to runs that did not pass or were not graded; efficiency does not price it24.4s per pass(22.9s–26.1s)17% went to the costliest 10% of attempts (20 of 198)

HumanEval

80.4 pts

HumanEval: 98.2% · 164 of 164 graded · scored

cost
0.89× the typical spend per passing run2.0% of this setup's cost here went to runs that did not pass or were not graded; efficiency does not price it$0.01 per pass($0.01–$0.01)21% went to the costliest 10% of attempts (17 of 164)
tokens
0.99× the typical spend per passing run1.9% of this setup's tokens here went to runs that did not pass or were not graded; efficiency does not price it11.2k per pass(11k–11.5k)11% went to the costliest 10% of attempts (17 of 164)cache split not recorded on 164 attempts
time
1.1× the typical spend per passing run2.0% of this setup's time here went to runs that did not pass or were not graded; efficiency does not price it18.6s per pass(18.1s–19.1s)12% went to the costliest 10% of attempts (17 of 164)

Efficiency and spend per pass can order setups differently: efficiency compares the typical (median) passing run, task by task, while spend per pass averages every attempt, so a few very expensive runs, passing or not, count in full; and spend per pass also counts the attempts that failed or did not finish, and divides by the pass rate. A setup is called the most frugal only where both put it first and both separate it from every other setup.

Evidence

Every published study its runs come from.