How the leaderboard is computed

The leaderboard counts every run of the publisher's public, published studies on benchmarks the hub has verified and listed, and nothing else. A study that is unpublished, or a benchmark that is unlisted, takes all of its runs and counts with it.

Inside that set, a run is left out, and counted in the ledger, for any of these:

• still running
• canceled
• lost to platform infrastructure
• paraphrased task variant
• not linked to a task of the benchmark version
• ran on a superseded benchmark version

These are counted, and exclude nothing:

• did not finish (scored as a fail)
• not measured (no grader verdict)

A verified benchmark the leaderboard cannot fold is listed as “not aggregated”, with its reason. A benchmark declared outside the leaderboard is never named on this site.

A row is one setup: a harness, a model, the model's provider and parameters, the launch profile, and the setup's add-ons, tools and skills. Setups that differ in any of these are separate rows, and rows are never blended.

An add-on the site cannot name prints “custom setup”, but its key still enters the row's identity, so different custom setups stay separate rows. A setup with every tool turned off prints “tools off”, and a custom tool list prints “custom tools”, never the names an organization typed.

Not in a row's identity: the harness build, and a provider changing a model behind a stable name or alias. Each row shows its first and last run day, so it can be read against them.

Every row runs under one launch profile:

as published

A model-API call with tools off — closed-book, the one shape comparable to the benchmark's own published leaderboard.

harness default

A real harness session with its native tools, nothing added and nothing removed: open-book on knowledge questions.

with my setup

The publisher's own setup, kept as configured. Each setup is its own row, never blended with another.own row

A run passes or fails by the benchmark's own grader, read the same way for every setup. Each benchmark is graded by one of these:

• the benchmark's own answer key or checks
• the benchmark's own checks
• a judge panel against the benchmark's pre-registered rubric, every criterion passing

A run that did not finish counts as a fail: an agent that does not finish within its budget has not done the task. A run lost to platform infrastructure is left out instead. A run that finished with no grader verdict is “not measured”: it is counted and shown beside every rate, never as a pass or a fail.

Where a benchmark can be passed by guessing (multiple choice), the outcome is the pass rate above chance: the pass rate rescaled so that chance scores nothing and a perfect run everything. A setup below chance prints “below chance” and earns no outcome points. Where nobody can guess, it is the plain pass rate.

Every setup scored on a benchmark is compared on the same tasks: its comparison set, which is the benchmark's whole corpus at one pinned version, or a task sample the benchmark declares in advance. Tasks outside it are not counted, and runs on another version are counted as superseded.

A cell is one setup on one benchmark. A cell that is not scored prints one of these instead of a score:

subset

the setup did not run every task of the comparison set: its rate is shown, compared only on the comparison set, and it is not scored against the others.

below the evidence floor

fewer graded tasks than the evidence floor — 10, or the floor the benchmark declares, or its whole comparison set when that is smaller. The cell prints no rate.

too many ungraded tasks

more than 10.0% of its comparison tasks have no grade.

too few passes to price

fewer passing tasks with a recorded spend than its pricing floor, so its efficiency cannot be priced.

no comparator

no other setup passed enough of the same tasks.

A benchmark joins the comparison once 2 setups are scored on it.

Efficiency prices passing runs only, and by medians at every stage: on each task, the median spend of a setup's passing runs, against the median of every setup that passed the same task; then the median of those ratios over the tasks, for cost, tokens and time each on its own. A task is priced only when at least 2 setups passed it.

Each benchmark anchors its own scale: 100 = the lowest typical (median) spend per passing run, task by task. A setup's efficiency on a benchmark is the mean of the per-aspect efficiency indexes, across benchmarks the mean of those, and each aspect also prints its spend as a multiple of the typical setup's (“× the typical spend per passing run”).

What it does not price: the spend of runs that did not pass. Each row's breakdown shows, per benchmark, the share of spend that went to runs that did not pass or were not graded; efficiency does not price it.

A setup that rarely passes is priced on few runs; below the evidence floor it is not scored.

Spend per pass is the average spend of an attempt (a graded run, or one that did not finish), averaged over tasks, divided by the pass rate, for cost, tokens and time each on its own. A setup with no pass prints “no pass, so no spend per pass”.

It is shown beside the score and is never part of it. Each figure carries its 95% interval from the same task bootstrap as every other interval, the share of its spend that went to the costliest 10% of attempts, and how many attempts' spend was not recorded.

A tokens figure says how much of it was cache reads and how much fresh input only where the harness and provider are declared to report cache reads, with a recorded run as proof. Everywhere else it prints “cache split not recorded”: a harness that never reports cache reads would otherwise read as one with none.

Efficiency and spend per pass can order setups differently: efficiency compares the typical (median) passing run, task by task, while spend per pass averages every attempt, so a few very expensive runs, passing or not, count in full; and spend per pass also counts the attempts that failed or did not finish, and divides by the pass rate. A setup is called the most frugal only where both put it first and both separate it from every other setup.

score = 75 × pass rate above chance + 25 × efficiency

On each benchmark a setup's score is 75 × its pass rate above chance (never below 0) plus 25 × its efficiency, the mean of the per-aspect efficiency indexes over cost, tokens and time.

A setup's score across benchmarks is the mean of its benchmark scores, each benchmark weighted equally.

A worked example: the first row of the stored leaderboard, Codex · gpt-6.1-sol · harness default, through its own stored parts (an order alone is not a claim).

Each part is cut to one decimal on its own, so they read 72.7 + 2.0 + 2.6 + 2.1 = 79.4: “=” only when the printed parts add up exactly to the printed score, else “≈”.

Scores compare only within a comparable group: the setups scored on exactly the same benchmarks. A group's heading names its benchmarks, and groups covering more benchmarks come first.

Coverage prints how many of the compared benchmarks a setup is scored on, with a dot per benchmark with results: scored, measured but not scored, or not run. Missing is never zero and never last place: a setup with no scored cell is listed under “Not scored” with its cells, and no rank.

An overall ranking needs at least 2 compared benchmarks; until then the page says its results are early.

A benchmark's own page compares every setup scored on it, whatever else that setup covers, so its bands can differ from any group's.

Every interval comes from one bootstrap of 1,000 draws over tasks. Each benchmark draws its own tasks from its own seeded stream, and the same drawn tasks apply to every setup, so a hard task stays hard for everyone: the comparisons are paired. Intervals are 95% intervals, given these benchmarks, tasks and runs.

A setup undefined in more than 5.0% of the draws, or whose interval has no width, prints “interval not quotable”; a perfect or a zero pass rate quotes its exact Clopper–Pearson bound instead. A score is never such a boundary case, and a score interval whose two ends print alike has no width either.

Every pair of setups in an order is tested, never only neighbours, and each order's score and outcome are each one Holm family. A pair is separated only when all of these hold:

• its Holm-adjusted p-value is below 0.05;
• its difference points the way the order does;
• and the two setups' values print differently at the page's own precision: a difference the page's strings show as equal is no difference.

A pair that fails any one of them is not separated, which is not the same as indistinguishable: two nearly identical setups can differ consistently and still print alike. A band holds the setups the data cannot split into tiers, and it splits only where every setup above a cut is separated from every setup below it. Separated pairs inside a band are named, up to 3, and counted beyond that.

Every sentence is a fixed template over these stored results: it names its setups by their labels, never by a position, and asserts nothing the stored pairs do not. “#1” is printed only when there are at least 2 compared benchmarks and more than one setup scored on every one of them, and the lead among those setups is separated from every other, holds with any one benchmark left out, and does not come from efficiency alone.

Each comparable group with at least 2 scored setups gets a map. On it, y is the pass rate on each benchmark's own grader; x is efficiency, 100 = the lowest typical (median) spend per passing run, task by task; runs that did not pass are not priced; scores compare only within a group.

Each setup is one point, at its means, and the diagonal lines join equal scores. Where a benchmark can be passed by guessing, y is the pass rate above chance instead.

There are no z-scores: a z-score is zero-sum, so one setup's gain is another's loss, and with two setups the map would be a mirror image.

A score says how a setup did on these benchmarks, these tasks and these runs. It does not cover which benchmarks were chosen, a provider changing a model behind a stable name, a harness changing between builds, a grader's own errors, or which setups the publisher chose to run.

A difference the page does not call separated is not a finding. A band is not a tie either: it says the data cannot order those setups yet, and the setups not separated may still differ.

Points are not an ability scale: they combine the outcome and efficiency by the published weights, and they compare only within a group.

• Some verified benchmarks were uploaded by the publisher of this leaderboard. Each benchmark's page and the Verified benchmarks list say which, from the benchmark's own record, never from the leaderboard.
• A benchmark whose licence the hub has not verified says so: “licence not yet verified”.
• Under “harness default”, an agent keeps its native tools: open-book on knowledge questions.
• Cost covers the model's tokens only; the harness and its tools are not priced.
• Efficiency is conditional on success. A setup that rarely passes is priced on few runs; below the evidence floor it is not scored.
• Some runs that did not pass have no recorded spend; each breakdown counts them as not recorded.
• A model can change behind a stable id or alias, and a harness between builds; neither is part of a row's identity.
• Study pages count a run that did not finish as not graded, while the leaderboard counts it as a fail, so their pass rates can differ.
• A benchmark's own board shows the newest eligible study per setup, and a study page compares that study's own arms, while the leaderboard pools every published run: each is labelled, and their numbers differ for related setups.
• The publisher chooses which setups run and which of its studies to publish. The leaderboard counts only published studies, and each row's coverage shows the benchmarks it was run on.

Cite the leaderboard by its citation line: the publisher, the version of the method, the digest of its inputs, and the day it was compiled. The digest changes whenever an input does, so a citation names exactly the numbers it read.

kalanu · leaderboard v1 · 07ab47641906 · 2026-10-21