Skip to content
kalanu
Theme
AIME 2025, benchme — intent tasks, GPQA Diamond and 2 more · 11 setups · 5 of 6 compared benchmarks

Codex · gpt-6.1-sol · harness default

openai

harness default, open-book on knowledge questions

Its score and figures

Every figure the leaderboard's table prints for this setup.

Its score and figures
rank in group1–3
Score79.4 pts78.2–80.6 pts
Pass rate97.2%95.9–98.4%97.2% · 603 of 605 graded
Above chance96.9%95.5–98.2%
Efficiency26.724.3–29.7
cost1.9×
tokens1.0×
time1.4×
Coverage5/6
Runs604

72.7 + 2.0 + 2.6 + 2.1 = 79.4 (outcome + cost + tokens + time)

  • pass rate on each benchmark's own grader
  • cost
  • tokens
  • time
  • not measured
First run
2026-10-02
Last run
2026-10-03
did not finish (scored as a fail)
2
Tasks with no grade
2

On each benchmark

Each benchmark it ran: its status, pass rate, spend and spend per pass. The studies behind them are its evidence, below.

WindTunnel

75.0 pts

WindTunnel: 98.0% · 50 of 50 graded · scored

cost
2.3× the typical spend per passing run4.6% of this setup's cost here went to runs that did not pass or were not graded; efficiency does not price it$0.17 per pass($0.14–$0.21)22% went to the costliest 10% of attempts (5 of 50)
tokens
2.1× the typical spend per passing run5.0% of this setup's tokens here went to runs that did not pass or were not graded; efficiency does not price it730.8k per pass(574k–915.4k)26% went to the costliest 10% of attempts (5 of 50)of which 95% cache reads and 4% fresh input
time
3.4× the typical spend per passing run3.8% of this setup's time here went to runs that did not pass or were not graded; efficiency does not price it3m 00s per pass(2m 27s–3m 38s)22% went to the costliest 10% of attempts (5 of 50)

benchme — intent tasks

73.2 pts

benchme — intent tasks: 94.4% · 162 of 163 graded · scored

cost
3.1× the typical spend per passing run7.8% of this setup's cost here went to runs that did not pass or were not graded; efficiency does not price it (1 such run's cost was not recorded)$0.14 per pass($0.12–$0.15)22% went to the costliest 10% of attempts (16 of 160)2 attempts' cost were not recorded
tokens
2.9× the typical spend per passing run8.5% of this setup's tokens here went to runs that did not pass or were not graded; efficiency does not price it (1 such run's tokens was not recorded)492.9k per pass(425.9k–562.2k)25% went to the costliest 10% of attempts (16 of 160)2 attempts' tokens were not recordedof which 93% cache reads and 6% fresh input
time
3.8× the typical spend per passing run8.2% of this setup's time here went to runs that did not pass or were not graded; efficiency does not price it2m 24s per pass(2m 07s–2m 40s)22% went to the costliest 10% of attempts (17 of 162)

AIME 2025

94.8 pts

AIME 2025: 100.0% · 30 of 30 graded · scored

cost
0.53× the typical spend per passing run0.0% of this setup's cost here went to runs that did not pass or were not graded; efficiency does not price it$0.03 per pass($0.03–$0.04)17% went to the costliest 10% of attempts (3 of 30)
tokens
0.25× the typical spend per passing run0.0% of this setup's tokens here went to runs that did not pass or were not graded; efficiency does not price it13.6k per pass(12.4k–15.8k)18% went to the costliest 10% of attempts (3 of 30)of which 6% cache reads and 88% fresh input
time
0.45× the typical spend per passing run0.0% of this setup's time here went to runs that did not pass or were not graded; efficiency does not price it29.6s per pass(24.1s–36.7s)25% went to the costliest 10% of attempts (3 of 30)

GPQA Diamond

77.1 pts

GPQA Diamond: 96.0% · 198 of 198 graded · scored

cost
1.6× the typical spend per passing run7.9% of this setup's cost here went to runs that did not pass or were not graded; efficiency does not price it$0.03 per pass($0.03–$0.04)19% went to the costliest 10% of attempts (20 of 197)1 attempt's cost was not recorded
tokens
0.72× the typical spend per passing run11.7% of this setup's tokens here went to runs that did not pass or were not graded; efficiency does not price it17.4k per pass(15.6k–20k)22% went to the costliest 10% of attempts (20 of 197)1 attempt's tokens was not recordedof which 6% cache reads and 94% fresh input
time
0.93× the typical spend per passing run8.2% of this setup's time here went to runs that did not pass or were not graded; efficiency does not price it23.9s per pass(21.9s–26.4s)21% went to the costliest 10% of attempts (20 of 198)

HumanEval

76.6 pts

HumanEval: 97.5% · 163 of 164 graded · scored

cost
4.5× the typical spend per passing run2.6% of this setup's cost here went to runs that did not pass or were not graded; efficiency does not price it$0.03 per pass($0.02–$0.03)11% went to the costliest 10% of attempts (17 of 162)1 attempt's cost was not recorded
tokens
1.1× the typical spend per passing run2.5% of this setup's tokens here went to runs that did not pass or were not graded; efficiency does not price it12.3k per pass(12k–12.7k)11% went to the costliest 10% of attempts (17 of 162)1 attempt's tokens was not recordedof which 0% cache reads and 99% fresh input
time
1.0× the typical spend per passing run2.8% of this setup's time here went to runs that did not pass or were not graded; efficiency does not price it17.3s per pass(16.7s–18s)14% went to the costliest 10% of attempts (17 of 163)

Efficiency and spend per pass can order setups differently: efficiency compares the typical (median) passing run, task by task, while spend per pass averages every attempt, so a few very expensive runs, passing or not, count in full; and spend per pass also counts the attempts that failed or did not finish, and divides by the pass rate. A setup is called the most frugal only where both put it first and both separate it from every other setup.

Evidence

Every published study its runs come from.