Skip to content
kalanu
Theme
AIME 2025, benchme — intent tasks, GPQA Diamond and 2 more · 11 setups · 5 of 6 compared benchmarks

Codex · gpt-5.4-mini · harness default

openai

harness default, open-book on knowledge questions

Its score and figures

Every figure the leaderboard's table prints for this setup.

Its score and figures
rank in group1–3
Score78.1 pts75.4–80.6 pts
Pass rate90.1%86.7–93.2%90.1% · 603 of 605 graded
Above chance88.7%85.1–91.9%
Efficiency46.543.4–50.1
cost0.50×
tokens0.67×
time0.89×
Coverage5/6
Runs605

66.5 + 6.4 + 2.6 + 2.6 = 78.1 (outcome + cost + tokens + time)

  • pass rate on each benchmark's own grader
  • cost
  • tokens
  • time
  • not measured
First run
2026-10-02
Last run
2026-10-03
did not finish (scored as a fail)
1
Tasks with no grade
2

On each benchmark

Each benchmark it ran: its status, pass rate, spend and spend per pass. The studies behind them are its evidence, below.

WindTunnel

70.2 pts

WindTunnel: 87.8% · 49 of 50 graded · scored

cost
1.0× the typical spend per passing run39.9% of this setup's cost here went to runs that did not pass or were not graded; efficiency does not price it$0.41 per pass($0.19–$0.75)62% went to the costliest 10% of attempts (5 of 49)
tokens
1.0× the typical spend per passing run41.3% of this setup's tokens here went to runs that did not pass or were not graded; efficiency does not price it2.9M per pass(1.3M–5.3M)65% went to the costliest 10% of attempts (5 of 49)of which 93% cache reads and 7% fresh input
time
1.0× the typical spend per passing run35.4% of this setup's time here went to runs that did not pass or were not graded; efficiency does not price it2m 28s per pass(1m 18s–4m 14s)56% went to the costliest 10% of attempts (5 of 49)

benchme — intent tasks

88.1 pts

benchme — intent tasks: 100.0% · 162 of 163 graded · scored

cost
0.45× the typical spend per passing run0.9% of this setup's cost here went to runs that did not pass or were not graded; efficiency does not price it$0.02 per pass($0.02–$0.02)20% went to the costliest 10% of attempts (17 of 162)
tokens
0.58× the typical spend per passing run0.7% of this setup's tokens here went to runs that did not pass or were not graded; efficiency does not price it66k per pass(61.3k–71.9k)21% went to the costliest 10% of attempts (17 of 162)of which 84% cache reads and 14% fresh input
time
1.0× the typical spend per passing run0.7% of this setup's time here went to runs that did not pass or were not graded; efficiency does not price it31.3s per pass(29.7s–32.7s)16% went to the costliest 10% of attempts (17 of 162)

AIME 2025

85.8 pts

AIME 2025: 90.0% · 30 of 30 graded · scored

cost
0.37× the typical spend per passing run6.4% of this setup's cost here went to runs that did not pass or were not graded; efficiency does not price it$0.03 per pass($0.02–$0.04)28% went to the costliest 10% of attempts (3 of 30)
tokens
0.35× the typical spend per passing run4.1% of this setup's tokens here went to runs that did not pass or were not graded; efficiency does not price it36.5k per pass(23.5k–50.7k)36% went to the costliest 10% of attempts (3 of 30)of which 66% cache reads and 19% fresh input
time
0.63× the typical spend per passing run6.7% of this setup's time here went to runs that did not pass or were not graded; efficiency does not price it49.7s per pass(37s–1m 03s)25% went to the costliest 10% of attempts (3 of 30)

GPQA Diamond

64.9 pts

GPQA Diamond: 78.8% · 198 of 198 graded · scored

cost
0.42× the typical spend per passing run47.9% of this setup's cost here went to runs that did not pass or were not graded; efficiency does not price it$0.02 per pass($0.02–$0.03)44% went to the costliest 10% of attempts (20 of 198)
tokens
0.67× the typical spend per passing run38.2% of this setup's tokens here went to runs that did not pass or were not graded; efficiency does not price it21.2k per pass(17k–27.6k)34% went to the costliest 10% of attempts (20 of 198)of which 53% cache reads and 32% fresh input
time
0.88× the typical spend per passing run37.1% of this setup's time here went to runs that did not pass or were not graded; efficiency does not price it35.2s per pass(30.1s–42s)31% went to the costliest 10% of attempts (20 of 198)

HumanEval

81.6 pts

HumanEval: 93.9% · 164 of 164 graded · scored

cost
0.44× the typical spend per passing run9.1% of this setup's cost here went to runs that did not pass or were not graded; efficiency does not price it$0.01 per pass($0.0044–$0.01)22% went to the costliest 10% of attempts (17 of 163)1 attempt's cost was not recorded
tokens
0.97× the typical spend per passing run6.4% of this setup's tokens here went to runs that did not pass or were not graded; efficiency does not price it11.5k per pass(11.1k–12k)11% went to the costliest 10% of attempts (17 of 163)1 attempt's tokens was not recordedof which 58% cache reads and 39% fresh input
time
1.0× the typical spend per passing run7.0% of this setup's time here went to runs that did not pass or were not graded; efficiency does not price it16.4s per pass(15.6s–17.5s)15% went to the costliest 10% of attempts (17 of 164)

Efficiency and spend per pass can order setups differently: efficiency compares the typical (median) passing run, task by task, while spend per pass averages every attempt, so a few very expensive runs, passing or not, count in full; and spend per pass also counts the attempts that failed or did not finish, and divides by the pass rate. A setup is called the most frugal only where both put it first and both separate it from every other setup.

Evidence

Every published study its runs come from.