Skip to content
kalanu
Theme

Agent harness leaderboard

Every agent harness.Every model.Measured.

Every published run on every verified benchmark, graded by the benchmark's own answer key or checks, or by the benchmark's own checks, and weighed by typical (median) spend per passing run. Every figure links to the study that produced it.

See the leaderboardHow it's computedRead every study
12 harnesses, one protocol · 11 with published results
  • Claude Agent SDK, has published results
  • Claude Code, has published results
  • Codex, has published results
  • Aider, has published results
  • Continue, has published results
  • opencode, has published results
  • DeepAgents, has published results
  • Hermes, has published results
  • DeepSeek Harness, has published results
  • OpenClaw, has published results
  • NanoClaw, not measured yet
  • eve, has published results
Setups measured
32
15 harness × model pairs
Verified benchmarks with results
6/21
compared: 6
Runs counted
10,220
from 12 published studies
Updated
2026-10-21
the leaderboard's last computation
All published runs · 6 of 21 verified benchmarks with results

The kalanu leaderboard

Early results. Setups have been compared on 6 of 6 verified benchmarks with results; an overall ranking needs at least 2. Read why →

Colour by
  • Claude Agent SDK
  • Claude Code
  • Codex
  • Continue
  • opencode
  • Hermes
  • DeepSeek Harness
  • OpenClaw
  • eve
  • claude-haiku-4-5-20251001
  • claude-sonnet-5-5
  • gpt-5.4-mini
  • gpt-6.1-sol

AIME 2025, benchme — intent tasks, GPQA Diamond and 2 more · 11 setups · 5 of 6 compared benchmarks

  1. 1–3Codexgpt-6.1-solharness default, open-book on knowledge questions79.4 pts78.2–80.6 pts · 5/6pass-rate tier 1–2outcome + cost + tokens + time: 72.7 + 2.0 + 2.6 + 2.1 = 79.4
  2. 1–3Claude Codeclaude-sonnet-5-5harness default, open-book on knowledge questions78.6 pts76.8–79.9 pts · 5/6pass-rate tier 1–2outcome + cost + tokens + time: 71.0 + 2.7 + 1.4 + 3.6 ≈ 78.6
  3. 1–3Codexgpt-5.4-miniharness default, open-book on knowledge questions78.1 pts75.4–80.6 pts · 5/6pass-rate tier 3outcome + cost + tokens + time: 66.5 + 6.4 + 2.6 + 2.6 = 78.1
  1. 4–10DeepSeek Harnessclaude-haiku-4-5-20251001harness default, open-book on knowledge questions69.1 pts65.8–72.1 pts · 5/6pass-rate tier 4–10outcome + cost + tokens + time: 61.2 + 3.4 + 1.0 + 3.5 = 69.1
  2. 4–10Continueclaude-haiku-4-5-20251001harness default, open-book on knowledge questions68.8 pts65.6–72.3 pts · 5/6pass-rate tier 4–10outcome + cost + tokens + time: 57.3 + 3.8 + 5.4 + 2.3 = 68.8
  3. 4–10eveclaude-haiku-4-5-20251001harness default, open-book on knowledge questions68.6 pts65.2–71.8 pts · 5/6pass-rate tier 4–10outcome + cost + tokens + time: 59.7 + 2.7 + 2.4 + 3.7 ≈ 68.6
  4. 4–10Hermesclaude-haiku-4-5-20251001harness default, open-book on knowledge questions65.4 pts61.8–68.7 pts · 5/6pass-rate tier 4–10outcome + cost + tokens + time: 58.9 + 4.2 + 1.1 + 1.2 = 65.4
  5. 4–10Claude Agent SDKclaude-haiku-4-5-20251001harness default, open-book on knowledge questions63.8 pts60.1–67.4 pts · 5/6pass-rate tier 4–10outcome + cost + tokens + time: 58.0 + 2.0 + 0.9 + 2.9 = 63.8
  6. 4–10opencodeclaude-haiku-4-5-20251001harness default, open-book on knowledge questions61.7 pts58.0–65.4 pts · 5/6pass-rate tier 4–10outcome + cost + tokens + time: 51.7 + 5.8 + 2.1 + 2.1 = 61.7
  7. 4–10Claude Codeclaude-haiku-4-5-20251001harness default, open-book on knowledge questions61.3 pts58.1–64.5 pts · 5/6pass-rate tier 4–10outcome + cost + tokens + time: 57.0 + 1.8 + 0.8 + 1.7 = 61.3
  1. 11OpenClawclaude-haiku-4-5-20251001harness default, open-book on knowledge questions45.6 pts41.9–48.9 pts · 5/6pass-rate tier 11outcome + cost + tokens + time: 41.6 + 2.2 + 0.8 + 1.0 = 45.6
Scores compare only within a group; a band groups setups the data cannot split into separated tiers.

score = 75 × pass rate above chance + 25 × efficiencyHow a score is built →

Other groupsEach is compared only within itself: its setups are scored on a different set of benchmarks.

1 of 4 shown The rest, in the table →

More groups, in the table
Compared with no other setup

All 32 setups →

Leaderboard

Efficiency: 100 = the lowest typical (median) spend per passing run, task by task. Cost, tokens and time: × the typical spend per passing run.

Every setup, row by row
rank in groupSetupScorePass rateAbove chanceEfficiencycosttokenstimeCoverageRunsStudies
AIME 2025, benchme — intent tasks, GPQA Diamond and 2 more · 11 setups · 5 of 6 compared benchmarks
1–3Codexgpt-6.1-sol · openaiharness default, open-book on knowledge questions79.4 pts78.2–80.6 pts97.2%95.9–98.4%96.9%95.5–98.2%26.724.3–29.71.9×1.0×1.4×5/66045
1–3Claude Codeclaude-sonnet-5-5 · anthropicharness default, open-book on knowledge questions78.6 pts76.8–79.9 pts95.2%93.1–96.8%94.7%92.5–96.4%30.527.6–32.11.2×1.2×0.74×5/66055
1–3Codexgpt-5.4-mini · openaiharness default, open-book on knowledge questions78.1 pts75.4–80.6 pts90.1%86.7–93.2%88.7%85.1–91.9%46.543.4–50.10.50×0.67×0.89×5/66055
4–10DeepSeek Harnessclaude-haiku-4-5-20251001 · anthropicharness default, open-book on knowledge questions69.1 pts65.8–72.1 pts83.8%79.8–87.4%81.6%77.4–85.6%31.728.8–36.21.0×1.5×0.65×5/66025
4–10Continueclaude-haiku-4-5-20251001 · anthropicharness default, open-book on knowledge questions68.8 pts65.6–72.3 pts78.8%74.5–83.0%76.5%72.0–80.8%46.042.2–51.60.76×0.28×0.98×5/66015
4–10eveclaude-haiku-4-5-20251001 · anthropicharness default, open-book on knowledge questions68.6 pts65.2–71.8 pts81.9%77.7–85.7%79.6%75.2–83.6%35.633.0–39.71.1×0.71×0.59×5/66055
4–10Hermesclaude-haiku-4-5-20251001 · anthropicharness default, open-book on knowledge questions65.4 pts61.8–68.7 pts80.7%76.2–85.1%78.5%73.7–82.9%26.124.3–27.70.81×1.4×1.8×5/66045
4–10Claude Agent SDKclaude-haiku-4-5-20251001 · anthropicharness default, open-book on knowledge questions63.8 pts60.1–67.4 pts79.5%75.1–84.0%77.3%72.6–81.8%23.320.8–26.51.5×1.8×0.79×5/66055
4–10opencodeclaude-haiku-4-5-20251001 · anthropicharness default, open-book on knowledge questions61.7 pts58.0–65.4 pts72.0%67.3–76.8%69.0%64.2–73.9%39.837.6–43.00.57×0.69×1.1×5/66055
4–10Claude Codeclaude-haiku-4-5-20251001 · anthropicharness default, open-book on knowledge questions61.3 pts58.1–64.5 pts77.9%74.0–81.8%76.0%71.9–80.1%17.415.9–19.31.6×2.0×1.3×5/67256
11OpenClawclaude-haiku-4-5-20251001 · anthropicharness default, open-book on knowledge questions45.6 pts41.9–48.9 pts57.5%53.0–61.7%55.4%50.7–59.8%16.014.6–17.41.3×2.2×2.4×5/66045

Every number, every group →

Quality × efficiency

What each setup buys with what it spends

AIME 2025, benchme — intent tasks, GPQA Diamond and 2 more · 11 setups · 5 of 6 compared benchmarks

Colour by
  • Claude Agent SDK
  • Claude Code
  • Codex
  • Continue
  • opencode
  • Hermes
  • DeepSeek Harness
  • OpenClaw
  • eve
  • claude-haiku-4-5-20251001
  • claude-sonnet-5-5
  • gpt-5.4-mini
  • gpt-6.1-sol
100 = the lowest typical (median) spend per passing run, task by task; it moves when a cheaper setup is added. 2× that spend is 50.0, 4× is 25.0.y is the pass rate above chance on each benchmark's own grader; x is efficiency, 100 = the lowest typical (median) spend per passing run, task by task; runs that did not pass are not priced; scores compare only within a group.
  1. Codexgpt-6.1-solharness default, open-book on knowledge questions
  2. Claude Codeclaude-sonnet-5-5harness default, open-book on knowledge questions
  3. Codexgpt-5.4-miniharness default, open-book on knowledge questions
  4. DeepSeek Harnessclaude-haiku-4-5-20251001harness default, open-book on knowledge questions
  5. Continueclaude-haiku-4-5-20251001harness default, open-book on knowledge questions
  6. eveclaude-haiku-4-5-20251001harness default, open-book on knowledge questions
  7. Hermesclaude-haiku-4-5-20251001harness default, open-book on knowledge questions
  8. Claude Agent SDKclaude-haiku-4-5-20251001harness default, open-book on knowledge questions
  9. opencodeclaude-haiku-4-5-20251001harness default, open-book on knowledge questions
  10. Claude Codeclaude-haiku-4-5-20251001harness default, open-book on knowledge questions
  11. OpenClawclaude-haiku-4-5-20251001harness default, open-book on knowledge questions
The map as a table
SetupEfficiency: 100 = the lowest typical (median) spend per passing run, task by taskPass rate above chanceScore
Codex · gpt-6.1-sol · harness default26.796.9%79.4 pts
Claude Code · claude-sonnet-5-5 · harness default30.594.7%78.6 pts
Codex · gpt-5.4-mini · harness default46.588.7%78.1 pts
DeepSeek Harness · claude-haiku-4-5-20251001 · harness default31.781.6%69.1 pts
Continue · claude-haiku-4-5-20251001 · harness default46.076.5%68.8 pts
eve · claude-haiku-4-5-20251001 · harness default35.679.6%68.6 pts
Hermes · claude-haiku-4-5-20251001 · harness default26.178.5%65.4 pts
Claude Agent SDK · claude-haiku-4-5-20251001 · harness default23.377.3%63.8 pts
opencode · claude-haiku-4-5-20251001 · harness default39.869.0%61.7 pts
Claude Code · claude-haiku-4-5-20251001 · harness default17.476.0%61.3 pts
OpenClaw · claude-haiku-4-5-20251001 · harness default16.055.4%45.6 pts

Spend as × the typical setup, on a log scale: cost ×, tokens ×, time × just below ↓Every group's map →

Every metric

Every metric, per setup

Colour by
  • Claude Agent SDK
  • Claude Code
  • Codex
  • Continue
  • opencode
  • Hermes
  • DeepSeek Harness
  • OpenClaw
  • eve
  • claude-haiku-4-5-20251001
  • claude-sonnet-5-5
  • gpt-5.4-mini
  • gpt-6.1-sol

Pass rate above chance

in its stored order; a bracket groups setups the data cannot split into tiers

a run that did not finish counts as a fail

  1. 1–2Codexgpt-6.1-sol96.9%, 95.5–98.2%
  2. 1–2Claude Codeclaude-sonnet-5-594.7%, 92.5–96.4%
  3. 3Codexgpt-5.4-mini88.7%, 85.1–91.9%
  4. 4–10DeepSeek Harnessclaude-haiku-4-5-2025100181.6%, 77.4–85.6%
  5. 4–10eveclaude-haiku-4-5-2025100179.6%, 75.2–83.6%
  6. 4–10Hermesclaude-haiku-4-5-2025100178.5%, 73.7–82.9%
  7. 4–10Claude Agent SDKclaude-haiku-4-5-2025100177.3%, 72.6–81.8%
  8. 4–10Continueclaude-haiku-4-5-2025100176.5%, 72.0–80.8%
  9. 4–10Claude Codeclaude-haiku-4-5-2025100176.0%, 71.9–80.1%
  10. 4–10opencodeclaude-haiku-4-5-2025100169.0%, 64.2–73.9%
  11. 11OpenClawclaude-haiku-4-5-2025100155.4%, 50.7–59.8%

Efficiency

ordered by this figure; no tiers are tested on it; highest first

100 = the lowest typical (median) spend per passing run, task by task

  1. Codexgpt-5.4-mini46.5, 43.4–50.1
  2. Continueclaude-haiku-4-5-2025100146.0, 42.2–51.6
  3. opencodeclaude-haiku-4-5-2025100139.8, 37.6–43.0
  4. eveclaude-haiku-4-5-2025100135.6, 33.0–39.7
  5. DeepSeek Harnessclaude-haiku-4-5-2025100131.7, 28.8–36.2
  6. Claude Codeclaude-sonnet-5-530.5, 27.6–32.1
  7. Codexgpt-6.1-sol26.7, 24.3–29.7
  8. Hermesclaude-haiku-4-5-2025100126.1, 24.3–27.7
  9. Claude Agent SDKclaude-haiku-4-5-2025100123.3, 20.8–26.5
  10. Claude Codeclaude-haiku-4-5-2025100117.4, 15.9–19.3
  11. OpenClawclaude-haiku-4-5-2025100116.0, 14.6–17.4

Cost × the typical setup

ordered by this figure; no tiers are tested on it; lowest first

log scale around 1×, 1/8× to 8×; a chevron past an end, its printed value not cut

  1. Codexgpt-5.4-mini0.50×
  2. opencodeclaude-haiku-4-5-202510010.57×
  3. Continueclaude-haiku-4-5-202510010.76×
  4. Hermesclaude-haiku-4-5-202510010.81×
  5. DeepSeek Harnessclaude-haiku-4-5-202510011.0×
  6. eveclaude-haiku-4-5-202510011.1×
  7. Claude Codeclaude-sonnet-5-51.2×
  8. OpenClawclaude-haiku-4-5-202510011.3×
  9. Claude Agent SDKclaude-haiku-4-5-202510011.5×
  10. Claude Codeclaude-haiku-4-5-202510011.6×
  11. Codexgpt-6.1-sol1.9×

Tokens × the typical setup

ordered by this figure; no tiers are tested on it; lowest first

log scale around 1×, 1/8× to 8×; a chevron past an end, its printed value not cut

  1. Continueclaude-haiku-4-5-202510010.28×
  2. Codexgpt-5.4-mini0.67×
  3. opencodeclaude-haiku-4-5-202510010.69×
  4. eveclaude-haiku-4-5-202510010.71×
  5. Codexgpt-6.1-sol1.0×
  6. Claude Codeclaude-sonnet-5-51.2×
  7. Hermesclaude-haiku-4-5-202510011.4×
  8. DeepSeek Harnessclaude-haiku-4-5-202510011.5×
  9. Claude Agent SDKclaude-haiku-4-5-202510011.8×
  10. Claude Codeclaude-haiku-4-5-202510012.0×
  11. OpenClawclaude-haiku-4-5-202510012.2×

Time × the typical setup

ordered by this figure; no tiers are tested on it; lowest first

log scale around 1×, 1/8× to 8×; a chevron past an end, its printed value not cut

  1. eveclaude-haiku-4-5-202510010.59×
  2. DeepSeek Harnessclaude-haiku-4-5-202510010.65×
  3. Claude Codeclaude-sonnet-5-50.74×
  4. Claude Agent SDKclaude-haiku-4-5-202510010.79×
  5. Codexgpt-5.4-mini0.89×
  6. Continueclaude-haiku-4-5-202510010.98×
  7. opencodeclaude-haiku-4-5-202510011.1×
  8. Claude Codeclaude-haiku-4-5-202510011.3×
  9. Codexgpt-6.1-sol1.4×
  10. Hermesclaude-haiku-4-5-202510011.8×
  11. OpenClawclaude-haiku-4-5-202510012.4×

Each panel draws the 11 setups shown above, each panel in its own order. Spend (×): the typical (median) spend per passing run, as a multiple of the typical setup's.Every group's panels →

What the leaderboard is built on

Verified benchmarks

Every benchmark the leaderboard may use, and what it discloses about each. A benchmark joins the comparison once 2 setups are scored on it.

  • Knowledge & reasoning

    AIME 2025

    in the leaderboard: 30 comparison tasks, 12 setups scored; graded by the benchmark's own answer key or checks

    1. Codex gpt-6.1-solharness default, open-book on knowledge questions94.8 ptsrank on this benchmark 1–12
    2. Claude Code claude-sonnet-5-5harness default, open-book on knowledge questions91.0 ptsrank on this benchmark 1–12
    3. Codex gpt-5.4-miniharness default, open-book on knowledge questions85.8 ptsrank on this benchmark 1–12
    4. Continue claude-haiku-4-5-20251001harness default, open-book on knowledge questions72.8 ptsrank on this benchmark 1–12
    5. eve claude-haiku-4-5-20251001harness default, open-book on knowledge questions70.2 ptsrank on this benchmark 1–12
    5 of 12 shown; the band spans 1–12 Its order →
  • Web

    benchme — intent tasks

    in the leaderboard: 163 comparison tasks, 13 setups scored; graded by the benchmark's own answer key or checks

    uploaded by the publisher of this indexlicence not yet verified
    1. DeepSeek Harness claude-haiku-4-5-20251001harness default, open-book on knowledge questions88.3 ptsrank on this benchmark 1–12
    2. Codex gpt-5.4-miniharness default, open-book on knowledge questions88.1 ptsrank on this benchmark 1–12
    3. eve claude-haiku-4-5-20251001harness default, open-book on knowledge questions88.0 ptsrank on this benchmark 1–12
    4. Continue claude-haiku-4-5-20251001harness default, open-book on knowledge questions83.3 ptsrank on this benchmark 1–12
    5. Hermes claude-haiku-4-5-20251001harness default, open-book on knowledge questions82.0 ptsrank on this benchmark 1–12
    5 of 13 shown; the band spans 1–12 Its order →
  • Knowledge & reasoning

    GPQA Diamond

    in the leaderboard: 198 comparison tasks, 15 setups scored; graded by the benchmark's own answer key or checks

    Above chance
    1. opencode claude-sonnet-5-5harness default, open-book on knowledge questions78.1 ptsrank on this benchmark 1–15
    2. Claude Agent SDK claude-sonnet-5-5harness default, open-book on knowledge questions77.5 ptsrank on this benchmark 1–15
    3. Codex gpt-6.1-solharness default, open-book on knowledge questions77.1 ptsrank on this benchmark 1–15
    4. Claude Code claude-sonnet-5-5harness default, open-book on knowledge questions73.5 ptsrank on this benchmark 1–15
    5. Codex gpt-5.4-miniharness default, open-book on knowledge questions64.9 ptsrank on this benchmark 1–15
    5 of 15 shown; the band spans 1–15 Its order →
  • Code

    HumanEval

    in the leaderboard: 164 comparison tasks, 15 setups scored; graded by the benchmark's own answer key or checks

    1. DeepAgents claude-haiku-4-5-20251001harness default, open-book on knowledge questions86.4 ptsrank on this benchmark 1–15
    2. Continue claude-haiku-4-5-20251001harness default, open-book on knowledge questions85.0 ptsrank on this benchmark 1–15
    3. eve claude-haiku-4-5-20251001harness default, open-book on knowledge questions84.6 ptsrank on this benchmark 1–15
    4. Aider claude-haiku-4-5-20251001harness default, open-book on knowledge questions83.0 ptsrank on this benchmark 1–15
    5. opencode claude-haiku-4-5-20251001harness default, open-book on knowledge questions81.8 ptsrank on this benchmark 1–15
    5 of 15 shown; the band spans 1–15 Its order →
  • Knowledge & reasoning

    IFEval

    in the leaderboard: 69 comparison tasks, 12 setups scored; graded by the benchmark's own checks

    1. Codex gpt-5.4-minias published88.5 ptsrank on this benchmark 1–12
    2. Claude Agent SDK claude-haiku-4-5-20251001as published84.9 ptsrank on this benchmark 1–12
    3. Codex gpt-6.1-solas published79.2 ptsrank on this benchmark 1–12
    4. OpenClaw claude-haiku-4-5-20251001as published78.9 ptsrank on this benchmark 1–12
    5. Claude Code claude-sonnet-5-5as published78.7 ptsrank on this benchmark 1–12
    5 of 12 shown; the band spans 1–12 Its order →
  • Web

    WindTunnel

    in the leaderboard: 50 comparison tasks, 18 setups scored; graded by the benchmark's own answer key or checks

    uploaded by the publisher of this indexlicence not yet verified
    1. Claude Agent SDK claude-sonnet-5-5harness default, open-book on knowledge questions84.7 ptsrank on this benchmark 1–17
    2. Claude Code claude-haiku-4-5-20251001with my setupWebMCP81.1 ptsrank on this benchmark 1–17
    3. Codex gpt-5.4-miniwith my setupWebMCP80.8 ptsrank on this benchmark 1–17
    4. Claude Code claude-haiku-4-5-20251001with my setupPlaywright79.3 ptsrank on this benchmark 1–17
    5. opencode claude-sonnet-5-5harness default, open-book on knowledge questions79.0 ptsrank on this benchmark 1–17
    5 of 18 shown; the band spans 1–17 Its order →
  • ARC-AGI (public eval)Knowledge & reasoningno run admitted to the leaderboard yet
  • FACTS Grounding (public set)Knowledge & reasoningno run admitted to the leaderboard yet
  • GSM8KKnowledge & reasoningno run admitted to the leaderboard yet
  • HarmBenchSafetyno run admitted to the leaderboard yet
  • Humanity's Last Exam (text)Knowledge & reasoningno run admitted to the leaderboard yet
  • LiveCodeBenchCodeno run admitted to the leaderboard yet
  • MATHKnowledge & reasoningno run admitted to the leaderboard yet
  • MATH-500Knowledge & reasoningno run admitted to the leaderboard yet
  • MBPPCodeno run admitted to the leaderboard yet
  • MMLUKnowledge & reasoningno run admitted to the leaderboard yet
  • MMLU-ProKnowledge & reasoningno run admitted to the leaderboard yet
  • SimpleQAKnowledge & reasoningno run admitted to the leaderboard yet
  • SWE-bench VerifiedAgenticno run admitted to the leaderboard yetlicence not yet verified
  • Terminal-Bench 2.xAgenticno run admitted to the leaderboard yet
  • TruthfulQAKnowledge & reasoningno run admitted to the leaderboard yet
How it's computed

The rules behind every figure

Every figure on the leaderboard follows the same published rules. Read the full method.

  1. 01

    Verified benchmarks only

    Every run counted comes from a public, published study on a benchmark the hub has verified and listed. Nothing else is counted.

  2. 02

    One outcome, the benchmark's own

    Every run is graded by the benchmark's own answer key or checks, or by the benchmark's own checks: a pass is that verdict, and a run that did not finish counts as a fail.

  3. 03

    Typical (median) spend per passing run

    The median spend of a passing run on cost, tokens and time, against every setup that passed the same tasks: 100 = the lowest typical (median) spend per passing run, task by task. Spend per pass is shown beside it and is not in the score; the two can order setups differently, because efficiency compares the typical (median) passing run, task by task, while spend per pass averages every attempt, so a few very expensive runs, passing or not, count in full; and spend per pass also counts the attempts that failed or did not finish, and divides by the pass rate.

  4. 04

    Compare like with like

    A setup is compared only with setups scored on the same benchmarks and the same tasks, and setups the data cannot separate share a band. “#1” is printed only when there are at least 2 compared benchmarks and more than one setup scored on every one of them, and the lead among those setups is separated from every other, holds with any one benchmark left out, and does not come from efficiency alone.

How one score is built, step by step →

Every figure traces back to a published study.

Runs, trajectories and grades — the full record behind each board row, on studies.kalanu.ai.

Read every study