Skip to content
kalanu
Theme
All published runs · 6 of 21 verified benchmarks with results

The kalanu leaderboard

Early results. Setups have been compared on 6 of 6 verified benchmarks with results; an overall ranking needs at least 2. Read why →

AIME 2025, benchme — intent tasks, GPQA Diamond and 2 more · 11 setups · 5 of 6 compared benchmarks

Colour by
  • Claude Agent SDK
  • Claude Code
  • Codex
  • Continue
  • opencode
  • Hermes
  • DeepSeek Harness
  • OpenClaw
  • eve
  • claude-haiku-4-5-20251001
  • claude-sonnet-5-5
  • gpt-5.4-mini
  • gpt-6.1-sol
  1. 1–3Codexgpt-6.1-solharness default, open-book on knowledge questions79.4 pts78.2–80.6 pts · 5/6pass-rate tier 1–2outcome + cost + tokens + time: 72.7 + 2.0 + 2.6 + 2.1 = 79.4
  2. 1–3Claude Codeclaude-sonnet-5-5harness default, open-book on knowledge questions78.6 pts76.8–79.9 pts · 5/6pass-rate tier 1–2outcome + cost + tokens + time: 71.0 + 2.7 + 1.4 + 3.6 ≈ 78.6
  3. 1–3Codexgpt-5.4-miniharness default, open-book on knowledge questions78.1 pts75.4–80.6 pts · 5/6pass-rate tier 3outcome + cost + tokens + time: 66.5 + 6.4 + 2.6 + 2.6 = 78.1
  1. 4–10DeepSeek Harnessclaude-haiku-4-5-20251001harness default, open-book on knowledge questions69.1 pts65.8–72.1 pts · 5/6pass-rate tier 4–10outcome + cost + tokens + time: 61.2 + 3.4 + 1.0 + 3.5 = 69.1
  2. 4–10Continueclaude-haiku-4-5-20251001harness default, open-book on knowledge questions68.8 pts65.6–72.3 pts · 5/6pass-rate tier 4–10outcome + cost + tokens + time: 57.3 + 3.8 + 5.4 + 2.3 = 68.8
  3. 4–10eveclaude-haiku-4-5-20251001harness default, open-book on knowledge questions68.6 pts65.2–71.8 pts · 5/6pass-rate tier 4–10outcome + cost + tokens + time: 59.7 + 2.7 + 2.4 + 3.7 ≈ 68.6
  4. 4–10Hermesclaude-haiku-4-5-20251001harness default, open-book on knowledge questions65.4 pts61.8–68.7 pts · 5/6pass-rate tier 4–10outcome + cost + tokens + time: 58.9 + 4.2 + 1.1 + 1.2 = 65.4
  5. 4–10Claude Agent SDKclaude-haiku-4-5-20251001harness default, open-book on knowledge questions63.8 pts60.1–67.4 pts · 5/6pass-rate tier 4–10outcome + cost + tokens + time: 58.0 + 2.0 + 0.9 + 2.9 = 63.8
  6. 4–10opencodeclaude-haiku-4-5-20251001harness default, open-book on knowledge questions61.7 pts58.0–65.4 pts · 5/6pass-rate tier 4–10outcome + cost + tokens + time: 51.7 + 5.8 + 2.1 + 2.1 = 61.7
  7. 4–10Claude Codeclaude-haiku-4-5-20251001harness default, open-book on knowledge questions61.3 pts58.1–64.5 pts · 5/6pass-rate tier 4–10outcome + cost + tokens + time: 57.0 + 1.8 + 0.8 + 1.7 = 61.3
  1. 11OpenClawclaude-haiku-4-5-20251001harness default, open-book on knowledge questions45.6 pts41.9–48.9 pts · 5/6pass-rate tier 11outcome + cost + tokens + time: 41.6 + 2.2 + 0.8 + 1.0 = 45.6
Scores compare only within a group; a band groups setups the data cannot split into separated tiers.

score = 75 × pass rate above chance + 25 × efficiencyHow a score is built →

Other groupsEach is compared only within itself: its setups are scored on a different set of benchmarks.

benchme — intent tasks, GPQA Diamond, HumanEval and WindTunnel only · 2 setups · 4 of 6 compared benchmarks

Colour by
  • Claude Agent SDK
  • opencode
  • claude-sonnet-5-5
  1. 1–2opencodeclaude-sonnet-5-5harness default, open-book on knowledge questions79.6 pts78.0–81.0 pts · 4/6pass-rate tier 1outcome + cost + tokens + time: 71.2 + 4.1 + 2.1 + 2.2 = 79.6
  2. 1–2Claude Agent SDKclaude-sonnet-5-5harness default, open-book on knowledge questions79.4 pts77.7–80.7 pts · 4/6pass-rate tier 2outcome + cost + tokens + time: 69.4 + 1.7 + 0.8 + 7.4 ≈ 79.4
Scores compare only within a group; a band groups setups the data cannot split into separated tiers.

IFEval only · 12 setups · 1 of 6 compared benchmarks

Colour by
  • Claude Agent SDK
  • Claude Code
  • Codex
  • Aider
  • opencode
  • DeepAgents
  • Hermes
  • DeepSeek Harness
  • OpenClaw
  • eve
  • claude-haiku-4-5-20251001
  • claude-sonnet-5-5
  • gpt-5.4-mini
  • gpt-6.1-sol
  1. 1–12Codexgpt-5.4-minias published88.5 pts86.1–89.9 pts · 1/6pass-rate tier 1–12outcome + cost + tokens + time: 75.0 + 7.9 + 2.1 + 3.4 ≈ 88.5
  2. 1–12Claude Agent SDKclaude-haiku-4-5-20251001as published84.9 pts81.9–86.8 pts · 1/6pass-rate tier 1–12outcome + cost + tokens + time: 73.9 + 2.6 + 0.8 + 7.6 = 84.9
  3. 1–12Codexgpt-6.1-solas published79.2 pts76.3–80.7 pts · 1/6pass-rate tier 1–12outcome + cost + tokens + time: 73.9 + 0.9 + 1.9 + 2.5 = 79.2
  4. 1–12OpenClawclaude-haiku-4-5-20251001as published78.9 pts76.4–80.3 pts · 1/6pass-rate tier 1–12outcome + cost + tokens + time: 73.9 + 3.2 + 0.6 + 1.1 ≈ 78.9
  5. 1–12Claude Codeclaude-sonnet-5-5as published78.7 pts76.2–80.2 pts · 1/6pass-rate tier 1–12outcome + cost + tokens + time: 73.9 + 1.2 + 0.9 + 2.6 ≈ 78.7
  6. 1–12Claude Codeclaude-haiku-4-5-20251001as published76.9 pts72.6–80.6 pts · 1/6pass-rate tier 1–12outcome + cost + tokens + time: 70.7 + 2.7 + 0.7 + 2.9 ≈ 76.9
  7. 1–12DeepAgentsclaude-haiku-4-5-20251001as published73.8 pts65.6–81.0 pts · 1/6pass-rate tier 1–12outcome + cost + tokens + time: 58.7 + 2.6 + 4.2 + 8.3 = 73.8
  8. 1–12DeepSeek Harnessclaude-haiku-4-5-20251001as published73.6 pts65.5–80.1 pts · 1/6pass-rate tier 1–12outcome + cost + tokens + time: 62.0 + 4.0 + 1.8 + 5.9 ≈ 73.6
  9. 1–12Aiderclaude-haiku-4-5-20251001as published70.2 pts62.0–78.2 pts · 1/6pass-rate tier 1–12outcome + cost + tokens + time: 53.3 + 6.0 + 8.3 + 2.6 = 70.2
  10. 1–12eveclaude-haiku-4-5-20251001as published69.9 pts61.2–78.5 pts · 1/6pass-rate tier 1–12outcome + cost + tokens + time: 56.5 + 2.7 + 4.0 + 6.6 ≈ 69.9
  11. 1–12Hermesclaude-haiku-4-5-20251001as published69.8 pts62.8–76.7 pts · 1/6pass-rate tier 1–12outcome + cost + tokens + time: 59.8 + 6.5 + 1.7 + 1.8 = 69.8
  12. 1–12opencodeclaude-haiku-4-5-20251001as published62.7 pts54.5–70.5 pts · 1/6pass-rate tier 1–12outcome + cost + tokens + time: 48.9 + 8.3 + 2.6 + 2.8 ≈ 62.7
Scores compare only within a group; a band groups setups the data cannot split into separated tiers.

3 of 4 shown The rest, in the table →

More groups, in the table
Compared with no other setup

All 32 setups →

Every setup, row by row

Every setup, row by row
rank in groupSetupScorePass rateAbove chanceEfficiencycosttokenstimeCoverageRunsStudies
AIME 2025, benchme — intent tasks, GPQA Diamond and 2 more · 11 setups · 5 of 6 compared benchmarks
1–3Codexgpt-6.1-sol · openaiharness default, open-book on knowledge questions79.4 pts78.2–80.6 pts97.2%95.9–98.4%97.2% · 603 of 605 graded96.9%95.5–98.2%26.724.3–29.71.9×1.0×1.4×5/66045
1–3Claude Codeclaude-sonnet-5-5 · anthropicharness default, open-book on knowledge questions78.6 pts76.8–79.9 pts95.2%93.1–96.8%95.2% · 603 of 605 graded94.7%92.5–96.4%30.527.6–32.11.2×1.2×0.74×5/66055
1–3Codexgpt-5.4-mini · openaiharness default, open-book on knowledge questions78.1 pts75.4–80.6 pts90.1%86.7–93.2%90.1% · 603 of 605 graded88.7%85.1–91.9%46.543.4–50.10.50×0.67×0.89×5/66055
4–10DeepSeek Harnessclaude-haiku-4-5-20251001 · anthropicharness default, open-book on knowledge questions69.1 pts65.8–72.1 pts83.8%79.8–87.4%83.8% · 600 of 605 graded81.6%77.4–85.6%31.728.8–36.21.0×1.5×0.65×5/66025
4–10Continueclaude-haiku-4-5-20251001 · anthropicharness default, open-book on knowledge questions68.8 pts65.6–72.3 pts78.8%74.5–83.0%78.8% · 595 of 605 graded76.5%72.0–80.8%46.042.2–51.60.76×0.28×0.98×5/66015
4–10eveclaude-haiku-4-5-20251001 · anthropicharness default, open-book on knowledge questions68.6 pts65.2–71.8 pts81.9%77.7–85.7%81.9% · 604 of 605 graded79.6%75.2–83.6%35.633.0–39.71.1×0.71×0.59×5/66055
4–10Hermesclaude-haiku-4-5-20251001 · anthropicharness default, open-book on knowledge questions65.4 pts61.8–68.7 pts80.7%76.2–85.1%80.7% · 602 of 605 graded78.5%73.7–82.9%26.124.3–27.70.81×1.4×1.8×5/66045
4–10Claude Agent SDKclaude-haiku-4-5-20251001 · anthropicharness default, open-book on knowledge questions63.8 pts60.1–67.4 pts79.5%75.1–84.0%79.5% · 602 of 605 graded77.3%72.6–81.8%23.320.8–26.51.5×1.8×0.79×5/66055
4–10opencodeclaude-haiku-4-5-20251001 · anthropicharness default, open-book on knowledge questions61.7 pts58.0–65.4 pts72.0%67.3–76.8%72.0% · 603 of 605 graded69.0%64.2–73.9%39.837.6–43.00.57×0.69×1.1×5/66055
4–10Claude Codeclaude-haiku-4-5-20251001 · anthropicharness default, open-book on knowledge questions61.3 pts58.1–64.5 pts77.9%74.0–81.8%77.9% · 604 of 605 graded76.0%71.9–80.1%17.415.9–19.31.6×2.0×1.3×5/67256
11OpenClawclaude-haiku-4-5-20251001 · anthropicharness default, open-book on knowledge questions45.6 pts41.9–48.9 pts57.5%53.0–61.7%57.5% · 601 of 605 graded55.4%50.7–59.8%16.014.6–17.41.3×2.2×2.4×5/66045
benchme — intent tasks, GPQA Diamond, HumanEval and WindTunnel only · 2 setups · 4 of 6 compared benchmarks
1–2opencodeclaude-sonnet-5-5 · anthropicharness default, open-book on knowledge questions79.6 pts78.0–81.0 pts95.7%93.8–97.2%95.7% · 573 of 575 graded94.9%92.8–96.7%33.732.4–35.30.69×0.58×1.0×4/65734
1–2Claude Agent SDKclaude-sonnet-5-5 · anthropicharness default, open-book on knowledge questions79.4 pts77.7–80.7 pts93.3%91.4–95.0%93.3% · 575 of 575 graded92.5%90.3–94.4%39.939.1–40.51.5×1.5×0.27×4/65754
AIME 2025, GPQA Diamond and HumanEval only · 1 setup · 3 of 6 compared benchmarks
–DeepAgentsclaude-haiku-4-5-20251001 · anthropicharness default, open-book on knowledge questions59.8 pts54.6–65.6 pts70.7%64.5–77.1%70.7% · 605 of 605 graded67.0%60.5–73.8%38.433.8–47.11.3×0.95×0.45×3/66055
GPQA Diamond, HumanEval and WindTunnel only · 1 setup · 3 of 6 compared benchmarks
–Aiderclaude-haiku-4-5-20251001 · anthropicharness default, open-book on knowledge questions63.0 pts58.5–67.5 pts65.2%59.8–70.8%65.2% · 602 of 605 graded60.8%55.0–66.9%69.568.0–70.30.31×0.12×1.1×3/66055
IFEval only · 12 setups · 1 of 6 compared benchmarks
1–12Codexgpt-5.4-mini · openaias published88.5 pts86.1–89.9 pts100.0%≥ 94.8%100.0% · 69 of 69 gradednot measured: no benchmark in its group can be passed by guessing, so its pass rate is its outcome53.944.5–59.80.37×0.83×0.85×1/6691
1–12Claude Agent SDKclaude-haiku-4-5-20251001 · anthropicas published84.9 pts81.9–86.8 pts98.6%95.7–100.0%98.6% · 69 of 69 gradednot measured: no benchmark in its group can be passed by guessing, so its pass rate is its outcome43.935.0–47.81.1×2.3×0.39×1/6691
1–12Codexgpt-6.1-sol · openaias published79.2 pts76.3–80.7 pts98.6%95.7–100.0%98.6% · 69 of 69 gradednot measured: no benchmark in its group can be passed by guessing, so its pass rate is its outcome21.118.1–23.43.4×0.95×1.2×1/6691
1–12OpenClawclaude-haiku-4-5-20251001 · anthropicas published78.9 pts76.4–80.3 pts98.6%95.7–100.0%98.6% · 69 of 69 gradednot measured: no benchmark in its group can be passed by guessing, so its pass rate is its outcome19.817.4–21.80.92×2.8×2.6×1/6691
1–12Claude Codeclaude-sonnet-5-5 · anthropicas published78.7 pts76.2–80.2 pts98.6%95.7–100.0%98.6% · 69 of 69 gradednot measured: no benchmark in its group can be passed by guessing, so its pass rate is its outcome19.115.5–21.32.4×1.9×1.1×1/6691

Quality and efficiency

AIME 2025, benchme — intent tasks, GPQA Diamond and 2 more · 11 setups · 5 of 6 compared benchmarks

Colour by
  • Claude Agent SDK
  • Claude Code
  • Codex
  • Continue
  • opencode
  • Hermes
  • DeepSeek Harness
  • OpenClaw
  • eve
  • claude-haiku-4-5-20251001
  • claude-sonnet-5-5
  • gpt-5.4-mini
  • gpt-6.1-sol
100 = the lowest typical (median) spend per passing run, task by task; it moves when a cheaper setup is added. 2× that spend is 50.0, 4× is 25.0.y is the pass rate above chance on each benchmark's own grader; x is efficiency, 100 = the lowest typical (median) spend per passing run, task by task; runs that did not pass are not priced; scores compare only within a group.
  1. Codexgpt-6.1-solharness default, open-book on knowledge questions
  2. Claude Codeclaude-sonnet-5-5harness default, open-book on knowledge questions
  3. Codexgpt-5.4-miniharness default, open-book on knowledge questions
  4. DeepSeek Harnessclaude-haiku-4-5-20251001harness default, open-book on knowledge questions
  5. Continueclaude-haiku-4-5-20251001harness default, open-book on knowledge questions
  6. eveclaude-haiku-4-5-20251001harness default, open-book on knowledge questions
  7. Hermesclaude-haiku-4-5-20251001harness default, open-book on knowledge questions
  8. Claude Agent SDKclaude-haiku-4-5-20251001harness default, open-book on knowledge questions
  9. opencodeclaude-haiku-4-5-20251001harness default, open-book on knowledge questions
  10. Claude Codeclaude-haiku-4-5-20251001harness default, open-book on knowledge questions
  11. OpenClawclaude-haiku-4-5-20251001harness default, open-book on knowledge questions
The map as a table
SetupEfficiency: 100 = the lowest typical (median) spend per passing run, task by taskPass rate above chanceScore
Codex · gpt-6.1-sol · harness default26.796.9%79.4 pts
Claude Code · claude-sonnet-5-5 · harness default30.594.7%78.6 pts
Codex · gpt-5.4-mini · harness default46.588.7%78.1 pts
DeepSeek Harness · claude-haiku-4-5-20251001 · harness default31.781.6%69.1 pts
Continue · claude-haiku-4-5-20251001 · harness default46.076.5%68.8 pts
eve · claude-haiku-4-5-20251001 · harness default35.679.6%68.6 pts
Hermes · claude-haiku-4-5-20251001 · harness default26.178.5%65.4 pts
Claude Agent SDK · claude-haiku-4-5-20251001 · harness default23.377.3%63.8 pts
opencode · claude-haiku-4-5-20251001 · harness default39.869.0%61.7 pts
Claude Code · claude-haiku-4-5-20251001 · harness default17.476.0%61.3 pts
OpenClaw · claude-haiku-4-5-20251001 · harness default16.055.4%45.6 pts

benchme — intent tasks, GPQA Diamond, HumanEval and WindTunnel only · 2 setups · 4 of 6 compared benchmarks

Colour by
  • Claude Agent SDK
  • opencode
  • claude-sonnet-5-5
100 = the lowest typical (median) spend per passing run, task by task; it moves when a cheaper setup is added. 2× that spend is 50.0, 4× is 25.0.Pass rates above chance in this group are within 92.5–94.9%.y is the pass rate above chance on each benchmark's own grader; x is efficiency, 100 = the lowest typical (median) spend per passing run, task by task; runs that did not pass are not priced; scores compare only within a group.
  1. opencodeclaude-sonnet-5-5harness default, open-book on knowledge questions
  2. Claude Agent SDKclaude-sonnet-5-5harness default, open-book on knowledge questions
The map as a table
SetupEfficiency: 100 = the lowest typical (median) spend per passing run, task by taskPass rate above chanceScore
opencode · claude-sonnet-5-5 · harness default33.794.9%79.6 pts
Claude Agent SDK · claude-sonnet-5-5 · harness default39.992.5%79.4 pts

IFEval only · 12 setups · 1 of 6 compared benchmarks

Colour by
  • Claude Agent SDK
  • Claude Code
  • Codex
  • Aider
  • opencode
  • DeepAgents
  • Hermes
  • DeepSeek Harness
  • OpenClaw
  • eve
  • claude-haiku-4-5-20251001
  • claude-sonnet-5-5
  • gpt-5.4-mini
  • gpt-6.1-sol
100 = the lowest typical (median) spend per passing run, task by task; it moves when a cheaper setup is added. 2× that spend is 50.0, 4× is 25.0.y is the pass rate on each benchmark's own grader; x is efficiency, 100 = the lowest typical (median) spend per passing run, task by task; runs that did not pass are not priced; scores compare only within a group.
  1. Codexgpt-5.4-minias published
  2. Claude Agent SDKclaude-haiku-4-5-20251001as published
  3. Codexgpt-6.1-solas published
  4. OpenClawclaude-haiku-4-5-20251001as published
  5. Claude Codeclaude-sonnet-5-5as published
  6. Claude Codeclaude-haiku-4-5-20251001as published
  7. DeepAgentsclaude-haiku-4-5-20251001as published
  8. DeepSeek Harnessclaude-haiku-4-5-20251001as published
  9. Aiderclaude-haiku-4-5-20251001as published
  10. eveclaude-haiku-4-5-20251001as published
  11. Hermesclaude-haiku-4-5-20251001as published
  12. opencodeclaude-haiku-4-5-20251001as published
The map as a table
SetupEfficiency: 100 = the lowest typical (median) spend per passing run, task by taskPass rateScore
Codex · gpt-5.4-mini · as published53.9100.0%88.5 pts
Claude Agent SDK · claude-haiku-4-5-20251001 · as published43.998.6%84.9 pts
Codex · gpt-6.1-sol · as published21.198.6%79.2 pts
OpenClaw · claude-haiku-4-5-20251001 · as published19.898.6%78.9 pts
Claude Code · claude-sonnet-5-5 · as published19.198.6%78.7 pts
Claude Code · claude-haiku-4-5-20251001 · as published25.294.2%76.9 pts
DeepAgents · claude-haiku-4-5-20251001 · as published60.578.3%73.8 pts
DeepSeek Harness · claude-haiku-4-5-20251001 · as published46.482.6%73.6 pts
Aider · claude-haiku-4-5-20251001 · as published67.771.0%70.2 pts
eve · claude-haiku-4-5-20251001 · as published53.575.4%69.9 pts
Hermes · claude-haiku-4-5-20251001 · as published39.979.7%69.8 pts
opencode · claude-haiku-4-5-20251001 · as published55.165.2%62.7 pts

3 of 4 groups mapped The rest, in the table →

Not mapped

Fewer than 2 setups in each of these groups are scored.

  • AIME 2025, GPQA Diamond and HumanEval only · 1 setup · 3 of 6 compared benchmarks
  • GPQA Diamond, HumanEval and WindTunnel only · 1 setup · 3 of 6 compared benchmarks

By benchmark

Each benchmark with results, and how many setups are scored on it: its page draws its order, cell by cell.

What the leaderboard leaves out

Excluded from the leaderboard

  • lost to platform infrastructure6 studies · 12 runs

Counted, not excluded

  • did not finish (scored as a fail)5 studies · 37 runs
  • not measured (no grader verdict)6 studies · 35 runs