Skip to content
kalanu
Theme
All published runs · 6 of 21 verified benchmarks with results

The kalanu leaderboard

Early results. Setups have been compared on 6 of 6 verified benchmarks with results; an overall ranking needs at least 2. Read why →

AIME 2025, benchme — intent tasks, GPQA Diamond and 2 more · 11 setups · 5 of 6 compared benchmarks

Colour by
  • Claude Agent SDK
  • Claude Code
  • Codex
  • Continue
  • opencode
  • Hermes
  • DeepSeek Harness
  • OpenClaw
  • eve
  • claude-haiku-4-5-20251001
  • claude-sonnet-5-5
  • gpt-5.4-mini
  • gpt-6.1-sol
  1. 1–3Codexgpt-6.1-solharness default, open-book on knowledge questions79.4 pts78.2–80.6 pts · 5/6pass-rate tier 1–2outcome + cost + tokens + time: 72.7 + 2.0 + 2.6 + 2.1 = 79.4
  2. 1–3Claude Codeclaude-sonnet-5-5harness default, open-book on knowledge questions78.6 pts76.8–79.9 pts · 5/6pass-rate tier 1–2outcome + cost + tokens + time: 71.0 + 2.7 + 1.4 + 3.6 ≈ 78.6
  3. 1–3Codexgpt-5.4-miniharness default, open-book on knowledge questions78.1 pts75.4–80.6 pts · 5/6pass-rate tier 3outcome + cost + tokens + time: 66.5 + 6.4 + 2.6 + 2.6 = 78.1
  1. 4–10DeepSeek Harnessclaude-haiku-4-5-20251001harness default, open-book on knowledge questions69.1 pts65.8–72.1 pts · 5/6pass-rate tier 4–10outcome + cost + tokens + time: 61.2 + 3.4 + 1.0 + 3.5 = 69.1
  2. 4–10Continueclaude-haiku-4-5-20251001harness default, open-book on knowledge questions68.8 pts65.6–72.3 pts · 5/6pass-rate tier 4–10outcome + cost + tokens + time: 57.3 + 3.8 + 5.4 + 2.3 = 68.8
  3. 4–10eveclaude-haiku-4-5-20251001harness default, open-book on knowledge questions68.6 pts65.2–71.8 pts · 5/6pass-rate tier 4–10outcome + cost + tokens + time: 59.7 + 2.7 + 2.4 + 3.7 ≈ 68.6
  4. 4–10Hermesclaude-haiku-4-5-20251001harness default, open-book on knowledge questions65.4 pts61.8–68.7 pts · 5/6pass-rate tier 4–10outcome + cost + tokens + time: 58.9 + 4.2 + 1.1 + 1.2 = 65.4
  5. 4–10Claude Agent SDKclaude-haiku-4-5-20251001harness default, open-book on knowledge questions63.8 pts60.1–67.4 pts · 5/6pass-rate tier 4–10outcome + cost + tokens + time: 58.0 + 2.0 + 0.9 + 2.9 = 63.8
  6. 4–10opencodeclaude-haiku-4-5-20251001harness default, open-book on knowledge questions61.7 pts58.0–65.4 pts · 5/6pass-rate tier 4–10outcome + cost + tokens + time: 51.7 + 5.8 + 2.1 + 2.1 = 61.7
  7. 4–10Claude Codeclaude-haiku-4-5-20251001harness default, open-book on knowledge questions61.3 pts58.1–64.5 pts · 5/6pass-rate tier 4–10outcome + cost + tokens + time: 57.0 + 1.8 + 0.8 + 1.7 = 61.3
  1. 11OpenClawclaude-haiku-4-5-20251001harness default, open-book on knowledge questions45.6 pts41.9–48.9 pts · 5/6pass-rate tier 11outcome + cost + tokens + time: 41.6 + 2.2 + 0.8 + 1.0 = 45.6
Scores compare only within a group; a band groups setups the data cannot split into separated tiers.

score = 75 × pass rate above chance + 25 × efficiencyHow a score is built →

Other groupsEach is compared only within itself: its setups are scored on a different set of benchmarks.

benchme — intent tasks, GPQA Diamond, HumanEval and WindTunnel only · 2 setups · 4 of 6 compared benchmarks

Colour by
  • Claude Agent SDK
  • opencode
  • claude-sonnet-5-5
  1. 1–2opencodeclaude-sonnet-5-5harness default, open-book on knowledge questions79.6 pts78.0–81.0 pts · 4/6pass-rate tier 1outcome + cost + tokens + time: 71.2 + 4.1 + 2.1 + 2.2 = 79.6
  2. 1–2Claude Agent SDKclaude-sonnet-5-5harness default, open-book on knowledge questions79.4 pts77.7–80.7 pts · 4/6pass-rate tier 2outcome + cost + tokens + time: 69.4 + 1.7 + 0.8 + 7.4 ≈ 79.4
Scores compare only within a group; a band groups setups the data cannot split into separated tiers.

IFEval only · 12 setups · 1 of 6 compared benchmarks

Colour by
  • Claude Agent SDK
  • Claude Code
  • Codex
  • Aider
  • opencode
  • DeepAgents
  • Hermes
  • DeepSeek Harness
  • OpenClaw
  • eve
  • claude-haiku-4-5-20251001
  • claude-sonnet-5-5
  • gpt-5.4-mini
  • gpt-6.1-sol
  1. 1–12Codexgpt-5.4-minias published88.5 pts86.1–89.9 pts · 1/6pass-rate tier 1–12outcome + cost + tokens + time: 75.0 + 7.9 + 2.1 + 3.4 ≈ 88.5
  2. 1–12Claude Agent SDKclaude-haiku-4-5-20251001as published84.9 pts81.9–86.8 pts · 1/6pass-rate tier 1–12outcome + cost + tokens + time: 73.9 + 2.6 + 0.8 + 7.6 = 84.9
  3. 1–12Codexgpt-6.1-solas published79.2 pts76.3–80.7 pts · 1/6pass-rate tier 1–12outcome + cost + tokens + time: 73.9 + 0.9 + 1.9 + 2.5 = 79.2
  4. 1–12OpenClawclaude-haiku-4-5-20251001as published78.9 pts76.4–80.3 pts · 1/6pass-rate tier 1–12outcome + cost + tokens + time: 73.9 + 3.2 + 0.6 + 1.1 ≈ 78.9
  5. 1–12Claude Codeclaude-sonnet-5-5as published78.7 pts76.2–80.2 pts · 1/6pass-rate tier 1–12outcome + cost + tokens + time: 73.9 + 1.2 + 0.9 + 2.6 ≈ 78.7
  6. 1–12Claude Codeclaude-haiku-4-5-20251001as published76.9 pts72.6–80.6 pts · 1/6pass-rate tier 1–12outcome + cost + tokens + time: 70.7 + 2.7 + 0.7 + 2.9 ≈ 76.9
  7. 1–12DeepAgentsclaude-haiku-4-5-20251001as published73.8 pts65.6–81.0 pts · 1/6pass-rate tier 1–12outcome + cost + tokens + time: 58.7 + 2.6 + 4.2 + 8.3 = 73.8
  8. 1–12DeepSeek Harnessclaude-haiku-4-5-20251001as published73.6 pts65.5–80.1 pts · 1/6pass-rate tier 1–12outcome + cost + tokens + time: 62.0 + 4.0 + 1.8 + 5.9 ≈ 73.6
  9. 1–12Aiderclaude-haiku-4-5-20251001as published70.2 pts62.0–78.2 pts · 1/6pass-rate tier 1–12outcome + cost + tokens + time: 53.3 + 6.0 + 8.3 + 2.6 = 70.2
  10. 1–12eveclaude-haiku-4-5-20251001as published69.9 pts61.2–78.5 pts · 1/6pass-rate tier 1–12outcome + cost + tokens + time: 56.5 + 2.7 + 4.0 + 6.6 ≈ 69.9
  11. 1–12Hermesclaude-haiku-4-5-20251001as published69.8 pts62.8–76.7 pts · 1/6pass-rate tier 1–12outcome + cost + tokens + time: 59.8 + 6.5 + 1.7 + 1.8 = 69.8
  12. 1–12opencodeclaude-haiku-4-5-20251001as published62.7 pts54.5–70.5 pts · 1/6pass-rate tier 1–12outcome + cost + tokens + time: 48.9 + 8.3 + 2.6 + 2.8 ≈ 62.7
Scores compare only within a group; a band groups setups the data cannot split into separated tiers.

3 of 4 shown The rest, in the table →

More groups, in the table
Compared with no other setup

All 32 setups →

Every setup, row by row

Every setup, row by row
rank in groupSetupScorePass rateAbove chanceEfficiencycosttokenstimeCoverageRunsStudies
IFEval only · 12 setups · 1 of 6 compared benchmarks
1–12Claude Codeclaude-haiku-4-5-20251001 · anthropicas published76.9 pts72.6–80.6 pts94.2%88.4–98.6%94.2% · 69 of 69 gradednot measured: no benchmark in its group can be passed by guessing, so its pass rate is its outcome25.220.7–28.41.1×2.6×1.0×1/6691
1–12DeepAgentsclaude-haiku-4-5-20251001 · anthropicas published73.8 pts65.6–81.0 pts78.3%68.1–88.4%78.3% · 69 of 69 gradednot measured: no benchmark in its group can be passed by guessing, so its pass rate is its outcome60.548.7–65.01.1×0.42×0.35×1/6691
1–12DeepSeek Harnessclaude-haiku-4-5-20251001 · anthropicas published73.6 pts65.5–80.1 pts82.6%72.5–91.3%82.6% · 69 of 69 gradednot measured: no benchmark in its group can be passed by guessing, so its pass rate is its outcome46.440.0–51.20.74×1.0×0.50×1/6691
1–12Aiderclaude-haiku-4-5-20251001 · anthropicas published70.2 pts62.0–78.2 pts71.0%60.9–81.2%71.0% · 69 of 69 gradednot measured: no benchmark in its group can be passed by guessing, so its pass rate is its outcome67.763.3–70.90.49×0.21×1.1×1/6691
1–12eveclaude-haiku-4-5-20251001 · anthropicas published69.9 pts61.2–78.5 pts75.4%65.2–85.5%75.4% · 69 of 69 gradednot measured: no benchmark in its group can be passed by guessing, so its pass rate is its outcome53.546.6–60.81.1×0.45×0.44×1/6691
1–12Hermesclaude-haiku-4-5-20251001 · anthropicas published69.8 pts62.8–76.7 pts79.7%71.0–88.4%79.7% · 69 of 69 gradednot measured: no benchmark in its group can be passed by guessing, so its pass rate is its outcome39.936.1–42.00.45×1.1×1.6×1/6691
1–12opencodeclaude-haiku-4-5-20251001 · anthropicas published62.7 pts54.5–70.5 pts65.2%55.1–75.4%65.2% · 69 of 69 gradednot measured: no benchmark in its group can be passed by guessing, so its pass rate is its outcome55.150.6–57.90.35×0.67×1.0×1/6691
WindTunnel only · 4 setups · 1 of 6 compared benchmarks
1–4Claude Codeclaude-haiku-4-5-20251001 · anthropicwith my setupWebMCP81.1 pts79.9–82.6 pts100.0%≥ 92.7%100.0% · 49 of 50 gradednot measured: no benchmark in its group can be passed by guessing, so its pass rate is its outcome24.519.6–30.60.44×0.93×0.90×1/6501
1–4Codexgpt-5.4-mini · openaiwith my setupWebMCP80.8 pts76.9–84.3 pts98.0%93.8–100.0%98.0% · 49 of 50 gradednot measured: no benchmark in its group can be passed by guessing, so its pass rate is its outcome29.123.2–38.70.35×0.42×0.87×1/6501
1–4Claude Codeclaude-haiku-4-5-20251001 · anthropicwith my setupPlaywright79.3 pts78.5–79.8 pts100.0%≥ 92.6%100.0% · 48 of 50 gradednot measured: no benchmark in its group can be passed by guessing, so its pass rate is its outcome17.013.8–19.10.79×1.2×1.1×1/6501
1–4Codexgpt-5.4-mini · openaiwith my setupPlaywright79.0 pts75.8–81.4 pts98.0%94.0–100.0%98.0% · 50 of 50 gradednot measured: no benchmark in its group can be passed by guessing, so its pass rate is its outcome22.018.4–26.20.57×0.52×1.0×1/6501

Not scored

Each setup's cells, by benchmark: measured, but not scored against the others.

Quality and efficiency

AIME 2025, benchme — intent tasks, GPQA Diamond and 2 more · 11 setups · 5 of 6 compared benchmarks

Colour by
  • Claude Agent SDK
  • Claude Code
  • Codex
  • Continue
  • opencode
  • Hermes
  • DeepSeek Harness
  • OpenClaw
  • eve
  • claude-haiku-4-5-20251001
  • claude-sonnet-5-5
  • gpt-5.4-mini
  • gpt-6.1-sol
100 = the lowest typical (median) spend per passing run, task by task; it moves when a cheaper setup is added. 2× that spend is 50.0, 4× is 25.0.y is the pass rate above chance on each benchmark's own grader; x is efficiency, 100 = the lowest typical (median) spend per passing run, task by task; runs that did not pass are not priced; scores compare only within a group.
  1. Codexgpt-6.1-solharness default, open-book on knowledge questions
  2. Claude Codeclaude-sonnet-5-5harness default, open-book on knowledge questions
  3. Codexgpt-5.4-miniharness default, open-book on knowledge questions
  4. DeepSeek Harnessclaude-haiku-4-5-20251001harness default, open-book on knowledge questions
  5. Continueclaude-haiku-4-5-20251001harness default, open-book on knowledge questions
  6. eveclaude-haiku-4-5-20251001harness default, open-book on knowledge questions
  7. Hermesclaude-haiku-4-5-20251001harness default, open-book on knowledge questions
  8. Claude Agent SDKclaude-haiku-4-5-20251001harness default, open-book on knowledge questions
  9. opencodeclaude-haiku-4-5-20251001harness default, open-book on knowledge questions
  10. Claude Codeclaude-haiku-4-5-20251001harness default, open-book on knowledge questions
  11. OpenClawclaude-haiku-4-5-20251001harness default, open-book on knowledge questions
The map as a table
SetupEfficiency: 100 = the lowest typical (median) spend per passing run, task by taskPass rate above chanceScore
Codex · gpt-6.1-sol · harness default26.796.9%79.4 pts
Claude Code · claude-sonnet-5-5 · harness default30.594.7%78.6 pts
Codex · gpt-5.4-mini · harness default46.588.7%78.1 pts
DeepSeek Harness · claude-haiku-4-5-20251001 · harness default31.781.6%69.1 pts
Continue · claude-haiku-4-5-20251001 · harness default46.076.5%68.8 pts
eve · claude-haiku-4-5-20251001 · harness default35.679.6%68.6 pts
Hermes · claude-haiku-4-5-20251001 · harness default26.178.5%65.4 pts
Claude Agent SDK · claude-haiku-4-5-20251001 · harness default23.377.3%63.8 pts
opencode · claude-haiku-4-5-20251001 · harness default39.869.0%61.7 pts
Claude Code · claude-haiku-4-5-20251001 · harness default17.476.0%61.3 pts
OpenClaw · claude-haiku-4-5-20251001 · harness default16.055.4%45.6 pts

benchme — intent tasks, GPQA Diamond, HumanEval and WindTunnel only · 2 setups · 4 of 6 compared benchmarks

Colour by
  • Claude Agent SDK
  • opencode
  • claude-sonnet-5-5
100 = the lowest typical (median) spend per passing run, task by task; it moves when a cheaper setup is added. 2× that spend is 50.0, 4× is 25.0.Pass rates above chance in this group are within 92.5–94.9%.y is the pass rate above chance on each benchmark's own grader; x is efficiency, 100 = the lowest typical (median) spend per passing run, task by task; runs that did not pass are not priced; scores compare only within a group.
  1. opencodeclaude-sonnet-5-5harness default, open-book on knowledge questions
  2. Claude Agent SDKclaude-sonnet-5-5harness default, open-book on knowledge questions
The map as a table
SetupEfficiency: 100 = the lowest typical (median) spend per passing run, task by taskPass rate above chanceScore
opencode · claude-sonnet-5-5 · harness default33.794.9%79.6 pts
Claude Agent SDK · claude-sonnet-5-5 · harness default39.992.5%79.4 pts

IFEval only · 12 setups · 1 of 6 compared benchmarks

Colour by
  • Claude Agent SDK
  • Claude Code
  • Codex
  • Aider
  • opencode
  • DeepAgents
  • Hermes
  • DeepSeek Harness
  • OpenClaw
  • eve
  • claude-haiku-4-5-20251001
  • claude-sonnet-5-5
  • gpt-5.4-mini
  • gpt-6.1-sol
100 = the lowest typical (median) spend per passing run, task by task; it moves when a cheaper setup is added. 2× that spend is 50.0, 4× is 25.0.y is the pass rate on each benchmark's own grader; x is efficiency, 100 = the lowest typical (median) spend per passing run, task by task; runs that did not pass are not priced; scores compare only within a group.
  1. Codexgpt-5.4-minias published
  2. Claude Agent SDKclaude-haiku-4-5-20251001as published
  3. Codexgpt-6.1-solas published
  4. OpenClawclaude-haiku-4-5-20251001as published
  5. Claude Codeclaude-sonnet-5-5as published
  6. Claude Codeclaude-haiku-4-5-20251001as published
  7. DeepAgentsclaude-haiku-4-5-20251001as published
  8. DeepSeek Harnessclaude-haiku-4-5-20251001as published
  9. Aiderclaude-haiku-4-5-20251001as published
  10. eveclaude-haiku-4-5-20251001as published
  11. Hermesclaude-haiku-4-5-20251001as published
  12. opencodeclaude-haiku-4-5-20251001as published
The map as a table
SetupEfficiency: 100 = the lowest typical (median) spend per passing run, task by taskPass rateScore
Codex · gpt-5.4-mini · as published53.9100.0%88.5 pts
Claude Agent SDK · claude-haiku-4-5-20251001 · as published43.998.6%84.9 pts
Codex · gpt-6.1-sol · as published21.198.6%79.2 pts
OpenClaw · claude-haiku-4-5-20251001 · as published19.898.6%78.9 pts
Claude Code · claude-sonnet-5-5 · as published19.198.6%78.7 pts
Claude Code · claude-haiku-4-5-20251001 · as published25.294.2%76.9 pts
DeepAgents · claude-haiku-4-5-20251001 · as published60.578.3%73.8 pts
DeepSeek Harness · claude-haiku-4-5-20251001 · as published46.482.6%73.6 pts
Aider · claude-haiku-4-5-20251001 · as published67.771.0%70.2 pts
eve · claude-haiku-4-5-20251001 · as published53.575.4%69.9 pts
Hermes · claude-haiku-4-5-20251001 · as published39.979.7%69.8 pts
opencode · claude-haiku-4-5-20251001 · as published55.165.2%62.7 pts

3 of 4 groups mapped The rest, in the table →

Not mapped

Fewer than 2 setups in each of these groups are scored.

  • AIME 2025, GPQA Diamond and HumanEval only · 1 setup · 3 of 6 compared benchmarks
  • GPQA Diamond, HumanEval and WindTunnel only · 1 setup · 3 of 6 compared benchmarks

By benchmark

Each benchmark with results, and how many setups are scored on it: its page draws its order, cell by cell.

What the leaderboard leaves out

Excluded from the leaderboard

  • lost to platform infrastructure6 studies · 12 runs

Counted, not excluded

  • did not finish (scored as a fail)5 studies · 37 runs
  • not measured (no grader verdict)6 studies · 35 runs