Skip to content
kalanu
Theme
All published runs · 6 of 21 verified benchmarks with results

The kalanu leaderboard

score = 75 × pass rate above chance + 25 × efficiency

Early results. Setups have been compared on 6 of 6 verified benchmarks with results; an overall ranking needs at least 2. Read why →

Every metric, per setup, every group

Each comparable group's setups on their pass rate, efficiency, and cost, tokens and time as multiples of the typical setup. A setup is compared only within its group.

AIME 2025, benchme — intent tasks, GPQA Diamond and 2 more · 11 setups · 5 of 6 compared benchmarks

Colour by
  • Claude Agent SDK
  • Claude Code
  • Codex
  • Continue
  • opencode
  • Hermes
  • DeepSeek Harness
  • OpenClaw
  • eve
  • claude-haiku-4-5-20251001
  • claude-sonnet-5-5
  • gpt-5.4-mini
  • gpt-6.1-sol

Pass rate above chance

in its stored order; a bracket groups setups the data cannot split into tiers

a run that did not finish counts as a fail

  1. 1–2Codexgpt-6.1-sol96.9%, 95.5–98.2%
  2. 1–2Claude Codeclaude-sonnet-5-594.7%, 92.5–96.4%
  3. 3Codexgpt-5.4-mini88.7%, 85.1–91.9%
  4. 4–10DeepSeek Harnessclaude-haiku-4-5-2025100181.6%, 77.4–85.6%
  5. 4–10eveclaude-haiku-4-5-2025100179.6%, 75.2–83.6%
  6. 4–10Hermesclaude-haiku-4-5-2025100178.5%, 73.7–82.9%
  7. 4–10Claude Agent SDKclaude-haiku-4-5-2025100177.3%, 72.6–81.8%
  8. 4–10Continueclaude-haiku-4-5-2025100176.5%, 72.0–80.8%
  9. 4–10Claude Codeclaude-haiku-4-5-2025100176.0%, 71.9–80.1%
  10. 4–10opencodeclaude-haiku-4-5-2025100169.0%, 64.2–73.9%
  11. 11OpenClawclaude-haiku-4-5-2025100155.4%, 50.7–59.8%

Efficiency

ordered by this figure; no tiers are tested on it; highest first

100 = the lowest typical (median) spend per passing run, task by task

  1. Codexgpt-5.4-mini46.5, 43.4–50.1
  2. Continueclaude-haiku-4-5-2025100146.0, 42.2–51.6
  3. opencodeclaude-haiku-4-5-2025100139.8, 37.6–43.0
  4. eveclaude-haiku-4-5-2025100135.6, 33.0–39.7
  5. DeepSeek Harnessclaude-haiku-4-5-2025100131.7, 28.8–36.2
  6. Claude Codeclaude-sonnet-5-530.5, 27.6–32.1
  7. Codexgpt-6.1-sol26.7, 24.3–29.7
  8. Hermesclaude-haiku-4-5-2025100126.1, 24.3–27.7
  9. Claude Agent SDKclaude-haiku-4-5-2025100123.3, 20.8–26.5
  10. Claude Codeclaude-haiku-4-5-2025100117.4, 15.9–19.3
  11. OpenClawclaude-haiku-4-5-2025100116.0, 14.6–17.4

Cost × the typical setup

ordered by this figure; no tiers are tested on it; lowest first

log scale around 1×, 1/8× to 8×; a chevron past an end, its printed value not cut

  1. Codexgpt-5.4-mini0.50×
  2. opencodeclaude-haiku-4-5-202510010.57×
  3. Continueclaude-haiku-4-5-202510010.76×
  4. Hermesclaude-haiku-4-5-202510010.81×
  5. DeepSeek Harnessclaude-haiku-4-5-202510011.0×
  6. eveclaude-haiku-4-5-202510011.1×
  7. Claude Codeclaude-sonnet-5-51.2×
  8. OpenClawclaude-haiku-4-5-202510011.3×
  9. Claude Agent SDKclaude-haiku-4-5-202510011.5×
  10. Claude Codeclaude-haiku-4-5-202510011.6×
  11. Codexgpt-6.1-sol1.9×

Tokens × the typical setup

ordered by this figure; no tiers are tested on it; lowest first

log scale around 1×, 1/8× to 8×; a chevron past an end, its printed value not cut

  1. Continueclaude-haiku-4-5-202510010.28×
  2. Codexgpt-5.4-mini0.67×
  3. opencodeclaude-haiku-4-5-202510010.69×
  4. eveclaude-haiku-4-5-202510010.71×
  5. Codexgpt-6.1-sol1.0×
  6. Claude Codeclaude-sonnet-5-51.2×
  7. Hermesclaude-haiku-4-5-202510011.4×
  8. DeepSeek Harnessclaude-haiku-4-5-202510011.5×
  9. Claude Agent SDKclaude-haiku-4-5-202510011.8×
  10. Claude Codeclaude-haiku-4-5-202510012.0×
  11. OpenClawclaude-haiku-4-5-202510012.2×

Time × the typical setup

ordered by this figure; no tiers are tested on it; lowest first

log scale around 1×, 1/8× to 8×; a chevron past an end, its printed value not cut

  1. eveclaude-haiku-4-5-202510010.59×
  2. DeepSeek Harnessclaude-haiku-4-5-202510010.65×
  3. Claude Codeclaude-sonnet-5-50.74×
  4. Claude Agent SDKclaude-haiku-4-5-202510010.79×
  5. Codexgpt-5.4-mini0.89×
  6. Continueclaude-haiku-4-5-202510010.98×
  7. opencodeclaude-haiku-4-5-202510011.1×
  8. Claude Codeclaude-haiku-4-5-202510011.3×
  9. Codexgpt-6.1-sol1.4×
  10. Hermesclaude-haiku-4-5-202510011.8×
  11. OpenClawclaude-haiku-4-5-202510012.4×

Each panel draws every setup of the group, each panel in its own order. Spend (×): the typical (median) spend per passing run, as a multiple of the typical setup's.

benchme — intent tasks, GPQA Diamond, HumanEval and WindTunnel only · 2 setups · 4 of 6 compared benchmarks

Colour by
  • Claude Agent SDK
  • opencode
  • claude-sonnet-5-5

Pass rate above chance

in its stored order; a bracket groups setups the data cannot split into tiers

a run that did not finish counts as a fail

  1. 1opencodeclaude-sonnet-5-594.9%, 92.8–96.7%
  2. 2Claude Agent SDKclaude-sonnet-5-592.5%, 90.3–94.4%

Efficiency

ordered by this figure; no tiers are tested on it; highest first

100 = the lowest typical (median) spend per passing run, task by task

  1. Claude Agent SDKclaude-sonnet-5-539.9, 39.1–40.5
  2. opencodeclaude-sonnet-5-533.7, 32.4–35.3

Cost × the typical setup

ordered by this figure; no tiers are tested on it; lowest first

log scale around 1×, 1/8× to 8×; a chevron past an end, its printed value not cut

  1. opencodeclaude-sonnet-5-50.69×
  2. Claude Agent SDKclaude-sonnet-5-51.5×

Tokens × the typical setup

ordered by this figure; no tiers are tested on it; lowest first

log scale around 1×, 1/8× to 8×; a chevron past an end, its printed value not cut

  1. opencodeclaude-sonnet-5-50.58×
  2. Claude Agent SDKclaude-sonnet-5-51.5×

Time × the typical setup

ordered by this figure; no tiers are tested on it; lowest first

log scale around 1×, 1/8× to 8×; a chevron past an end, its printed value not cut

  1. Claude Agent SDKclaude-sonnet-5-50.27×
  2. opencodeclaude-sonnet-5-51.0×

Each panel draws every setup of the group, each panel in its own order. Spend (×): the typical (median) spend per passing run, as a multiple of the typical setup's.

IFEval only · 12 setups · 1 of 6 compared benchmarks

Colour by
  • Claude Agent SDK
  • Claude Code
  • Codex
  • Aider
  • opencode
  • DeepAgents
  • Hermes
  • DeepSeek Harness
  • OpenClaw
  • eve
  • claude-haiku-4-5-20251001
  • claude-sonnet-5-5
  • gpt-5.4-mini
  • gpt-6.1-sol

Pass rate

in its stored order; a bracket groups setups the data cannot split into tiers

a run that did not finish counts as a fail

  1. 1–12Codexgpt-5.4-mini100.0%, ≥ 94.8%
  2. 1–12Claude Agent SDKclaude-haiku-4-5-2025100198.6%, 95.7–100.0%
  3. 1–12Codexgpt-6.1-sol98.6%, 95.7–100.0%
  4. 1–12OpenClawclaude-haiku-4-5-2025100198.6%, 95.7–100.0%
  5. 1–12Claude Codeclaude-sonnet-5-598.6%, 95.7–100.0%
  6. 1–12Claude Codeclaude-haiku-4-5-2025100194.2%, 88.4–98.6%
  7. 1–12DeepSeek Harnessclaude-haiku-4-5-2025100182.6%, 72.5–91.3%
  8. 1–12Hermesclaude-haiku-4-5-2025100179.7%, 71.0–88.4%
  9. 1–12DeepAgentsclaude-haiku-4-5-2025100178.3%, 68.1–88.4%
  10. 1–12eveclaude-haiku-4-5-2025100175.4%, 65.2–85.5%
  11. 1–12Aiderclaude-haiku-4-5-2025100171.0%, 60.9–81.2%
  12. 1–12opencodeclaude-haiku-4-5-2025100165.2%, 55.1–75.4%

Efficiency

ordered by this figure; no tiers are tested on it; highest first

100 = the lowest typical (median) spend per passing run, task by task

  1. Aiderclaude-haiku-4-5-2025100167.7, 63.3–70.9
  2. DeepAgentsclaude-haiku-4-5-2025100160.5, 48.7–65.0
  3. opencodeclaude-haiku-4-5-2025100155.1, 50.6–57.9
  4. Codexgpt-5.4-mini53.9, 44.5–59.8
  5. eveclaude-haiku-4-5-2025100153.5, 46.6–60.8
  6. DeepSeek Harnessclaude-haiku-4-5-2025100146.4, 40.0–51.2
  7. Claude Agent SDKclaude-haiku-4-5-2025100143.9, 35.0–47.8
  8. Hermesclaude-haiku-4-5-2025100139.9, 36.1–42.0
  9. Claude Codeclaude-haiku-4-5-2025100125.2, 20.7–28.4
  10. Codexgpt-6.1-sol21.1, 18.1–23.4
  11. OpenClawclaude-haiku-4-5-2025100119.8, 17.4–21.8
  12. Claude Codeclaude-sonnet-5-519.1, 15.5–21.3

Cost × the typical setup

ordered by this figure; no tiers are tested on it; lowest first

log scale around 1×, 1/8× to 8×; a chevron past an end, its printed value not cut

  1. opencodeclaude-haiku-4-5-202510010.35×
  2. Codexgpt-5.4-mini0.37×
  3. Hermesclaude-haiku-4-5-202510010.45×
  4. Aiderclaude-haiku-4-5-202510010.49×
  5. DeepSeek Harnessclaude-haiku-4-5-202510010.74×
  6. OpenClawclaude-haiku-4-5-202510010.92×
  7. eveclaude-haiku-4-5-202510011.1×
  8. Claude Codeclaude-haiku-4-5-202510011.1×
  9. DeepAgentsclaude-haiku-4-5-202510011.1×
  10. Claude Agent SDKclaude-haiku-4-5-202510011.1×
  11. Claude Codeclaude-sonnet-5-52.4×
  12. Codexgpt-6.1-sol3.4×

Tokens × the typical setup

ordered by this figure; no tiers are tested on it; lowest first

log scale around 1×, 1/8× to 8×; a chevron past an end, its printed value not cut

  1. Aiderclaude-haiku-4-5-202510010.21×
  2. DeepAgentsclaude-haiku-4-5-202510010.42×
  3. eveclaude-haiku-4-5-202510010.45×
  4. opencodeclaude-haiku-4-5-202510010.67×
  5. Codexgpt-5.4-mini0.83×
  6. Codexgpt-6.1-sol0.95×
  7. DeepSeek Harnessclaude-haiku-4-5-202510011.0×
  8. Hermesclaude-haiku-4-5-202510011.1×
  9. Claude Codeclaude-sonnet-5-51.9×
  10. Claude Agent SDKclaude-haiku-4-5-202510012.3×
  11. Claude Codeclaude-haiku-4-5-202510012.6×
  12. OpenClawclaude-haiku-4-5-202510012.8×

Time × the typical setup

ordered by this figure; no tiers are tested on it; lowest first

log scale around 1×, 1/8× to 8×; a chevron past an end, its printed value not cut

  1. DeepAgentsclaude-haiku-4-5-202510010.35×
  2. Claude Agent SDKclaude-haiku-4-5-202510010.39×
  3. eveclaude-haiku-4-5-202510010.44×
  4. DeepSeek Harnessclaude-haiku-4-5-202510010.50×
  5. Codexgpt-5.4-mini0.85×
  6. Claude Codeclaude-haiku-4-5-202510011.0×
  7. opencodeclaude-haiku-4-5-202510011.0×
  8. Aiderclaude-haiku-4-5-202510011.1×
  9. Claude Codeclaude-sonnet-5-51.1×
  10. Codexgpt-6.1-sol1.2×
  11. Hermesclaude-haiku-4-5-202510011.6×
  12. OpenClawclaude-haiku-4-5-202510012.6×

Each panel draws every setup of the group, each panel in its own order. Spend (×): the typical (median) spend per passing run, as a multiple of the typical setup's.

3 of 4 shown The rest, in the table →

More groups, in the table
Compared with no other setup