Knowledge & reasoning
AIME 2025-style
30 tasks, v1, graded by the benchmark's own reference answers.
task text withheld
In the leaderboard
Every setup scored on AIME 2025, whichever other benchmarks it covers. The leaderboard compares a setup only with setups scored on exactly the same benchmarks, so its bands can differ from these.
| rank on this benchmark | Setup | Score | Pass rate | cost · tokens · time | median spend of a passing run |
|---|---|---|---|---|---|
| 1–12 | Codex · gpt-6.1-solharness default, open-book on knowledge questions | 94.8 pts92.2–98.5 pts | 100.0%≥ 88.4%100.0% · 30 of 30 graded | 0.53× · 0.25× · 0.45× | $0.03 · 12.4k · 22.5s0% cache reads |
| 1–12 | Claude Code · claude-sonnet-5-5harness default, open-book on knowledge questions | 91.0 pts83.6–95.2 pts | 96.7%90.0–100.0%96.7% · 30 of 30 graded | 0.46× · 0.60× · 0.30× | $0.02 · 29.8k · 18.4s86% cache reads |
| 1–12 | Codex · gpt-5.4-miniharness default, open-book on knowledge questions | 85.8 pts77.1–94.0 pts | 90.0%80.0–100.0%90.0% · 30 of 30 graded | 0.37× · 0.35× · 0.63× | $0.01 · 14.6k · 26.4s67% cache reads |
| 1–12 | Continue · claude-haiku-4-5-20251001harness default, open-book on knowledge questions | 72.8 pts63.1–81.7 pts | 86.7%73.3–96.7%86.7% · 30 of 30 graded | 1.1× · 0.80× · 1.0× | $0.06 · 43.1k · 57.6scache reads not recorded |
| 1–12 | eve · claude-haiku-4-5-20251001harness default, open-book on knowledge questions | 70.2 pts59.4–80.0 pts | 83.3%70.0–96.7%83.3% · 30 of 30 graded | 1.0× · 1.0× · 0.95× | $0.05 · 60.8k · 58.9scache reads not recorded |
| 1–12 | DeepSeek Harness · claude-haiku-4-5-20251001harness default, open-book on knowledge questions | 70.1 pts59.8–79.4 pts | 86.7%73.3–96.7%86.7% · 30 of 30 graded | 1.5× · 2.9× · 1.1× | $0.07 · 196.2k · 1m 06scache reads not recorded |
| 1–12 | Claude Code · claude-haiku-4-5-20251001harness default, open-book on knowledge questions | 68.3 pts59.7–76.0 pts | 83.3%73.3–92.0%83.3% · 30 of 30 graded | 1.5× · 1.1× · 1.4× | $0.10 · 52.7k · 1m 58s72% cache reads |
| 1–12 | opencode · claude-haiku-4-5-20251001harness default, open-book on knowledge questions | 66.6 pts55.2–77.1 pts | 80.0%63.3–93.3%80.0% · 30 of 30 graded | 1.0× · 1.4× · 1.2× | $0.04 · 83.5k · 50.6scache reads not recorded |
| 1–12 | Claude Agent SDK · claude-haiku-4-5-20251001harness default, open-book on knowledge questions | 65.0 pts53.3–76.3 pts | 76.7%60.0–90.0%76.7% · 30 of 30 graded | 1.1× · 0.98× · 1.0× | $0.08 · 45.9k · 1m 36scache reads not recorded |
| 1–12 | OpenClaw · claude-haiku-4-5-20251001harness default, open-book on knowledge questions | 59.3 pts46.6–71.6 pts | 70.0%53.3–86.7%70.0% · 30 of 30 graded | 0.90× · 1.0× · 1.9× | $0.05 · 45.4k · 1m 41scache reads not recorded |
| 1–12 | Hermes · claude-haiku-4-5-20251001harness default, open-book on knowledge questions | 55.0 pts41.8–66.9 pts | 66.7%50.0–83.3%66.7% · 30 of 30 graded | 1.1× · 2.4× · 2.0× | $0.05 · 105.3k · 1m 33scache reads not recorded |
| 1–12 | DeepAgents · claude-haiku-4-5-20251001harness default, open-book on knowledge questions | 44.0 pts31.3–59.4 pts | 53.3%36.7–70.0%53.3% · 30 of 30 graded | 1.9× · 2.9× · 1.5× | $0.11 · 149.6k · 1m 36scache reads not recorded |
| Setup | Spend per pass · cost | Spend per pass · tokens | Spend per pass · time |
|---|---|---|---|
| Codex · gpt-6.1-sol · harness default | $0.03 per pass($0.03–$0.04)17% went to the costliest 10% of attempts (3 of 30) | 13.6k per pass(12.4k–15.8k)18% went to the costliest 10% of attempts (3 of 30)of which 6% cache reads and 88% fresh input | 29.6s per pass(24.1s–36.7s)25% went to the costliest 10% of attempts (3 of 30) |
| Claude Code · claude-sonnet-5-5 · harness default | $0.03 per pass($0.02–$0.04)25% went to the costliest 10% of attempts (3 of 30) | 37.2k per pass(30.9k–46.8k)25% went to the costliest 10% of attempts (3 of 30)of which 83% cache reads and 14% fresh input | 22.3s per pass(19.5s–25.3s)17% went to the costliest 10% of attempts (3 of 30) |
| Codex · gpt-5.4-mini · harness default | $0.03 per pass($0.02–$0.04)28% went to the costliest 10% of attempts (3 of 30) | 36.5k per pass(23.5k–50.7k)36% went to the costliest 10% of attempts (3 of 30)of which 66% cache reads and 19% fresh input | 49.7s per pass(37s–1m 03s)25% went to the costliest 10% of attempts (3 of 30) |
| Continue · claude-haiku-4-5-20251001 · harness default | $0.16 per pass($0.09–$0.24)45% went to the costliest 10% of attempts (3 of 29)1 attempt's cost was not recorded | 116.2k per pass(61.8k–185.1k)49% went to the costliest 10% of attempts (3 of 29)1 attempt's tokens was not recordedcache split not recorded on 29 attempts | 2m 13s per pass(1m 25s–3m 19s)36% went to the costliest 10% of attempts (3 of 30) |
| eve · claude-haiku-4-5-20251001 · harness default | $0.09 per pass($0.07–$0.13)28% went to the costliest 10% of attempts (3 of 30) | 147.2k per pass(95.8k–209.5k)34% went to the costliest 10% of attempts (3 of 30)cache split not recorded on 30 attempts | 2m 37s per pass(1m 28s–4m 13s)37% went to the costliest 10% of attempts (3 of 30) |
| DeepSeek Harness · claude-haiku-4-5-20251001 · harness default | $0.15 per pass($0.10–$0.21)33% went to the costliest 10% of attempts (3 of 29)1 attempt's cost was not recorded | 381.7k per pass(244.7k–572k)35% went to the costliest 10% of attempts (3 of 29)1 attempt's tokens was not recordedcache split not recorded on 29 attempts | 4m 11s per pass(1m 45s–7m 54s)56% went to the costliest 10% of attempts (3 of 30) |
| Claude Code · claude-haiku-4-5-20251001 · harness default | $0.16 per pass($0.11–$0.24)35% went to the costliest 10% of attempts (15 of 149)1 attempt's cost was not recorded | 185.4k per pass(92.6k–327.1k)62% went to the costliest 10% of attempts (15 of 149)1 attempt's tokens was not recordedof which 81% cache reads and 4% fresh input | 3m 46s per pass(2m 25s–5m 39s)38% went to the costliest 10% of attempts (15 of 150) |
| opencode · claude-haiku-4-5-20251001 · harness default | $0.13 per pass($0.07–$0.20)44% went to the costliest 10% of attempts (3 of 30) | 333k per pass(147.9k–574.5k)59% went to the costliest 10% of attempts (3 of 30)cache split not recorded on 30 attempts | 3m 58s per pass(1m 50s–6m 28s)55% went to the costliest 10% of attempts (3 of 30) |
| Claude Agent SDK · claude-haiku-4-5-20251001 · harness default | $0.16 per pass($0.10–$0.25)33% went to the costliest 10% of attempts (3 of 28)2 attempts' cost were not recorded | 155.3k per pass(72.6k–295.5k)54% went to the costliest 10% of attempts (3 of 28)2 attempts' tokens were not recordedcache split not recorded on 28 attempts | 3m 23s per pass(2m 03s–5m 26s)30% went to the costliest 10% of attempts (3 of 30) |
| OpenClaw · claude-haiku-4-5-20251001 · harness default | $0.10 per pass($0.07–$0.16)25% went to the costliest 10% of attempts (3 of 30) | 122.6k per pass(61.2k–255.1k)50% went to the costliest 10% of attempts (3 of 30)cache split not recorded on 30 attempts | 3m 07s per pass(2m 11s–4m 39s)22% went to the costliest 10% of attempts (3 of 30) |
| Hermes · claude-haiku-4-5-20251001 · harness default | $0.14 per pass($0.08–$0.25)43% went to the costliest 10% of attempts (3 of 27)3 attempts' cost were not recorded | 376.2k per pass(194.1k–688.5k)48% went to the costliest 10% of attempts (3 of 27)3 attempts' tokens were not recordedcache split not recorded on 27 attempts | 4m 15s per pass(2m 41s–6m 53s)28% went to the costliest 10% of attempts (3 of 30) |
| DeepAgents · claude-haiku-4-5-20251001 · harness default | $0.31 per pass($0.18–$0.52)32% went to the costliest 10% of attempts (3 of 25)5 attempts' cost were not recorded | 588.4k per pass(322.4k–1M)40% went to the costliest 10% of attempts (3 of 25)5 attempts' tokens were not recordedcache split not recorded on 25 attempts | 6m 02s per pass(3m 29s–10m 36s)25% went to the costliest 10% of attempts (3 of 30) |
This benchmark's own board: newest eligible study per setup.
live · updated 2026-10-08
A real harness session with its native tools, nothing added and nothing removed: open-book on knowledge questions.
| Setup | Pass rate · 95% CI |
|---|---|
| Codexgpt-6.1-sol · openai | 100% |
| Claude Codeclaude-sonnet-5-5 · anthropic | 96.7%(95% CI 90–100) |
| eveclaude-haiku-4-5-20251001 · anthropic | 92.6%(95% CI 81.5–100) |
| Codexgpt-5.4-mini · openai | 90%(95% CI 76.7–100) |
| Continueclaude-haiku-4-5-20251001 · anthropic | 89.7%(95% CI 75.9–100) |
| DeepSeek Harnessclaude-haiku-4-5-20251001 · anthropic | 89.7%(95% CI 79.3–100) |
| Claude Codeclaude-haiku-4-5-20251001 · anthropic | 82.8%(95% CI 69–96.6) |
| Claude Agent SDKclaude-haiku-4-5-20251001 · anthropic | 82.1%(95% CI 67.9–96.4) |
| opencodeclaude-haiku-4-5-20251001 · anthropic | 80%(95% CI 66.7–93.3) |
| Hermesclaude-haiku-4-5-20251001 · anthropic | 74.1%(95% CI 55.6–88.9) |
| OpenClawclaude-haiku-4-5-20251001 · anthropic | 70%(95% CI 53.3–86.7) |
| DeepAgentsclaude-haiku-4-5-20251001 · anthropic | 64%(95% CI 44–84) |
| Aiderclaude-haiku-4-5-20251001 · anthropic | 28.6%(95% CI 14.3–46.4) |
Sorted by the benchmark's own headline figure. An order is not a claim that one setup beats another — read the intervals.
The card
Declared data only — nothing here is inferred or scored.
Identity
- Category
- Knowledge & reasoning
- Corpus
- 30 tasks · v1
- Lenses
- reference
- Graded by
- the benchmark's own reference answers
- Headline
- Pass rate
Protocol
- Repeats
- 4 recommended
- Sample floor
- 30 tasks
- Chance anchor
- Derived from the pinned corpus's answer formats.
Access and licence
- Dataset
- open
- Task text
- withheld — board figures publish, task text does not
- Licence
- MIT
- Catalog
- featured
Disclosures
- Contamination
- No contamination data declared.
- Conflict of interest
- None declared.
Evidence
The published studies of this benchmark: every run, trajectory and grade, on the studies site.
Published studies of AIME 2025
2