Web
benchme — intent tasks-style
163 tasks, v1, graded by the benchmark's own reference answers and verifier.
uploaded by the publisher of this indextask text withheldlicence unverified
In the leaderboard
Every setup scored on benchme — intent tasks, whichever other benchmarks it covers. The leaderboard compares a setup only with setups scored on exactly the same benchmarks, so its bands can differ from these.
| rank on this benchmark | Setup | Score | Pass rate | cost · tokens · time | median spend of a passing run |
|---|---|---|---|---|---|
| 1–12 | DeepSeek Harness · claude-haiku-4-5-20251001harness default, open-book on knowledge questions | 88.3 pts86.8–89.9 pts | 100.0%≥ 97.7%100.0% · 161 of 163 graded | 0.63× · 0.87× · 0.44× | $0.02 · 75.6k · 11.7scache reads not recorded |
| 1–12 | Codex · gpt-5.4-miniharness default, open-book on knowledge questions | 88.1 pts86.8–89.2 pts | 100.0%≥ 97.7%100.0% · 162 of 163 graded | 0.45× · 0.58× · 1.0× | $0.02 · 54.9k · 26.6s88% cache reads |
| 1–12 | eve · claude-haiku-4-5-20251001harness default, open-book on knowledge questions | 88.0 pts86.5–89.4 pts | 99.4%98.1–100.0%99.4% · 162 of 163 graded | 0.86× · 0.46× · 0.44× | $0.02 · 43.4k · 13.5scache reads not recorded |
| 1–12 | Continue · claude-haiku-4-5-20251001harness default, open-book on knowledge questions | 83.3 pts79.3–86.6 pts | 90.0%85.0–94.3%90.0% · 160 of 163 graded | 0.73× · 0.19× · 1.0× | $0.02 · 15k · 24.3scache reads not recorded |
| 1–12 | Hermes · claude-haiku-4-5-20251001harness default, open-book on knowledge questions | 82.0 pts81.3–82.7 pts | 100.0%≥ 97.7%100.0% · 161 of 163 graded | 0.85× · 1.1× · 1.8× | $0.02 · 122.6k · 50.6scache reads not recorded |
| 1–12 | opencode · claude-sonnet-5-5harness default, open-book on knowledge questions | 80.9 pts78.4–83.2 pts | 95.7%92.5–98.8%95.7% · 162 of 163 graded | 0.95× · 0.55× · 1.0× | $0.03 · 55.5k · 28.4scache reads not recorded |
| 1–12 | Claude Agent SDK · claude-haiku-4-5-20251001harness default, open-book on knowledge questions | 77.6 pts74.4–80.6 pts | 92.5%88.3–96.3%92.5% · 161 of 163 graded | 1.1× · 1.8× · 0.62× | $0.03 · 188.3k · 16.9scache reads not recorded |
| 1–12 | Claude Agent SDK · claude-sonnet-5-5harness default, open-book on knowledge questions | 75.1 pts70.7–78.8 pts | 84.7%78.5–89.6%84.7% · 163 of 163 graded | 1.8× · 1.3× · 0.31× | $0.05 · 129k · 8.5scache reads not recorded |
| 1–12 | Claude Code · claude-sonnet-5-5harness default, open-book on knowledge questions | 74.0 pts70.5–77.1 pts | 90.7%85.9–95.1%90.7% · 162 of 163 graded | 1.7× · 1.2× · 1.0× | $0.05 · 129k · 26.3s92% cache reads |
| 1–12 | Codex · gpt-6.1-solharness default, open-book on knowledge questions | 73.2 pts70.6–75.6 pts | 94.4%90.8–97.5%94.4% · 162 of 163 graded | 3.1× · 2.9× · 3.8× | $0.09 · 292.1k · 1m 37s91% cache reads |
| 1–12 | Claude Code · claude-haiku-4-5-20251001harness default, open-book on knowledge questions | 69.5 pts65.4–73.3 pts | 85.2%79.6–90.2%85.2% · 162 of 163 graded | 1.2× · 2.1× · 1.3× | $0.04 · 219.2k · 33.5s96% cache reads |
| 1–12 | opencode · claude-haiku-4-5-20251001harness default, open-book on knowledge questions | 68.7 pts63.6–73.6 pts | 72.8%65.6–79.6%72.8% · 162 of 163 graded | 0.42× · 0.49× · 1.1× | $0.01 · 41.4k · 28.3scache reads not recorded |
| 13 | OpenClaw · claude-haiku-4-5-20251001harness default, open-book on knowledge questions | 17.1 pts13.2–21.7 pts | 17.4%12.3–23.5%17.4% · 161 of 163 graded | 1.4× · 2.9× · 2.4× | $0.04 · 273.5k · 1m 01scache reads not recorded |
| Setup | Spend per pass · cost | Spend per pass · tokens | Spend per pass · time |
|---|---|---|---|
| DeepSeek Harness · claude-haiku-4-5-20251001 · harness default | $0.03 per pass($0.02–$0.03)26% went to the costliest 10% of attempts (17 of 161) | 121.5k per pass(106.6k–137.4k)28% went to the costliest 10% of attempts (17 of 161)cache split not recorded on 161 attempts | 18.8s per pass(15.9s–22.6s)34% went to the costliest 10% of attempts (17 of 161) |
| Codex · gpt-5.4-mini · harness default | $0.02 per pass($0.02–$0.02)20% went to the costliest 10% of attempts (17 of 162) | 66k per pass(61.3k–71.9k)21% went to the costliest 10% of attempts (17 of 162)of which 84% cache reads and 14% fresh input | 31.3s per pass(29.7s–32.7s)16% went to the costliest 10% of attempts (17 of 162) |
| eve · claude-haiku-4-5-20251001 · harness default | $0.03 per pass($0.03–$0.03)22% went to the costliest 10% of attempts (17 of 162) | 65.5k per pass(54.8k–79k)36% went to the costliest 10% of attempts (17 of 162)cache split not recorded on 162 attempts | 18s per pass(15s–22.7s)32% went to the costliest 10% of attempts (17 of 162) |
| Continue · claude-haiku-4-5-20251001 · harness default | $0.05 per pass($0.04–$0.07)43% went to the costliest 10% of attempts (16 of 160) | 50k per pass(40.2k–61.6k)44% went to the costliest 10% of attempts (16 of 160)cache split not recorded on 160 attempts | 36.8s per pass(33.9s–40.2s)19% went to the costliest 10% of attempts (16 of 160) |
| Hermes · claude-haiku-4-5-20251001 · harness default | $0.03 per pass($0.03–$0.03)21% went to the costliest 10% of attempts (17 of 161) | 130.5k per pass(121.5k–139.5k)21% went to the costliest 10% of attempts (17 of 161)cache split not recorded on 161 attempts | 54.4s per pass(52.4s–56.5s)16% went to the costliest 10% of attempts (17 of 161) |
| opencode · claude-sonnet-5-5 · harness default | $0.03 per pass($0.03–$0.04)20% went to the costliest 10% of attempts (17 of 162) | 64k per pass(59.6k–68.7k)19% went to the costliest 10% of attempts (17 of 162)cache split not recorded on 162 attempts | 34.6s per pass(32.1s–37.6s)17% went to the costliest 10% of attempts (17 of 162) |
| Claude Agent SDK · claude-haiku-4-5-20251001 · harness default | $0.04 per pass($0.04–$0.04)18% went to the costliest 10% of attempts (17 of 161) | 195.1k per pass(185.3k–205.6k)19% went to the costliest 10% of attempts (17 of 161)cache split not recorded on 161 attempts | 22.6s per pass(19.8s–26.6s)29% went to the costliest 10% of attempts (17 of 161) |
| Claude Agent SDK · claude-sonnet-5-5 · harness default | $0.07 per pass($0.06–$0.08)17% went to the costliest 10% of attempts (17 of 163) | 162.3k per pass(149.3k–178.2k)17% went to the costliest 10% of attempts (17 of 163)cache split not recorded on 163 attempts | 12.1s per pass(10.9s–13.6s)19% went to the costliest 10% of attempts (17 of 163) |
| Claude Code · claude-sonnet-5-5 · harness default | $0.06 per pass($0.06–$0.07)18% went to the costliest 10% of attempts (17 of 162) | 152.6k per pass(142.4k–166.2k)18% went to the costliest 10% of attempts (17 of 162)of which 92% cache reads and 7% fresh input | 34.1s per pass(31.4s–37.1s)17% went to the costliest 10% of attempts (17 of 162) |
| Codex · gpt-6.1-sol · harness default | $0.14 per pass($0.12–$0.15)22% went to the costliest 10% of attempts (16 of 160)2 attempts' cost were not recorded | 492.9k per pass(425.9k–562.2k)25% went to the costliest 10% of attempts (16 of 160)2 attempts' tokens were not recordedof which 93% cache reads and 6% fresh input | 2m 24s per pass(2m 07s–2m 40s)22% went to the costliest 10% of attempts (17 of 162) |
| Claude Code · claude-haiku-4-5-20251001 · harness default | $0.04 per pass($0.04–$0.05)19% went to the costliest 10% of attempts (17 of 162) | 232.9k per pass(219.7k–248k)21% went to the costliest 10% of attempts (17 of 162)of which 95% cache reads and 4% fresh input | 43.5s per pass(40.6s–47s)17% went to the costliest 10% of attempts (17 of 162) |
| opencode · claude-haiku-4-5-20251001 · harness default | $0.02 per pass($0.02–$0.02)32% went to the costliest 10% of attempts (17 of 162) | 76.9k per pass(65.1k–91.5k)35% went to the costliest 10% of attempts (17 of 162)cache split not recorded on 162 attempts | 44.5s per pass(40.1s–50.1s)17% went to the costliest 10% of attempts (17 of 162) |
| OpenClaw · claude-haiku-4-5-20251001 · harness default | $0.15 per pass($0.12–$0.21)19% went to the costliest 10% of attempts (16 of 160)1 attempt's cost was not recorded | 1M per pass(794.3k–1.4M)19% went to the costliest 10% of attempts (16 of 160)1 attempt's tokens was not recordedcache split not recorded on 160 attempts | 11m 01s per pass(6m 17s–18m 52s)55% went to the costliest 10% of attempts (17 of 161) |
This benchmark's own board: newest eligible study per setup.
live · updated 2026-10-04
A real harness session with its native tools, nothing added and nothing removed: open-book on knowledge questions.
| Setup | Pass rate · 95% CI |
|---|---|
| Codexgpt-5.4-mini · openai | 100% |
| DeepSeek Harnessclaude-haiku-4-5-20251001 · anthropic | 100% |
| Hermesclaude-haiku-4-5-20251001 · anthropic | 100% |
| eveclaude-haiku-4-5-20251001 · anthropic | 99.4%(95% CI 98.2–100) |
| opencodeclaude-sonnet-5-5 · anthropic | 95.7%(95% CI 92.6–98.8) |
| Codexgpt-6.1-sol · openai | 95.6%(95% CI 92.5–98.8) |
| Claude Agent SDKclaude-haiku-4-5-20251001 · anthropic | 92.6%(95% CI 88.2–96.3) |
| Claude Codeclaude-sonnet-5-5 · anthropic | 90.7%(95% CI 85.8–95.1) |
| Continueclaude-haiku-4-5-20251001 · anthropic | 90%(95% CI 85–94.4) |
| Claude Codeclaude-haiku-4-5-20251001 · anthropic | 85.2%(95% CI 79.6–90.7) |
| Claude Agent SDKclaude-sonnet-5-5 · anthropic | 84.7%(95% CI 79.1–89.6) |
| opencodeclaude-haiku-4-5-20251001 · anthropic | 72.8%(95% CI 66.1–79) |
| OpenClawclaude-haiku-4-5-20251001 · anthropic | 18%(95% CI 12.2–24.4) |
| Aiderclaude-haiku-4-5-20251001 · anthropic | 2.5%(95% CI 0.6–5) |
| DeepAgentsclaude-haiku-4-5-20251001 · anthropic | 0.6%(95% CI 0–1.8) |
Sorted by the benchmark's own headline figure. An order is not a claim that one setup beats another — read the intervals.
The card
Declared data only — nothing here is inferred or scored.
Identity
- Category
- Web
- Corpus
- 163 tasks · v1
- Lenses
- reference + verifier
- Graded by
- the benchmark's own reference answers and verifier
- Headline
- Pass rate
Protocol
- Repeats
- not declared
- Sample floor
- not declared
- Chance anchor
- Derived from the pinned corpus's answer formats.
Access and licence
- Dataset
- open
- Task text
- withheld — board figures publish, task text does not
- Licence
- MIT (unverified)
- Catalog
- community
Disclosures
- Contamination
- No contamination data declared.
- Conflict of interest
- Uploaded by the publisher of this index.
- Notice
- Task text is withheld from public pages until an admin verifies this licence. Running this benchmark is unaffected.
Evidence
The published studies of this benchmark: every run, trajectory and grade, on the studies site.
Published studies of benchme — intent tasks
2