Knowledge & reasoning
IFEval-style
69 tasks, v1, graded by the benchmark's own declared checks.
In the leaderboard
Every setup scored on IFEval, whichever other benchmarks it covers. The leaderboard compares a setup only with setups scored on exactly the same benchmarks, so its bands can differ from these.
| rank on this benchmark | Setup | Score | Pass rate | cost · tokens · time | median spend of a passing run |
|---|---|---|---|---|---|
| 1–12 | Codex · gpt-5.4-minias published | 88.5 pts86.1–89.9 pts | 100.0%≥ 94.8%100.0% · 69 of 69 graded | 0.37× · 0.83× · 0.85× | $0.0029 · 10.7k · 14.5s91% cache reads |
| 1–12 | Claude Agent SDK · claude-haiku-4-5-20251001as published | 84.9 pts81.9–86.8 pts | 98.6%95.7–100.0%98.6% · 69 of 69 graded | 1.1× · 2.3× · 0.39× | $0.01 · 29.9k · 7.1scache reads not recorded |
| 1–12 | Codex · gpt-6.1-solas published | 79.2 pts76.3–80.7 pts | 98.6%95.7–100.0%98.6% · 69 of 69 graded | 3.4× · 0.95× · 1.2× | $0.03 · 12.1k · 22.3s0% cache reads |
| 1–12 | OpenClaw · claude-haiku-4-5-20251001as published | 78.9 pts76.4–80.3 pts | 98.6%95.7–100.0%98.6% · 69 of 69 graded | 0.92× · 2.8× · 2.6× | $0.01 · 37.4k · 48.7scache reads not recorded |
| 1–12 | Claude Code · claude-sonnet-5-5as published | 78.7 pts76.2–80.2 pts | 98.6%95.7–100.0%98.6% · 69 of 69 graded | 2.4× · 1.9× · 1.1× | $0.02 · 29.7k · 21.4s87% cache reads |
| 1–12 | Claude Code · claude-haiku-4-5-20251001as published | 76.9 pts72.6–80.6 pts | 94.2%88.4–98.6%94.2% · 69 of 69 graded | 1.1× · 2.6× · 1.0× | $0.01 · 34.7k · 18.3s90% cache reads |
| 1–12 | DeepAgents · claude-haiku-4-5-20251001as published | 73.8 pts65.6–81.0 pts | 78.3%68.1–88.4%78.3% · 69 of 69 graded | 1.1× · 0.42× · 0.35× | $0.01 · 9.1k · 6.3scache reads not recorded |
| 1–12 | DeepSeek Harness · claude-haiku-4-5-20251001as published | 73.6 pts65.5–80.1 pts | 82.6%72.5–91.3%82.6% · 69 of 69 graded | 0.74× · 1.0× · 0.50× | $0.01 · 22.3k · 9.7scache reads not recorded |
| 1–12 | Aider · claude-haiku-4-5-20251001as published | 70.2 pts62.0–78.2 pts | 71.0%60.9–81.2%71.0% · 69 of 69 graded | 0.49× · 0.21× · 1.1× | $0.0038 · 2.7k · 20.4scache reads not recorded |
| 1–12 | eve · claude-haiku-4-5-20251001as published | 69.9 pts61.2–78.5 pts | 75.4%65.2–85.5%75.4% · 69 of 69 graded | 1.1× · 0.45× · 0.44× | $0.01 · 8.7k · 8.3scache reads not recorded |
| 1–12 | Hermes · claude-haiku-4-5-20251001as published | 69.8 pts62.8–76.7 pts | 79.7%71.0–88.4%79.7% · 69 of 69 graded | 0.45× · 1.1× · 1.6× | $0.0035 · 13.2k · 30.4scache reads not recorded |
| 1–12 | opencode · claude-haiku-4-5-20251001as published | 62.7 pts54.5–70.5 pts | 65.2%55.1–75.4%65.2% · 69 of 69 graded | 0.35× · 0.67× · 1.0× | $0.0026 · 8.3k · 18.5scache reads not recorded |
| Setup | Spend per pass · cost | Spend per pass · tokens | Spend per pass · time |
|---|---|---|---|
| Codex · gpt-5.4-mini · as published | $0.01 per pass($0.0050–$0.01)39% went to the costliest 10% of attempts (7 of 69) | 14k per pass(11.2k–18.3k)31% went to the costliest 10% of attempts (7 of 69)of which 60% cache reads and 36% fresh input | 17.5s per pass(15.9s–19.2s)20% went to the costliest 10% of attempts (7 of 69) |
| Claude Agent SDK · claude-haiku-4-5-20251001 · as published | $0.01 per pass($0.01–$0.02)30% went to the costliest 10% of attempts (7 of 69) | 35k per pass(32.1k–38.5k)20% went to the costliest 10% of attempts (7 of 69)cache split not recorded on 69 attempts | 11.9s per pass(9.2s–15.2s)36% went to the costliest 10% of attempts (7 of 69) |
| Codex · gpt-6.1-sol · as published | $0.03 per pass($0.03–$0.04)21% went to the costliest 10% of attempts (7 of 69) | 15.2k per pass(13.5k–17.1k)23% went to the costliest 10% of attempts (7 of 69)of which 5% cache reads and 92% fresh input | 25.6s per pass(23.1s–28.7s)20% went to the costliest 10% of attempts (7 of 69) |
| OpenClaw · claude-haiku-4-5-20251001 · as published | $0.01 per pass($0.01–$0.01)28% went to the costliest 10% of attempts (7 of 69) | 42.3k per pass(38.7k–47k)19% went to the costliest 10% of attempts (7 of 69)cache split not recorded on 69 attempts | 52.2s per pass(49.3s–56s)15% went to the costliest 10% of attempts (7 of 69) |
| Claude Code · claude-sonnet-5-5 · as published | $0.03 per pass($0.02–$0.03)27% went to the costliest 10% of attempts (7 of 69) | 32k per pass(29.9k–35k)15% went to the costliest 10% of attempts (7 of 69)of which 85% cache reads and 11% fresh input | 24.8s per pass(21.8s–29.2s)22% went to the costliest 10% of attempts (7 of 69) |
| Claude Code · claude-haiku-4-5-20251001 · as published | $0.01 per pass($0.01–$0.02)27% went to the costliest 10% of attempts (7 of 69) | 40.1k per pass(36.9k–44.2k)17% went to the costliest 10% of attempts (7 of 69)of which 90% cache reads and 8% fresh input | 22.9s per pass(19.9s–27.6s)22% went to the costliest 10% of attempts (7 of 69) |
| DeepAgents · claude-haiku-4-5-20251001 · as published | $0.02 per pass($0.01–$0.02)22% went to the costliest 10% of attempts (7 of 69) | 10.8k per pass(9.1k–12.7k)19% went to the costliest 10% of attempts (7 of 69)cache split not recorded on 69 attempts | 10.8s per pass(8.8s–13s)27% went to the costliest 10% of attempts (7 of 69) |
| DeepSeek Harness · claude-haiku-4-5-20251001 · as published | $0.01 per pass($0.01–$0.02)38% went to the costliest 10% of attempts (7 of 69) | 26k per pass(21.7k–31.3k)26% went to the costliest 10% of attempts (7 of 69)cache split not recorded on 69 attempts | 16.9s per pass(12.6s–22.9s)37% went to the costliest 10% of attempts (7 of 69) |
| Aider · claude-haiku-4-5-20251001 · as published | $0.01 per pass($0.01–$0.01)22% went to the costliest 10% of attempts (7 of 69) | 3.9k per pass(3.4k–4.6k)14% went to the costliest 10% of attempts (7 of 69)cache split not recorded on 69 attempts | 30.2s per pass(26.3s–35.4s)15% went to the costliest 10% of attempts (7 of 69) |
| eve · claude-haiku-4-5-20251001 · as published | $0.02 per pass($0.01–$0.02)28% went to the costliest 10% of attempts (7 of 69) | 15.1k per pass(9.8k–23.4k)43% went to the costliest 10% of attempts (7 of 69)cache split not recorded on 69 attempts | 12.7s per pass(10.4s–15.3s)26% went to the costliest 10% of attempts (7 of 69) |
| Hermes · claude-haiku-4-5-20251001 · as published | $0.01 per pass($0.0050–$0.01)29% went to the costliest 10% of attempts (7 of 69) | 21.6k per pass(18k–25.8k)18% went to the costliest 10% of attempts (7 of 69)cache split not recorded on 69 attempts | 40.1s per pass(35.4s–45.8s)14% went to the costliest 10% of attempts (7 of 69) |
| opencode · claude-haiku-4-5-20251001 · as published | $0.0046 per pass($0.0036–$0.01)32% went to the costliest 10% of attempts (7 of 69) | 14k per pass(11.8k–16.7k)19% went to the costliest 10% of attempts (7 of 69)cache split not recorded on 69 attempts | 29.7s per pass(25.5s–35.5s)15% went to the costliest 10% of attempts (7 of 69) |
This benchmark's own board: newest eligible study per setup.
live
No rows on this board yet.
A row appears here the moment a study of this benchmark is published.
The card
Declared data only — nothing here is inferred or scored.
Identity
- Category
- Knowledge & reasoning
- Corpus
- 69 tasks · v1
- Lenses
- evals
- Graded by
- the benchmark's own declared checks
- Headline
- Pass rate
Protocol
- Repeats
- 1 recommended
- Sample floor
- 100 tasks
- Chance anchor
- Derived from the pinned corpus's answer formats.
Access and licence
- Dataset
- open
- Task text
- available
- Licence
- Apache-2.0
- Catalog
- featured
Disclosures
- Contamination
- No contamination data declared.
- Conflict of interest
- None declared.
Evidence
The published studies of this benchmark: every run, trajectory and grade, on the studies site.
Published studies of IFEval
1