~/system-one-bench / benchmarks/typed-decisions
Text track · English · All benchmarks
Agent and product states, 5 questions per case. Dexy-Decision-9B has the top score, 1.0 points ahead of Microsoft-Decision-1, but Microsoft-Decision-1 and 8 more are within sampling noise: a statistical tie for first.
| Rank | Model | Accuracy | 95% interval | Macro-F1 | Calibration error | Failed | |
|---|---|---|---|---|---|---|---|
| 1 | Dexy-Decision-9B Patrick-18Leaderin-domain | 80.0 | 76.1–83.9 | 78.8 | 11.3% | 0 | |
| 2 | 79.0 | 75.0–83.0 | 77.9 | 2.6% | 0 | ||
| 3 | Celeris-1 Decision CelerisTied for 1st | 76.4 | 72.3–80.6 | 74.7 | 4.8% | 0 | |
| 4 | Mammoth 1 BasetenTied for 1st | 76.4 | 72.2–80.6 | 76.2 | 11.0% | 0 | |
| 5 | 75.1 | 70.9–79.4 | 74.3 | 1.6% | 0 | ||
| 6 | GLiDE FastinoTied for 1st | 75.0 | 70.8–79.3 | 73.6 | 6.3% | 0 | |
| 7 | d1-ultra Liquid AITied for 1st | 75.0 | 70.8–79.2 | 74.5 | 3.1% | 0 | |
| 8 | d1 Liquid AITied for 1st | 74.5 | 70.2–78.7 | 73.5 | 3.2% | 0 | |
| 9 | Elephant 1 BasetenTied for 1st | 74.2 | 69.9–78.5 | 72.5 | 10.5% | 0 | |
| 9 | 74.2 | 69.9–78.5 | 74.8 | 15.2% | 0 | ||
| 11 | Jev 1.14 TypeSafe | 74.2 | 69.9–78.4 | 74.5 | 5.0% | 0 | |
| 12 | 74.0 | 69.7–78.3 | 72.4 | 8.4% | 0 | ||
| 13 | 73.0 | 68.6–77.4 | 72.8 | 4.3% | 0 | ||
| 13 | 73.0 | 68.6–77.4 | 72.8 | 12.0% | 0 | ||
| 15 | Jev 1.13 TypeSafe | 72.7 | 68.3–77.0 | 72.9 | 4.0% | 0 | |
| 16 | 72.2 | 67.9–76.6 | 69.6 | 2.6% | 0 | ||
| 17 | Span-02 Respan | 72.1 | 67.7–76.5 | 70.9 | 13.0% | 0 | |
| 18 | d1.1 Liquid AI | 72.0 | 67.6–76.4 | 72.0 | 5.9% | 0 | |
| 19 | Mercury Decide Inception | 71.1 | 66.7–75.5 | 69.5 | 18.8% | 0 | |
| 20 | 71.0 | 66.6–75.5 | 71.6 | 8.4% | 0 | ||
| 21 | 70.5 | 66.1–75.0 | 67.9 | 1.2% | 0 | ||
| 22 | 69.9 | 65.4–74.4 | 70.1 | 21.0% | 0 | ||
| 23 | 68.7 | 64.1–73.2 | 66.7 | 4.5% | 0 | ||
| 24 | 67.0 | 62.4–71.7 | 66.3 | 4.7% | 0 | ||
| 25 | 67.0 | 62.4–71.6 | 67.1 | 2.7% | 0 | ||
| 26 | Kev 4B Jared Palmer | 66.3 | 61.7–71.0 | 65.4 | 9.0% | 0 | |
| 27 | d1-3B Liquid AI | 64.8 | 60.2–69.5 | 62.4 | 3.1% | 0 | |
| 28 | 64.8 | 60.1–69.4 | 63.0 | 13.6% | 0 | ||
| 29 | 62.6 | 57.9–67.4 | 61.6 | 11.5% | 0 | ||
| 30 | 62.5 | 57.8–67.3 | 63.0 | 12.5% | 0 | ||
| 31 | Joey 1 Baseten | 62.4 | 57.6–67.1 | 60.2 | 6.0% | 0 | |
| 32 | d1-omni-600M Liquid AI | 57.5 | 52.6–62.3 | 56.0 | 5.7% | 0 | |
| 33 | Naluri-308M Jericho Siahaya | 54.7 | 49.8–59.6 | 43.6 | 6.5% | 0 | |
| 34 | 52.6 | 47.7–57.5 | 50.6 | 13.4% | 0 | ||
| 35 | Naluri-279M Jericho Siahaya | 47.3 | 42.4–52.2 | 45.0 | 7.8% | 0 | |
| 36 | Kev 0.8B Jared Palmer | 46.3 | 41.4–51.2 | 45.7 | 4.6% | 0 | |
| 37 | 42.3 | 37.5–47.1 | 38.3 | 34.4% | 0 | ||
| 38 | Laya Router Convai Innovations | 36.4 | 31.6–41.1 | 32.7 | 17.3% | 0 | |
| 39 | Classify-1 ScaleDown | 27.2 | 22.8–31.5 | 8.5 | 4.6% | 2000 |
The bar is accuracy; the bracket on it is the 95% interval for 2,000 questions over 400 cases, counted as 400 samples. Tied for 1st marks every model whose gap to the leader is within sampling noise (1.96 standard errors of the difference, treating the runs as independent). A failed call counts as wrong. In-domain models trained on another split of this dataset, by their maker's own account.
It counts towards the System One Index and its English sub-index. Every model gets the same cases and the same request, under the benchmark rules.