~/system-one-bench / benchmarks/boolq
Text track · English · All benchmarks
Reading comprehension, yes/no. Drex Max has the top score, 0.5 points ahead of Drex v2.0, but Drex v2.0 and 23 more are within sampling noise: a statistical tie for first.
| Rank | Model | Accuracy | 95% interval | Macro-F1 | Calibration error | Failed | |
|---|---|---|---|---|---|---|---|
| 1 | 93.0 | 89.5–96.5 | 92.5 | 1.9% | 0 | ||
| 2 | 92.5 | 88.8–96.2 | 92.1 | 3.3% | 0 | ||
| 3 | d1.1 Liquid AITied for 1st | 92.0 | 88.2–95.8 | 91.5 | 2.6% | 0 | |
| 3 | Mammoth 1 BasetenTied for 1st | 92.0 | 88.2–95.8 | 91.4 | 4.0% | 0 | |
| 3 | GLiDE FastinoTied for 1st | 92.0 | 88.2–95.8 | 91.6 | 2.8% | 0 | |
| 6 | Kev 4B Jared PalmerTied for 1stin-domain | 91.5 | 87.6–95.4 | 90.9 | 2.1% | 0 | |
| 6 | 91.5 | 87.6–95.4 | 91.0 | 2.4% | 0 | ||
| 6 | d1-ultra Liquid AITied for 1st | 91.5 | 87.6–95.4 | 91.0 | 3.2% | 0 | |
| 9 | 91.0 | 87.0–95.0 | 90.5 | 6.1% | 0 | ||
| 9 | 91.0 | 87.0–95.0 | 90.7 | 5.8% | 0 | ||
| 9 | Elephant 1 BasetenTied for 1st | 91.0 | 87.0–95.0 | 90.4 | 4.4% | 0 | |
| 12 | Mercury Decide InceptionTied for 1st | 90.5 | 86.4–94.6 | 90.1 | 5.3% | 0 | |
| 13 | Jev 1.14 TypeSafeTied for 1st | 90.0 | 85.8–94.2 | 89.6 | 2.7% | 0 | |
| 13 | d1 Liquid AITied for 1st | 90.0 | 85.8–94.2 | 89.4 | 4.2% | 0 | |
| 13 | 90.0 | 85.8–94.2 | 89.4 | 7.2% | 0 | ||
| 13 | Span-02 RespanTied for 1st | 90.0 | 85.8–94.2 | 89.4 | 6.1% | 0 | |
| 17 | Jev 1.13 TypeSafeTied for 1st | 89.5 | 85.3–93.7 | 89.1 | 3.9% | 0 | |
| 17 | Celeris-1 Decision CelerisTied for 1st | 89.5 | 85.3–93.7 | 88.8 | 3.1% | 0 | |
| 17 | Dexy-Decision-9B Patrick-18Tied for 1st | 89.5 | 85.3–93.7 | 88.9 | 5.4% | 0 | |
| 20 | 89.0 | 84.7–93.3 | 88.3 | 6.6% | 0 | ||
| 20 | 89.0 | 84.7–93.3 | 88.3 | 3.2% | 0 | ||
| 20 | 89.0 | 84.7–93.3 | 88.5 | 3.6% | 0 | ||
| 23 | 88.5 | 84.1–92.9 | 87.9 | 6.6% | 0 | ||
| 23 | 88.5 | 84.1–92.9 | 87.9 | 6.7% | 0 | ||
| 25 | 88.0 | 83.5–92.5 | 87.4 | 4.5% | 0 | ||
| 26 | 87.0 | 82.3–91.7 | 86.5 | 9.9% | 0 | ||
| 27 | 86.0 | 81.2–90.8 | 85.4 | 7.5% | 0 | ||
| 28 | 85.5 | 80.6–90.4 | 84.8 | 8.6% | 0 | ||
| 29 | d1-3B Liquid AI | 85.0 | 80.1–89.9 | 84.1 | 6.1% | 0 | |
| 30 | Laya Router Convai Innovations | 81.5 | 76.1–86.9 | 80.0 | 9.3% | 0 | |
| 31 | 81.0 | 75.6–86.4 | 79.7 | 8.5% | 0 | ||
| 32 | Joey 1 Baseten | 80.0 | 74.5–85.5 | 78.4 | 4.7% | 0 | |
| 32 | 80.0 | 74.5–85.5 | 78.9 | 8.6% | 0 | ||
| 34 | 79.5 | 73.9–85.1 | 78.5 | 5.8% | 0 | ||
| 35 | Kev 0.8B Jared Palmerin-domain | 76.0 | 70.1–81.9 | 72.9 | 12.0% | 0 | |
| 36 | d1-omni-600M Liquid AI | 74.0 | 67.9–80.1 | 73.1 | 13.1% | 0 | |
| 37 | Naluri-308M Jericho Siahaya | 56.5 | 49.6–63.4 | 55.3 | 19.1% | 0 | |
| 38 | Naluri-279M Jericho Siahaya | 54.5 | 47.6–61.4 | 47.5 | 13.6% | 0 | |
| 39 | Classify-1 ScaleDown | 38.0 | 31.3–44.7 | 27.5 | 12.0% | 200 |
The bar is accuracy; the bracket on it is the 95% interval for 200 cases. Tied for 1st marks every model whose gap to the leader is within sampling noise (1.96 standard errors of the difference, treating the runs as independent). A failed call counts as wrong. In-domain models trained on another split of this dataset, by their maker's own account.
It counts towards the System One Index and its English sub-index. Every model gets the same cases and the same request, under the benchmark rules.