~/system-one-bench / benchmarks/ag-news
Text track · English · All benchmarks
News topic, 4 options. Laya Router has the top score, 0.5 points ahead of Mercury Decide, but Mercury Decide and 28 more are within sampling noise: a statistical tie for first. But its score is unusually high for Laya Router: 92.5%, 13 points above the 80% its other datasets predict.
| Rank | Model | Accuracy | 95% interval | Macro-F1 | Calibration error | Failed | |
|---|---|---|---|---|---|---|---|
| 1 | Laya Router Convai InnovationsLeaderunusually high | 92.5 | 88.8–96.2 | 92.3 | 2.8% | 0 | |
| 2 | Mercury Decide InceptionTied for 1st | 92.0 | 88.2–95.8 | 91.9 | 5.6% | 0 | |
| 2 | Mammoth 1 BasetenTied for 1st | 92.0 | 88.2–95.8 | 91.8 | 5.8% | 0 | |
| 2 | GLiDE FastinoTied for 1st | 92.0 | 88.2–95.8 | 91.9 | 6.2% | 0 | |
| 5 | 91.5 | 87.6–95.4 | 91.3 | 4.0% | 0 | ||
| 5 | 91.5 | 87.6–95.4 | 91.3 | 4.9% | 0 | ||
| 5 | Dexy-Decision-9B Patrick-18Tied for 1st | 91.5 | 87.6–95.4 | 91.4 | 5.0% | 0 | |
| 8 | Kev 0.8B Jared PalmerTied for 1stin-domain | 91.0 | 87.0–95.0 | 90.8 | 4.0% | 0 | |
| 8 | d1-ultra Liquid AITied for 1st | 91.0 | 87.0–95.0 | 90.8 | 3.2% | 0 | |
| 10 | 90.5 | 86.4–94.6 | 90.3 | 5.7% | 0 | ||
| 10 | Kev 4B Jared PalmerTied for 1stin-domain | 90.5 | 86.4–94.6 | 90.3 | 3.4% | 0 | |
| 10 | 90.5 | 86.4–94.6 | 90.3 | 3.9% | 0 | ||
| 10 | d1.1 Liquid AITied for 1st | 90.5 | 86.4–94.6 | 90.3 | 5.6% | 0 | |
| 10 | Celeris-1 Decision CelerisTied for 1st | 90.5 | 86.4–94.6 | 90.3 | 3.5% | 0 | |
| 15 | d1 Liquid AITied for 1st | 90.0 | 85.8–94.2 | 89.7 | 6.1% | 0 | |
| 15 | 90.0 | 85.8–94.2 | 89.8 | 4.0% | 0 | ||
| 15 | 90.0 | 85.8–94.2 | 89.7 | 7.0% | 0 | ||
| 18 | Jev 1.14 TypeSafeTied for 1st | 89.0 | 84.7–93.3 | 88.7 | 6.9% | 0 | |
| 18 | 89.0 | 84.7–93.3 | 88.7 | 6.7% | 0 | ||
| 18 | 89.0 | 84.7–93.3 | 88.7 | 3.4% | 0 | ||
| 18 | 89.0 | 84.7–93.3 | 88.7 | 3.1% | 0 | ||
| 18 | Elephant 1 BasetenTied for 1st | 89.0 | 84.7–93.3 | 88.6 | 5.1% | 0 | |
| 23 | Jev 1.13 TypeSafeTied for 1st | 88.5 | 84.1–92.9 | 88.1 | 8.6% | 0 | |
| 23 | d1-3B Liquid AITied for 1st | 88.5 | 84.1–92.9 | 88.2 | 7.1% | 0 | |
| 23 | 88.5 | 84.1–92.9 | 88.1 | 5.3% | 0 | ||
| 26 | 88.0 | 83.5–92.5 | 87.8 | 10.1% | 0 | ||
| 26 | 88.0 | 83.5–92.5 | 87.6 | 4.3% | 0 | ||
| 28 | 87.5 | 82.9–92.1 | 87.0 | 11.2% | 0 | ||
| 28 | d1-omni-600M Liquid AITied for 1st | 87.5 | 82.9–92.1 | 87.1 | 6.5% | 0 | |
| 28 | 87.5 | 82.9–92.1 | 87.1 | 3.4% | 0 | ||
| 31 | Span-02 Respan | 86.5 | 81.8–91.2 | 86.0 | 9.9% | 0 | |
| 32 | 86.0 | 81.2–90.8 | 85.5 | 12.6% | 0 | ||
| 32 | 86.0 | 81.2–90.8 | 85.5 | 10.2% | 0 | ||
| 34 | 85.5 | 80.6–90.4 | 85.1 | 10.1% | 0 | ||
| 35 | Joey 1 Baseten | 85.0 | 80.1–89.9 | 84.5 | 4.9% | 0 | |
| 36 | 80.0 | 74.5–85.5 | 79.0 | 9.9% | 0 | ||
| 37 | Naluri-308M Jericho Siahaya | 79.5 | 73.9–85.1 | 79.1 | 8.3% | 0 | |
| 38 | Naluri-279M Jericho Siahaya | 77.5 | 71.7–83.3 | 74.2 | 9.9% | 0 | |
| 39 | Classify-1 ScaleDown | 26.5 | 20.4–32.6 | 10.5 | 1.5% | 200 |
The bar is accuracy; the bracket on it is the 95% interval for 200 cases. Tied for 1st marks every model whose gap to the leader is within sampling noise (1.96 standard errors of the difference, treating the runs as independent). A failed call counts as wrong. In-domain models trained on another split of this dataset, by their maker's own account. Unusually high marks a score far above what that model's other datasets, and every other model's score here, predict (hover for the numbers). It often means the model trained on this dataset or one like it; it isn't proof, and the score still counts.
It counts towards the System One Index and its English sub-index. Every model gets the same cases and the same request, under the benchmark rules.