~/system-one-bench / benchmarks/xnli-english
Text track · English · All benchmarks
Does one sentence follow from another, 3 options. Drex Max has the top score, 0.7 points ahead of GLiDE, but GLiDE and 12 more are within sampling noise: a statistical tie for first.
| Rank | Model | Accuracy | 95% interval | Macro-F1 | Calibration error | Failed | |
|---|---|---|---|---|---|---|---|
| 1 | 95.3 | 92.0–98.7 | 95.1 | 3.1% | 0 | ||
| 2 | GLiDE FastinoTied for 1st | 94.7 | 91.1–98.3 | 94.4 | 4.2% | 0 | |
| 2 | Dexy-Decision-9B Patrick-18Tied for 1st | 94.7 | 91.1–98.3 | 94.4 | 3.0% | 0 | |
| 4 | 94.0 | 90.2–97.8 | 93.6 | 4.2% | 0 | ||
| 4 | Elephant 1 BasetenTied for 1st | 94.0 | 90.2–97.8 | 93.7 | 4.0% | 0 | |
| 6 | 92.7 | 88.5–96.8 | 92.3 | 3.2% | 0 | ||
| 6 | d1-ultra Liquid AITied for 1st | 92.7 | 88.5–96.8 | 92.6 | 11.0% | 0 | |
| 8 | d1 Liquid AITied for 1st | 91.3 | 86.8–95.8 | 90.7 | 2.5% | 0 | |
| 8 | Mercury Decide InceptionTied for 1st | 91.3 | 86.8–95.8 | 91.2 | 4.6% | 0 | |
| 8 | Mammoth 1 BasetenTied for 1st | 91.3 | 86.8–95.8 | 91.2 | 3.1% | 0 | |
| 8 | Celeris-1 Decision CelerisTied for 1st | 91.3 | 86.8–95.8 | 90.6 | 9.2% | 0 | |
| 12 | Kev 4B Jared PalmerTied for 1st | 90.7 | 86.0–95.3 | 90.3 | 6.7% | 0 | |
| 12 | 90.7 | 86.0–95.3 | 90.3 | 6.0% | 0 | ||
| 14 | d1.1 Liquid AITied for 1st | 90.0 | 85.2–94.8 | 89.7 | 5.9% | 0 | |
| 15 | 89.3 | 84.4–94.3 | 89.0 | 2.6% | 0 | ||
| 16 | 88.7 | 83.6–93.7 | 88.7 | 6.2% | 0 | ||
| 16 | 88.7 | 83.6–93.7 | 88.6 | 3.1% | 0 | ||
| 18 | Laya Router Convai Innovations | 87.3 | 82.0–92.7 | 87.1 | 4.3% | 0 | |
| 18 | 87.3 | 82.0–92.7 | 86.5 | 4.7% | 0 | ||
| 18 | 87.3 | 82.0–92.7 | 86.6 | 5.4% | 0 | ||
| 18 | 87.3 | 82.0–92.7 | 87.1 | 6.3% | 0 | ||
| 18 | 87.3 | 82.0–92.7 | 87.1 | 3.4% | 0 | ||
| 23 | Jev 1.13 TypeSafe | 86.0 | 80.4–91.6 | 85.7 | 8.2% | 0 | |
| 23 | Jev 1.14 TypeSafe | 86.0 | 80.4–91.6 | 85.7 | 7.0% | 0 | |
| 23 | d1-3B Liquid AI | 86.0 | 80.4–91.6 | 85.5 | 3.1% | 0 | |
| 26 | 84.7 | 78.9–90.4 | 84.6 | 5.8% | 0 | ||
| 27 | 84.0 | 78.1–89.9 | 83.6 | 3.6% | 0 | ||
| 27 | Joey 1 Baseten | 84.0 | 78.1–89.9 | 83.7 | 5.7% | 0 | |
| 29 | Span-02 Respan | 83.3 | 77.4–89.3 | 83.5 | 4.4% | 0 | |
| 30 | 82.7 | 76.6–88.7 | 82.0 | 10.8% | 0 | ||
| 31 | d1-omni-600M Liquid AI | 82.0 | 75.9–88.1 | 81.7 | 11.6% | 0 | |
| 32 | Kev 0.8B Jared Palmer | 81.3 | 75.1–87.6 | 80.7 | 3.8% | 0 | |
| 33 | 80.7 | 74.4–87.0 | 79.8 | 3.6% | 0 | ||
| 34 | Naluri-308M Jericho Siahayain-domain | 76.0 | 69.2–82.8 | 75.2 | 4.7% | 0 | |
| 35 | 74.7 | 67.7–81.6 | 75.1 | 20.8% | 0 | ||
| 36 | 66.7 | 59.1–74.2 | 67.5 | 20.7% | 0 | ||
| 37 | 65.3 | 57.7–72.9 | 65.9 | 21.0% | 0 | ||
| 38 | Naluri-279M Jericho Siahayain-domain | 30.7 | 23.3–38.0 | 16.1 | 40.6% | 0 | |
| 38 | Classify-1 ScaleDown | 30.7 | 23.3–38.0 | 15.7 | 2.7% | 150 |
The bar is accuracy; the bracket on it is the 95% interval for 150 cases. Tied for 1st marks every model whose gap to the leader is within sampling noise (1.96 standard errors of the difference, treating the runs as independent). A failed call counts as wrong. In-domain models trained on another split of this dataset, by their maker's own account.
It counts towards the System One Index and its English sub-index. Every model gets the same cases and the same request, under the benchmark rules.