~/system-one-bench / benchmarks/xnli-arabic
Text track · Arabic · All benchmarks
Does one sentence follow from another, 3 options. Decider V1.2 has the top score, 0.7 points ahead of Drex Max, but Drex Max and 3 more are within sampling noise: a statistical tie for first.
| Rank | Model | Accuracy | 95% interval | Macro-F1 | Calibration error | Failed | |
|---|---|---|---|---|---|---|---|
| 1 | 90.0 | 85.2–94.8 | 89.6 | 7.7% | 0 | ||
| 2 | 89.3 | 84.4–94.3 | 89.0 | 3.8% | 0 | ||
| 2 | Dexy-Decision-9B Patrick-18Tied for 1st | 89.3 | 84.4–94.3 | 88.6 | 4.4% | 0 | |
| 4 | 85.3 | 79.7–91.0 | 84.9 | 5.6% | 0 | ||
| 5 | GLiDE FastinoTied for 1st | 83.3 | 77.4–89.3 | 83.2 | 2.9% | 0 | |
| 6 | 82.0 | 75.9–88.1 | 81.7 | 6.9% | 0 | ||
| 7 | d1 Liquid AI | 80.0 | 73.6–86.4 | 79.5 | 6.1% | 0 | |
| 7 | Celeris-1 Decision Celeris | 80.0 | 73.6–86.4 | 79.6 | 5.5% | 0 | |
| 9 | d1.1 Liquid AI | 79.3 | 72.8–85.8 | 79.5 | 8.2% | 0 | |
| 9 | Elephant 1 Baseten | 79.3 | 72.8–85.8 | 79.3 | 7.4% | 0 | |
| 11 | Mammoth 1 Baseten | 78.7 | 72.1–85.2 | 78.9 | 4.9% | 0 | |
| 12 | 78.0 | 71.4–84.6 | 78.3 | 7.5% | 0 | ||
| 12 | 78.0 | 71.4–84.6 | 77.9 | 11.7% | 0 | ||
| 14 | Mercury Decide Inception | 76.0 | 69.2–82.8 | 75.4 | 13.8% | 0 | |
| 14 | 76.0 | 69.2–82.8 | 74.6 | 8.5% | 0 | ||
| 14 | d1-ultra Liquid AI | 76.0 | 69.2–82.8 | 75.9 | 9.8% | 0 | |
| 17 | laya-ara Wouzein-domain | 75.3 | 68.4–82.2 | 74.8 | 18.6% | 0 | |
| 18 | 74.0 | 67.0–81.0 | 74.1 | 14.0% | 0 | ||
| 19 | 73.3 | 66.3–80.4 | 72.3 | 8.2% | 0 | ||
| 19 | d1-3B Liquid AI | 73.3 | 66.3–80.4 | 72.7 | 7.5% | 0 | |
| 21 | Kev 4B Jared Palmer | 72.7 | 65.5–79.8 | 71.9 | 20.1% | 0 | |
| 21 | 72.7 | 65.5–79.8 | 72.0 | 6.3% | 0 | ||
| 23 | 72.0 | 64.8–79.2 | 72.5 | 12.5% | 0 | ||
| 24 | Jev 1.14 TypeSafe | 71.3 | 64.1–78.6 | 71.5 | 11.4% | 0 | |
| 24 | 71.3 | 64.1–78.6 | 70.5 | 6.2% | 0 | ||
| 26 | laya-ara-rag Wouze | 70.0 | 62.7–77.3 | 69.6 | 8.7% | 0 | |
| 27 | Jev 1.13 TypeSafe | 69.3 | 62.0–76.7 | 69.8 | 14.0% | 0 | |
| 28 | 68.7 | 61.2–76.1 | 67.7 | 7.8% | 0 | ||
| 29 | Laya Router Convai Innovations | 68.0 | 60.5–75.5 | 66.9 | 23.4% | 0 | |
| 30 | 67.3 | 59.8–74.8 | 65.6 | 2.4% | 0 | ||
| 31 | Span-02 Respan | 66.0 | 58.4–73.6 | 66.1 | 17.9% | 0 | |
| 32 | Joey 1 Baseten | 64.7 | 57.0–72.3 | 65.2 | 12.1% | 0 | |
| 33 | 59.3 | 51.5–67.2 | 58.9 | 12.2% | 0 | ||
| 34 | Kev 0.8B Jared Palmer | 58.7 | 50.8–66.6 | 55.4 | 18.6% | 0 | |
| 35 | 58.0 | 50.1–65.9 | 57.8 | 36.8% | 0 | ||
| 36 | Naluri-308M Jericho Siahayain-domain | 56.7 | 48.7–64.6 | 56.6 | 14.3% | 0 | |
| 37 | 54.0 | 46.0–62.0 | 51.9 | 29.8% | 0 | ||
| 38 | d1-omni-600M Liquid AI | 52.7 | 44.7–60.7 | 52.2 | 13.9% | 0 | |
| 39 | 52.0 | 44.0–60.0 | 49.8 | 27.9% | 0 | ||
| 40 | Naluri-279M Jericho Siahayain-domain | 30.7 | 23.3–38.0 | 15.9 | 40.9% | 0 | |
| 40 | Classify-1 ScaleDown | 30.7 | 23.3–38.0 | 15.7 | 2.7% | 150 |
The bar is accuracy; the bracket on it is the 95% interval for 150 cases. Tied for 1st marks every model whose gap to the leader is within sampling noise (1.96 standard errors of the difference, treating the runs as independent). A failed call counts as wrong. In-domain models trained on another split of this dataset, by their maker's own account.
It counts towards the System One Index and its Arabic sub-index. Every model gets the same cases and the same request, under the benchmark rules.