~/system-one-bench / benchmarks/arabic-massive-intent
Text track · Arabic · All benchmarks
Assistant request intent, 20 options. Clef Omni and Dexy-Decision-9B share the top score, and 15 more are within sampling noise.
| Rank | Model | Accuracy | 95% interval | Macro-F1 | Calibration error | Failed | |
|---|---|---|---|---|---|---|---|
| 1 | 91.5 | 87.6–95.4 | 91.7 | 7.2% | 0 | ||
| 1 | Dexy-Decision-9B Patrick-18Tied for 1stin-domain | 91.5 | 87.6–95.4 | 91.4 | 5.2% | 0 | |
| 3 | 91.0 | 87.0–95.0 | 91.1 | 4.9% | 0 | ||
| 3 | Mammoth 1 BasetenTied for 1st | 91.0 | 87.0–95.0 | 90.7 | 3.6% | 0 | |
| 3 | GLiDE FastinoTied for 1st | 91.0 | 87.0–95.0 | 91.1 | 2.9% | 0 | |
| 6 | 90.5 | 86.4–94.6 | 90.8 | 4.4% | 0 | ||
| 7 | d1 Liquid AITied for 1st | 90.0 | 85.8–94.2 | 90.6 | 4.1% | 0 | |
| 8 | 89.5 | 85.3–93.7 | 89.8 | 5.8% | 0 | ||
| 8 | Elephant 1 BasetenTied for 1st | 89.5 | 85.3–93.7 | 89.6 | 3.9% | 0 | |
| 10 | 89.0 | 84.7–93.3 | 89.7 | 3.5% | 0 | ||
| 11 | 88.0 | 83.5–92.5 | 88.4 | 5.1% | 0 | ||
| 11 | d1.1 Liquid AITied for 1st | 88.0 | 83.5–92.5 | 88.4 | 4.9% | 0 | |
| 11 | Celeris-1 Decision CelerisTied for 1st | 88.0 | 83.5–92.5 | 88.1 | 6.7% | 0 | |
| 14 | d1-ultra Liquid AITied for 1st | 87.0 | 82.3–91.7 | 87.0 | 6.2% | 0 | |
| 14 | 87.0 | 82.3–91.7 | 87.2 | 4.2% | 0 | ||
| 16 | 86.0 | 81.2–90.8 | 85.8 | 4.1% | 0 | ||
| 16 | Span-02 RespanTied for 1st | 86.0 | 81.2–90.8 | 86.7 | 2.7% | 0 | |
| 18 | Jev 1.14 TypeSafe | 85.0 | 80.1–89.9 | 85.4 | 2.8% | 0 | |
| 18 | Mercury Decide Inception | 85.0 | 80.1–89.9 | 85.5 | 7.8% | 0 | |
| 20 | 84.5 | 79.5–89.5 | 84.4 | 5.1% | 1 | ||
| 20 | Joey 1 Baseten | 84.5 | 79.5–89.5 | 84.9 | 3.4% | 0 | |
| 22 | Jev 1.13 TypeSafe | 84.0 | 78.9–89.1 | 83.6 | 3.5% | 0 | |
| 23 | d1-3B Liquid AI | 83.5 | 78.4–88.6 | 84.4 | 6.1% | 0 | |
| 24 | 82.5 | 77.2–87.8 | 82.5 | 8.1% | 0 | ||
| 25 | 81.0 | 75.6–86.4 | 81.2 | 8.0% | 0 | ||
| 26 | laya-ara Wouzein-domain | 77.5 | 71.7–83.3 | 77.3 | 16.1% | 0 | |
| 26 | 77.5 | 71.7–83.3 | 78.1 | 11.5% | 0 | ||
| 26 | 77.5 | 71.7–83.3 | 77.0 | 3.4% | 0 | ||
| 29 | Kev 4B Jared Palmer | 76.5 | 70.6–82.4 | 76.4 | 22.5% | 0 | |
| 30 | 75.0 | 69.0–81.0 | 74.7 | 18.9% | 0 | ||
| 31 | 70.0 | 63.6–76.4 | 68.6 | 9.6% | 0 | ||
| 32 | Naluri-308M Jericho Siahayain-domain | 65.5 | 58.9–72.1 | 64.5 | 32.2% | 0 | |
| 33 | 63.0 | 56.3–69.7 | 63.2 | 18.3% | 0 | ||
| 34 | laya-ara-rag Wouzein-domain | 55.0 | 48.1–61.9 | 54.2 | 23.5% | 0 | |
| 34 | 55.0 | 48.1–61.9 | 54.9 | 10.6% | 0 | ||
| 36 | d1-omni-600M Liquid AI | 52.0 | 45.1–58.9 | 51.7 | 25.1% | 0 | |
| 36 | Naluri-279M Jericho Siahayain-domain | 52.0 | 45.1–58.9 | 52.1 | 12.6% | 0 | |
| 38 | Kev 0.8B Jared Palmer | 51.5 | 44.6–58.4 | 50.4 | 8.4% | 0 | |
| 39 | Laya Router Convai Innovations | 32.5 | 26.0–39.0 | 32.2 | 42.4% | 0 | |
| 40 | Classify-1 ScaleDown | 3.5 | 1.0–6.0 | 0.3 | 1.5% | 200 | |
| 40 | 3.5 | 1.0–6.0 | 0.3 | 1.5% | 200 |
The bar is accuracy; the bracket on it is the 95% interval for 200 cases. Tied for 1st marks every model whose gap to the leader is within sampling noise (1.96 standard errors of the difference, treating the runs as independent). A failed call counts as wrong. In-domain models trained on another split of this dataset, by their maker's own account.
It counts towards the System One Index and its Arabic sub-index. Every model gets the same cases and the same request, under the benchmark rules.