~/system-one-bench / benchmarks/arabic-miracl-rerank
Text track · Arabic · All benchmarks
Pick the relevant passage out of 8. d1 has the top score, 0.5 points ahead of Microsoft-Decision-1, but Microsoft-Decision-1 and 22 more are within sampling noise: a statistical tie for first.
| Rank | Model | Accuracy | 95% interval | Macro-F1 | Calibration error | Failed | |
|---|---|---|---|---|---|---|---|
| 1 | d1 Liquid AILeader | 90.5 | 86.4–94.6 | 90.2 | 3.0% | 0 | |
| 2 | 90.0 | 85.8–94.2 | 89.7 | 4.7% | 0 | ||
| 3 | 89.5 | 85.3–93.7 | 89.5 | 4.5% | 0 | ||
| 3 | 89.5 | 85.3–93.7 | 89.3 | 4.9% | 0 | ||
| 5 | 89.0 | 84.7–93.3 | 88.9 | 13.4% | 0 | ||
| 5 | 89.0 | 84.7–93.3 | 88.9 | 5.3% | 0 | ||
| 5 | d1-ultra Liquid AITied for 1st | 89.0 | 84.7–93.3 | 88.8 | 4.6% | 0 | |
| 5 | Mammoth 1 BasetenTied for 1st | 89.0 | 84.7–93.3 | 88.8 | 6.2% | 0 | |
| 9 | 88.5 | 84.1–92.9 | 88.1 | 4.4% | 0 | ||
| 9 | Span-02 RespanTied for 1st | 88.5 | 84.1–92.9 | 88.3 | 7.9% | 0 | |
| 9 | d1.1 Liquid AITied for 1st | 88.5 | 84.1–92.9 | 88.1 | 2.2% | 0 | |
| 12 | 88.0 | 83.5–92.5 | 88.2 | 12.1% | 0 | ||
| 12 | Mercury Decide InceptionTied for 1st | 88.0 | 83.5–92.5 | 88.0 | 8.2% | 0 | |
| 12 | Elephant 1 BasetenTied for 1st | 88.0 | 83.5–92.5 | 88.0 | 5.0% | 0 | |
| 12 | Dexy-Decision-9B Patrick-18Tied for 1st | 88.0 | 83.5–92.5 | 87.6 | 5.4% | 0 | |
| 16 | Jev 1.14 TypeSafeTied for 1st | 87.5 | 82.9–92.1 | 87.1 | 6.6% | 0 | |
| 17 | 87.0 | 82.3–91.7 | 86.6 | 4.5% | 0 | ||
| 17 | GLiDE FastinoTied for 1st | 87.0 | 82.3–91.7 | 86.8 | 5.4% | 0 | |
| 19 | Jev 1.13 TypeSafeTied for 1st | 86.5 | 81.8–91.2 | 85.8 | 6.1% | 0 | |
| 19 | 86.5 | 81.8–91.2 | 86.4 | 4.7% | 0 | ||
| 19 | Celeris-1 Decision CelerisTied for 1st | 86.5 | 81.8–91.2 | 86.2 | 3.0% | 0 | |
| 22 | 86.0 | 81.2–90.8 | 85.6 | 4.5% | 1 | ||
| 23 | 85.5 | 80.6–90.4 | 85.3 | 8.7% | 0 | ||
| 24 | 84.5 | 79.5–89.5 | 84.2 | 11.0% | 0 | ||
| 25 | 83.0 | 77.8–88.2 | 82.6 | 17.7% | 0 | ||
| 26 | 82.5 | 77.2–87.8 | 82.3 | 4.3% | 0 | ||
| 26 | d1-3B Liquid AI | 82.5 | 77.2–87.8 | 82.1 | 6.5% | 0 | |
| 28 | 81.0 | 75.6–86.4 | 80.9 | 13.5% | 0 | ||
| 29 | Kev 4B Jared Palmer | 79.5 | 73.9–85.1 | 79.6 | 13.0% | 0 | |
| 30 | 77.5 | 71.7–83.3 | 77.3 | 8.4% | 0 | ||
| 31 | laya-ara-rag Wouzein-domain | 71.5 | 65.2–77.8 | 71.4 | 11.3% | 0 | |
| 31 | 71.5 | 65.2–77.8 | 71.7 | 5.7% | 0 | ||
| 33 | 70.0 | 63.6–76.4 | 69.2 | 5.6% | 0 | ||
| 34 | Kev 0.8B Jared Palmer | 62.5 | 55.8–69.2 | 62.1 | 24.8% | 0 | |
| 35 | Joey 1 Baseten | 62.0 | 55.3–68.7 | 62.1 | 7.3% | 0 | |
| 36 | d1-omni-600M Liquid AI | 55.0 | 48.1–61.9 | 55.0 | 10.7% | 0 | |
| 37 | Naluri-308M Jericho Siahaya | 34.0 | 27.4–40.6 | 32.5 | 8.4% | 0 | |
| 38 | laya-ara Wouze | 32.0 | 25.5–38.5 | 32.3 | 26.4% | 0 | |
| 39 | Laya Router Convai Innovations | 31.5 | 25.1–37.9 | 30.9 | 18.9% | 0 | |
| 40 | Naluri-279M Jericho Siahaya | 28.0 | 21.8–34.2 | 24.4 | 10.4% | 0 | |
| 41 | Classify-1 ScaleDown | 12.0 | 7.5–16.5 | 2.7 | 0.5% | 200 |
The bar is accuracy; the bracket on it is the 95% interval for 200 cases. Tied for 1st marks every model whose gap to the leader is within sampling noise (1.96 standard errors of the difference, treating the runs as independent). A failed call counts as wrong. In-domain models trained on another split of this dataset, by their maker's own account.
It counts towards the System One Index and its Arabic sub-index. Every model gets the same cases and the same request, under the benchmark rules.