~/system-one-bench / benchmarks/arabic-miracl-pairwise
Text track · Arabic · All benchmarks
Is this passage relevant, yes/no. Decider V1.2 and Span-02 share the top score, and 27 more are within sampling noise.
| Rank | Model | Accuracy | 95% interval | Macro-F1 | Calibration error | Failed | |
|---|---|---|---|---|---|---|---|
| 1 | 91.5 | 87.6–95.4 | 91.5 | 5.5% | 0 | ||
| 1 | Span-02 RespanTied for 1st | 91.5 | 87.6–95.4 | 91.5 | 4.7% | 0 | |
| 3 | d1 Liquid AITied for 1st | 91.0 | 87.0–95.0 | 91.0 | 3.5% | 0 | |
| 3 | 91.0 | 87.0–95.0 | 91.0 | 1.6% | 0 | ||
| 3 | d1-ultra Liquid AITied for 1st | 91.0 | 87.0–95.0 | 91.0 | 2.1% | 0 | |
| 3 | Elephant 1 BasetenTied for 1st | 91.0 | 87.0–95.0 | 91.0 | 1.6% | 0 | |
| 3 | Dexy-Decision-9B Patrick-18Tied for 1st | 91.0 | 87.0–95.0 | 91.0 | 4.8% | 0 | |
| 3 | 91.0 | 87.0–95.0 | 91.0 | 3.2% | 0 | ||
| 9 | Jev 1.13 TypeSafeTied for 1st | 90.5 | 86.4–94.6 | 90.5 | 2.7% | 0 | |
| 9 | 90.5 | 86.4–94.6 | 90.5 | 3.1% | 0 | ||
| 9 | GLiDE FastinoTied for 1st | 90.5 | 86.4–94.6 | 90.5 | 2.4% | 0 | |
| 12 | Jev 1.14 TypeSafeTied for 1st | 90.0 | 85.8–94.2 | 90.0 | 2.1% | 0 | |
| 12 | 90.0 | 85.8–94.2 | 90.0 | 2.7% | 0 | ||
| 12 | Mercury Decide InceptionTied for 1st | 90.0 | 85.8–94.2 | 90.0 | 7.6% | 0 | |
| 12 | Mammoth 1 BasetenTied for 1st | 90.0 | 85.8–94.2 | 90.0 | 4.2% | 0 | |
| 12 | Celeris-1 Decision CelerisTied for 1st | 90.0 | 85.8–94.2 | 90.0 | 4.0% | 0 | |
| 17 | 89.5 | 85.3–93.7 | 89.5 | 3.3% | 0 | ||
| 17 | d1.1 Liquid AITied for 1st | 89.5 | 85.3–93.7 | 89.5 | 3.8% | 0 | |
| 19 | 89.0 | 84.7–93.3 | 89.0 | 6.4% | 0 | ||
| 19 | d1-3B Liquid AITied for 1st | 89.0 | 84.7–93.3 | 89.0 | 5.2% | 0 | |
| 19 | 89.0 | 84.7–93.3 | 89.0 | 2.9% | 0 | ||
| 22 | 87.5 | 82.9–92.1 | 87.5 | 10.7% | 0 | ||
| 22 | 87.5 | 82.9–92.1 | 87.5 | 3.6% | 0 | ||
| 24 | laya-ara-rag WouzeTied for 1stin-domain | 86.5 | 81.8–91.2 | 86.4 | 3.7% | 0 | |
| 24 | Kev 4B Jared PalmerTied for 1st | 86.5 | 81.8–91.2 | 86.5 | 4.3% | 0 | |
| 24 | 86.5 | 81.8–91.2 | 86.5 | 10.0% | 0 | ||
| 24 | 86.5 | 81.8–91.2 | 86.5 | 4.4% | 0 | ||
| 28 | 86.0 | 81.2–90.8 | 85.9 | 9.1% | 0 | ||
| 29 | 85.5 | 80.6–90.4 | 85.5 | 10.6% | 0 | ||
| 30 | 84.5 | 79.5–89.5 | 84.5 | 5.7% | 0 | ||
| 31 | 83.0 | 77.8–88.2 | 83.0 | 7.2% | 0 | ||
| 31 | Joey 1 Baseten | 83.0 | 77.8–88.2 | 83.0 | 10.3% | 0 | |
| 33 | Laya Router Convai Innovations | 82.0 | 76.7–87.3 | 82.0 | 5.0% | 0 | |
| 34 | 78.0 | 72.3–83.7 | 77.9 | 5.9% | 0 | ||
| 35 | 76.0 | 70.1–81.9 | 75.6 | 5.0% | 0 | ||
| 35 | Naluri-308M Jericho Siahaya | 76.0 | 70.1–81.9 | 75.5 | 2.4% | 0 | |
| 37 | laya-ara Wouze | 74.5 | 68.5–80.5 | 74.0 | 9.2% | 0 | |
| 38 | Kev 0.8B Jared Palmer | 67.0 | 60.5–73.5 | 66.9 | 4.4% | 0 | |
| 39 | d1-omni-600M Liquid AI | 66.5 | 60.0–73.0 | 65.2 | 7.1% | 0 | |
| 40 | Naluri-279M Jericho Siahaya | 50.5 | 43.6–57.4 | 35.3 | 42.8% | 0 | |
| 41 | Classify-1 ScaleDown | 50.0 | 43.1–56.9 | 33.3 | 0.0% | 200 |
The bar is accuracy; the bracket on it is the 95% interval for 200 cases. Tied for 1st marks every model whose gap to the leader is within sampling noise (1.96 standard errors of the difference, treating the runs as independent). A failed call counts as wrong. In-domain models trained on another split of this dataset, by their maker's own account.
It counts towards the System One Index and its Arabic sub-index. Every model gets the same cases and the same request, under the benchmark rules.