~/system-one-bench / benchmarks/arabic-tweet-sentiment
Text track · Arabic · All benchmarks
Sentiment, 3 options. GPT-6 Luna Decisions has the top score, 2.0 points ahead of Gemini Decision 1, but Gemini Decision 1 and 7 more are within sampling noise: a statistical tie for first.
| Rank | Model | Accuracy | 95% interval | Macro-F1 | Calibration error | Failed | |
|---|---|---|---|---|---|---|---|
| 1 | 80.0 | 74.5–85.5 | 79.7 | 10.7% | 0 | ||
| 2 | 78.0 | 72.3–83.7 | 77.6 | 4.7% | 0 | ||
| 3 | Jev 1.13 TypeSafeTied for 1st | 76.0 | 70.1–81.9 | 75.6 | 9.7% | 0 | |
| 3 | Jev 1.14 TypeSafeTied for 1st | 76.0 | 70.1–81.9 | 75.6 | 9.5% | 0 | |
| 3 | d1-ultra Liquid AITied for 1st | 76.0 | 70.1–81.9 | 76.4 | 8.1% | 0 | |
| 3 | Span-02 RespanTied for 1st | 76.0 | 70.1–81.9 | 76.2 | 10.1% | 0 | |
| 7 | 75.5 | 69.5–81.5 | 75.3 | 3.7% | 0 | ||
| 8 | GLiDE FastinoTied for 1st | 73.5 | 67.4–79.6 | 73.7 | 5.7% | 0 | |
| 9 | d1.1 Liquid AITied for 1st | 72.5 | 66.3–78.7 | 72.9 | 4.4% | 0 | |
| 10 | 71.5 | 65.2–77.8 | 71.2 | 24.8% | 0 | ||
| 10 | 71.5 | 65.2–77.8 | 71.8 | 4.4% | 0 | ||
| 12 | 71.0 | 64.7–77.3 | 71.4 | 6.1% | 0 | ||
| 13 | 70.5 | 64.2–76.8 | 69.9 | 6.8% | 0 | ||
| 14 | 69.5 | 63.1–75.9 | 69.9 | 5.7% | 0 | ||
| 15 | 69.0 | 62.6–75.4 | 69.5 | 4.5% | 0 | ||
| 15 | Celeris-1 Decision Celeris | 69.0 | 62.6–75.4 | 69.7 | 6.3% | 0 | |
| 17 | 68.5 | 62.1–74.9 | 69.0 | 14.8% | 0 | ||
| 18 | d1 Liquid AI | 67.5 | 61.0–74.0 | 67.9 | 12.5% | 0 | |
| 19 | 67.0 | 60.5–73.5 | 67.4 | 4.9% | 0 | ||
| 19 | Dexy-Decision-9B Patrick-18 | 67.0 | 60.5–73.5 | 67.2 | 13.1% | 0 | |
| 21 | Elephant 1 Baseten | 66.0 | 59.4–72.6 | 66.3 | 13.7% | 0 | |
| 21 | Joey 1 Baseten | 66.0 | 59.4–72.6 | 65.7 | 7.3% | 0 | |
| 23 | 65.5 | 58.9–72.1 | 65.1 | 21.8% | 0 | ||
| 24 | Mammoth 1 Baseten | 64.5 | 57.9–71.1 | 64.8 | 15.1% | 0 | |
| 25 | 64.0 | 57.3–70.7 | 64.1 | 9.6% | 0 | ||
| 26 | 63.5 | 56.8–70.2 | 63.2 | 17.8% | 0 | ||
| 26 | d1-3B Liquid AI | 63.5 | 56.8–70.2 | 63.2 | 11.2% | 0 | |
| 28 | Kev 4B Jared Palmer | 63.0 | 56.3–69.7 | 63.0 | 12.9% | 0 | |
| 29 | Mercury Decide Inception | 62.5 | 55.8–69.2 | 62.8 | 22.0% | 0 | |
| 30 | 61.5 | 54.8–68.2 | 61.5 | 4.3% | 0 | ||
| 31 | 58.0 | 51.2–64.8 | 57.8 | 20.6% | 0 | ||
| 31 | Naluri-308M Jericho Siahaya | 58.0 | 51.2–64.8 | 58.3 | 6.1% | 0 | |
| 33 | Kev 0.8B Jared Palmer | 57.0 | 50.1–63.9 | 56.7 | 10.8% | 0 | |
| 34 | Naluri-279M Jericho Siahayaunusually high | 53.0 | 46.1–59.9 | 53.2 | 6.2% | 0 | |
| 35 | 50.0 | 43.1–56.9 | 47.1 | 34.8% | 0 | ||
| 36 | laya-ara Wouze | 49.0 | 42.1–55.9 | 49.7 | 25.4% | 0 | |
| 37 | 47.0 | 40.1–53.9 | 43.7 | 35.7% | 0 | ||
| 38 | laya-ara-rag Wouze | 44.0 | 37.1–50.9 | 40.0 | 17.0% | 0 | |
| 39 | Laya Router Convai Innovations | 37.0 | 30.3–43.7 | 32.5 | 45.6% | 0 | |
| 40 | Classify-1 ScaleDown | 35.0 | 28.4–41.6 | 17.3 | 1.7% | 200 | |
| 41 | d1-omni-600M Liquid AI | 32.5 | 26.0–39.0 | 22.3 | 44.1% | 0 |
The bar is accuracy; the bracket on it is the 95% interval for 200 cases. Tied for 1st marks every model whose gap to the leader is within sampling noise (1.96 standard errors of the difference, treating the runs as independent). A failed call counts as wrong. Unusually high marks a score far above what that model's other datasets, and every other model's score here, predict (hover for the numbers). It often means the model trained on this dataset or one like it; it isn't proof, and the score still counts.
It counts towards the System One Index and its Arabic sub-index. Every model gets the same cases and the same request, under the benchmark rules.