~/system-one-bench / benchmarks/arabic-sentiment-as-an-image
Vision track · All benchmarks
The 200 Arabic tweets rendered as a picture, 3 options. Gemini Decision 1 has the top score, 5.0 points ahead of Span-02, but Span-02 and 1 more are within sampling noise: a statistical tie for first.
| Rank | Model | Accuracy | 95% interval | Macro-F1 | Calibration error | Failed | |
|---|---|---|---|---|---|---|---|
| 1 | 75.0 | 69.0–81.0 | 74.6 | 6.6% | 0 | ||
| 2 | Span-02 RespanTied for 1st | 70.0 | 63.6–76.4 | 70.7 | 11.5% | 0 | |
| 3 | 67.0 | 60.5–73.5 | 66.6 | 13.5% | 0 | ||
| 4 | Jev 1.14 TypeSafe | 63.5 | 56.8–70.2 | 63.0 | 5.4% | 0 | |
| 5 | d1-ultra Liquid AI | 62.5 | 55.8–69.2 | 62.4 | 5.9% | 0 | |
| 6 | d1.1 Liquid AI | 61.0 | 54.2–67.8 | 61.4 | 5.7% | 0 | |
| 7 | Jev 1.13 TypeSafe | 59.0 | 52.2–65.8 | 57.3 | 10.5% | 0 | |
| 8 | 57.0 | 50.1–63.9 | 57.2 | 5.9% | 0 | ||
| 9 | GLiDE Fastino | 51.5 | 44.6–58.4 | 51.1 | 13.0% | 0 | |
| 10 | Mercury Decide Inception | 39.0 | 32.2–45.8 | 35.6 | 37.5% | 0 | |
| 11 | 35.0 | 28.4–41.6 | 28.0 | 36.0% | 0 | ||
| 11 | 35.0 | 28.4–41.6 | 28.2 | 28.3% | 0 | ||
| 13 | 33.5 | 27.0–40.0 | 25.5 | 45.3% | 0 | ||
| 14 | 28.5 | 22.2–34.8 | 14.8 | 63.6% | 0 |
The bar is accuracy; the bracket on it is the 95% interval for 200 cases. Tied for 1st marks every model whose gap to the leader is within sampling noise (1.96 standard errors of the difference, treating the runs as independent). A failed call counts as wrong.
It counts towards the Vision Index. Every model gets the same cases and the same request, under the benchmark rules.