system-one-bench

~/system-one-bench / benchmarks/arabic-sentiment-as-an-image

Vision track · All benchmarks

Arabic sentiment, as an image

The 200 Arabic tweets rendered as a picture, 3 options. Gemini Decision 1 has the top score, 5.0 points ahead of Span-02, but Span-02 and 1 more are within sampling noise: a statistical tie for first.

Top score
Gemini Decision 1
75.0, tied with 2 within noise
Lead over 2nd
5.0
Points of accuracy
Models
14
Charted on the leaderboard
Cases
200
Best constant answer
36.5
Always giving the most common label
Random guess
33.3
Uniform over the options

Leaderboard

RankModelAccuracy95% intervalMacro-F1Calibration errorFailed
1Gemini Decision 1 GoogleLeader75.069.0–81.0
74.66.6%0
2Span-02 RespanTied for 1st70.063.6–76.4
70.711.5%0
3GPT-6 Luna Decisions OpenAITied for 1st67.060.5–73.5
66.613.5%0
4Jev 1.14 TypeSafe63.556.8–70.2
63.05.4%0
5d1-ultra Liquid AI62.555.8–69.2
62.45.9%0
6d1.1 Liquid AI61.054.2–67.8
61.45.7%0
7Jev 1.13 TypeSafe59.052.2–65.8
57.310.5%0
8Decider V1.2 Perplexity57.050.1–63.9
57.25.9%0
9GLiDE Fastino51.544.6–58.4
51.113.0%0
10Mercury Decide Inception39.032.2–45.8
35.637.5%0
11Clef Cloudflare35.028.4–41.6
28.036.0%0
11Decider Flash V1.2 Perplexity35.028.4–41.6
28.228.3%0
13Clef Flash Cloudflare33.527.0–40.0
25.545.3%0
14Clef Omni Cloudflare28.522.2–34.8
14.863.6%0

The bar is accuracy; the bracket on it is the 95% interval for 200 cases. Tied for 1st marks every model whose gap to the leader is within sampling noise (1.96 standard errors of the difference, treating the runs as independent). A failed call counts as wrong.

About this benchmark

Question
The 200 Arabic tweets rendered as a picture, 3 options
Source
mteb/tweet_sentiment_multilingual, test split
Sample
The 200 Arabic tweets from the text track, rendered in Noto Naskh Arabic
Pinned
Every source is pinned to an exact commit, and the cases are frozen by hash in data.lock.json.

It counts towards the Vision Index. Every model gets the same cases and the same request, under the benchmark rules.