system-one-bench

~/system-one-bench / benchmarks/arabic-tweet-sentiment

Text track · Arabic · All benchmarks

Tweet sentiment

Sentiment, 3 options. GPT-6 Luna Decisions has the top score, 2.0 points ahead of Gemini Decision 1, but Gemini Decision 1 and 7 more are within sampling noise: a statistical tie for first.

Top score
GPT-6 Luna Decisions
80.0, tied with 8 within noise
Lead over 2nd
2.0
Points of accuracy
Models
41
Charted on the leaderboard
Cases
200
Best constant answer
36.5
Always giving the most common label
Random guess
33.3
Uniform over the options

Leaderboard

RankModelAccuracy95% intervalMacro-F1Calibration errorFailed
1GPT-6 Luna Decisions OpenAILeader80.074.5–85.5
79.710.7%0
2Gemini Decision 1 GoogleTied for 1st78.072.3–83.7
77.64.7%0
3Jev 1.13 TypeSafeTied for 1st76.070.1–81.9
75.69.7%0
3Jev 1.14 TypeSafeTied for 1st76.070.1–81.9
75.69.5%0
3d1-ultra Liquid AITied for 1st76.070.1–81.9
76.48.1%0
3Span-02 RespanTied for 1st76.070.1–81.9
76.210.1%0
7Decider V1.2 PerplexityTied for 1st75.569.5–81.5
75.33.7%0
8GLiDE FastinoTied for 1st73.567.4–79.6
73.75.7%0
9d1.1 Liquid AITied for 1st72.566.3–78.7
72.94.4%0
10Solar Decide Upstage71.565.2–77.8
71.224.8%0
10Decider Flash V1.2 Perplexity71.565.2–77.8
71.84.4%0
12d3 nano vLLM Semantic Router71.064.7–77.3
71.46.1%0
13Tev1 4B Experimental Together AI70.564.2–76.8
69.96.8%0
14Drex v2.0 Nace AI69.563.1–75.9
69.95.7%0
15Drex Max Nace AI69.062.6–75.4
69.54.5%0
15Celeris-1 Decision Celeris69.062.6–75.4
69.76.3%0
17Microsoft-Decision-1 Microsoft68.562.1–74.9
69.014.8%0
18d1 Liquid AI67.561.0–74.0
67.912.5%0
19d3 lite vLLM Semantic Router67.060.5–73.5
67.44.9%0
19Dexy-Decision-9B Patrick-1867.060.5–73.5
67.213.1%0
21Elephant 1 Baseten66.059.4–72.6
66.313.7%0
21Joey 1 Baseten66.059.4–72.6
65.77.3%0
23Solar Decide Flash Upstage65.558.9–72.1
65.121.8%0
24Mammoth 1 Baseten64.557.9–71.1
64.815.1%0
25Decider 1 meraGPT64.057.3–70.7
64.19.6%0
26Drex v1.5 Nace AI63.556.8–70.2
63.217.8%0
26d1-3B Liquid AI63.556.8–70.2
63.211.2%0
28Kev 4B Jared Palmer63.056.3–69.7
63.012.9%0
29Mercury Decide Inception62.555.8–69.2
62.822.0%0
30d3 edge vLLM Semantic Router61.554.8–68.2
61.54.3%0
31Clef Cloudflare58.051.2–64.8
57.820.6%0
31Naluri-308M Jericho Siahaya58.051.2–64.8
58.36.1%0
33Kev 0.8B Jared Palmer57.050.1–63.9
56.710.8%0
34Naluri-279M Jericho Siahayaunusually high53.046.1–59.9
53.26.2%0
35Clef Omni Cloudflare50.043.1–56.9
47.134.8%0
36laya-ara Wouze49.042.1–55.9
49.725.4%0
37Clef Flash Cloudflare47.040.1–53.9
43.735.7%0
38laya-ara-rag Wouze44.037.1–50.9
40.017.0%0
39Laya Router Convai Innovations37.030.3–43.7
32.545.6%0
40Classify-1 ScaleDown35.028.4–41.6
17.31.7%200
41d1-omni-600M Liquid AI32.526.0–39.0
22.344.1%0

The bar is accuracy; the bracket on it is the 95% interval for 200 cases. Tied for 1st marks every model whose gap to the leader is within sampling noise (1.96 standard errors of the difference, treating the runs as independent). A failed call counts as wrong. Unusually high marks a score far above what that model's other datasets, and every other model's score here, predict (hover for the numbers). It often means the model trained on this dataset or one like it; it isn't proof, and the score still counts.

About this benchmark

Question
Sentiment, 3 options
Source
mteb/tweet_sentiment_multilingual, test split
Sample
200 random Arabic tweets
Pinned
Every source is pinned to an exact commit, and the cases are frozen by hash in data.lock.json.

It counts towards the System One Index and its Arabic sub-index. Every model gets the same cases and the same request, under the benchmark rules.