system-one-bench

~/system-one-bench / benchmarks/xnli-arabic

Text track · Arabic · All benchmarks

XNLI Arabic

Does one sentence follow from another, 3 options. Decider V1.2 has the top score, 0.7 points ahead of Drex Max, but Drex Max and 3 more are within sampling noise: a statistical tie for first.

Top score
Decider V1.2
90.0, tied with 4 within noise
Lead over 2nd
0.7
Points of accuracy
Models
41
Charted on the leaderboard
Cases
150
Best constant answer
40.0
Always giving the most common label
Random guess
33.3
Uniform over the options

Leaderboard

RankModelAccuracy95% intervalMacro-F1Calibration errorFailed
1Decider V1.2 PerplexityLeader90.085.2–94.8
89.67.7%0
2Drex Max Nace AITied for 1st89.384.4–94.3
89.03.8%0
2Dexy-Decision-9B Patrick-18Tied for 1st89.384.4–94.3
88.64.4%0
4Decider Flash V1.2 PerplexityTied for 1st85.379.7–91.0
84.95.6%0
5GLiDE FastinoTied for 1st83.377.4–89.3
83.22.9%0
6d3 nano vLLM Semantic Router82.075.9–88.1
81.76.9%0
7d1 Liquid AI80.073.6–86.4
79.56.1%0
7Celeris-1 Decision Celeris80.073.6–86.4
79.65.5%0
9d1.1 Liquid AI79.372.8–85.8
79.58.2%0
9Elephant 1 Baseten79.372.8–85.8
79.37.4%0
11Mammoth 1 Baseten78.772.1–85.2
78.94.9%0
12Clef Cloudflare78.071.4–84.6
78.37.5%0
12Gemini Decision 1 Google78.071.4–84.6
77.911.7%0
14Mercury Decide Inception76.069.2–82.8
75.413.8%0
14Tev1 4B Experimental Together AI76.069.2–82.8
74.68.5%0
14d1-ultra Liquid AI76.069.2–82.8
75.99.8%0
17laya-ara Wouzein-domain75.368.4–82.2
74.818.6%0
18Drex v1.5 Nace AI74.067.0–81.0
74.114.0%0
19Clef Omni Cloudflare73.366.3–80.4
72.38.2%0
19d1-3B Liquid AI73.366.3–80.4
72.77.5%0
21Kev 4B Jared Palmer72.765.5–79.8
71.920.1%0
21d3 lite vLLM Semantic Router72.765.5–79.8
72.06.3%0
23Microsoft-Decision-1 Microsoft72.064.8–79.2
72.512.5%0
24Jev 1.14 TypeSafe71.364.1–78.6
71.511.4%0
24Drex v2.0 Nace AI71.364.1–78.6
70.56.2%0
26laya-ara-rag Wouze70.062.7–77.3
69.68.7%0
27Jev 1.13 TypeSafe69.362.0–76.7
69.814.0%0
28Clef Flash Cloudflare68.761.2–76.1
67.77.8%0
29Laya Router Convai Innovations68.060.5–75.5
66.923.4%0
30Decider 1 meraGPT67.359.8–74.8
65.62.4%0
31Span-02 Respan66.058.4–73.6
66.117.9%0
32Joey 1 Baseten64.757.0–72.3
65.212.1%0
33d3 edge vLLM Semantic Router59.351.5–67.2
58.912.2%0
34Kev 0.8B Jared Palmer58.750.8–66.6
55.418.6%0
35Solar Decide Upstage58.050.1–65.9
57.836.8%0
36Naluri-308M Jericho Siahayain-domain56.748.7–64.6
56.614.3%0
37GPT-6 Luna Decisions OpenAI54.046.0–62.0
51.929.8%0
38d1-omni-600M Liquid AI52.744.7–60.7
52.213.9%0
39Solar Decide Flash Upstage52.044.0–60.0
49.827.9%0
40Naluri-279M Jericho Siahayain-domain30.723.3–38.0
15.940.9%0
40Classify-1 ScaleDown30.723.3–38.0
15.72.7%150

The bar is accuracy; the bracket on it is the 95% interval for 150 cases. Tied for 1st marks every model whose gap to the leader is within sampling noise (1.96 standard errors of the difference, treating the runs as independent). A failed call counts as wrong. In-domain models trained on another split of this dataset, by their maker's own account.

About this benchmark

Question
Does one sentence follow from another, 3 options
Source
facebook/xnli, test split
Sample
150 random pairs, the same items as XNLI English and French
Pinned
Every source is pinned to an exact commit, and the cases are frozen by hash in data.lock.json.

It counts towards the System One Index and its Arabic sub-index. Every model gets the same cases and the same request, under the benchmark rules.