system-one-bench

~/system-one-bench / benchmarks/xnli-french

Text track · French · All benchmarks

XNLI French

Does one sentence follow from another, 3 options. Drex Max has the top score, 1.3 points ahead of Decider V1.2, but Decider V1.2 and 8 more are within sampling noise: a statistical tie for first.

Top score
Drex Max
91.3, tied with 9 within noise
Lead over 2nd
1.3
Points of accuracy
Models
39
Charted on the leaderboard
Cases
150
Best constant answer
40.0
Always giving the most common label
Random guess
33.3
Uniform over the options

Leaderboard

RankModelAccuracy95% intervalMacro-F1Calibration errorFailed
1Drex Max Nace AILeader91.386.8–95.8
90.73.5%0
2Decider V1.2 PerplexityTied for 1st90.085.2–94.8
89.35.6%0
2Elephant 1 BasetenTied for 1st90.085.2–94.8
89.75.4%0
2Celeris-1 Decision CelerisTied for 1st90.085.2–94.8
89.19.1%0
5Dexy-Decision-9B Patrick-18Tied for 1st89.384.4–94.3
88.76.2%0
6Decider Flash V1.2 PerplexityTied for 1st88.082.8–93.2
86.94.9%0
6GLiDE FastinoTied for 1st88.082.8–93.2
87.93.9%0
8d3 nano vLLM Semantic RouterTied for 1st86.080.4–91.6
85.44.8%0
8Mammoth 1 BasetenTied for 1st86.080.4–91.6
86.04.3%0
10d1 Liquid AITied for 1st84.778.9–90.4
84.27.7%0
11d1-ultra Liquid AI83.377.4–89.3
83.210.0%0
11d1.1 Liquid AI83.377.4–89.3
82.97.8%0
13Mercury Decide Inception82.776.6–88.7
82.210.5%0
13Tev1 4B Experimental Together AI82.776.6–88.7
81.85.9%0
13Clef Omni Cloudflare82.776.6–88.7
81.86.9%0
16Microsoft-Decision-1 Microsoft82.075.9–88.1
82.011.0%0
16Drex v1.5 Nace AI82.075.9–88.1
82.210.1%0
18d3 lite vLLM Semantic Router81.375.1–87.6
80.94.7%0
18Gemini Decision 1 Google81.375.1–87.6
81.25.6%0
20Clef Cloudflare80.073.6–86.4
79.86.2%0
20Clef Flash Cloudflare80.073.6–86.4
79.03.5%0
22Kev 4B Jared Palmer79.372.8–85.8
78.514.1%0
23d1-3B Liquid AI78.772.1–85.2
78.17.0%0
24Drex v2.0 Nace AI77.370.6–84.0
76.77.1%0
25Jev 1.13 TypeSafe76.769.9–83.4
76.712.2%0
25Jev 1.14 TypeSafe76.769.9–83.4
76.710.7%0
27Laya Router Convai Innovations76.069.2–82.8
75.116.7%0
28Joey 1 Baseten74.767.7–81.6
74.711.8%0
29Span-02 Respan73.366.3–80.4
73.911.8%0
30Naluri-308M Jericho Siahayain-domain68.060.5–75.5
67.211.1%0
31Solar Decide Upstage67.359.8–74.8
66.227.8%0
31Decider 1 meraGPT67.359.8–74.8
65.98.1%0
33d3 edge vLLM Semantic Router66.058.4–73.6
66.08.4%0
33Kev 0.8B Jared Palmer66.058.4–73.6
63.712.7%0
35d1-omni-600M Liquid AI65.357.7–72.9
65.215.8%0
36GPT-6 Luna Decisions OpenAI62.054.2–69.8
62.021.5%0
37Solar Decide Flash Upstage61.353.5–69.1
60.421.4%0
38Naluri-279M Jericho Siahayain-domain31.323.9–38.8
17.641.6%0
39Classify-1 ScaleDown30.723.3–38.0
15.72.7%150

The bar is accuracy; the bracket on it is the 95% interval for 150 cases. Tied for 1st marks every model whose gap to the leader is within sampling noise (1.96 standard errors of the difference, treating the runs as independent). A failed call counts as wrong. In-domain models trained on another split of this dataset, by their maker's own account.

About this benchmark

Question
Does one sentence follow from another, 3 options
Source
facebook/xnli, test split
Sample
150 random pairs, the same items as XNLI English and Arabic
Pinned
Every source is pinned to an exact commit, and the cases are frozen by hash in data.lock.json.

It counts towards the System One Index. Every model gets the same cases and the same request, under the benchmark rules.