system-one-bench

~/system-one-bench / benchmarks/xnli-english

Text track · English · All benchmarks

XNLI English

Does one sentence follow from another, 3 options. Drex Max has the top score, 0.7 points ahead of GLiDE, but GLiDE and 12 more are within sampling noise: a statistical tie for first.

Top score
Drex Max
95.3, tied with 13 within noise
Lead over 2nd
0.7
Points of accuracy
Models
39
Charted on the leaderboard
Cases
150
Best constant answer
40.0
Always giving the most common label
Random guess
33.3
Uniform over the options

Leaderboard

RankModelAccuracy95% intervalMacro-F1Calibration errorFailed
1Drex Max Nace AILeader95.392.0–98.7
95.13.1%0
2GLiDE FastinoTied for 1st94.791.1–98.3
94.44.2%0
2Dexy-Decision-9B Patrick-18Tied for 1st94.791.1–98.3
94.43.0%0
4Decider V1.2 PerplexityTied for 1st94.090.2–97.8
93.64.2%0
4Elephant 1 BasetenTied for 1st94.090.2–97.8
93.74.0%0
6Decider Flash V1.2 PerplexityTied for 1st92.788.5–96.8
92.33.2%0
6d1-ultra Liquid AITied for 1st92.788.5–96.8
92.611.0%0
8d1 Liquid AITied for 1st91.386.8–95.8
90.72.5%0
8Mercury Decide InceptionTied for 1st91.386.8–95.8
91.24.6%0
8Mammoth 1 BasetenTied for 1st91.386.8–95.8
91.23.1%0
8Celeris-1 Decision CelerisTied for 1st91.386.8–95.8
90.69.2%0
12Kev 4B Jared PalmerTied for 1st90.786.0–95.3
90.36.7%0
12d3 lite vLLM Semantic RouterTied for 1st90.786.0–95.3
90.36.0%0
14d1.1 Liquid AITied for 1st90.085.2–94.8
89.75.9%0
15Microsoft-Decision-1 Microsoft89.384.4–94.3
89.02.6%0
16Clef Cloudflare88.783.6–93.7
88.76.2%0
16Drex v1.5 Nace AI88.783.6–93.7
88.63.1%0
18Laya Router Convai Innovations87.382.0–92.7
87.14.3%0
18Tev1 4B Experimental Together AI87.382.0–92.7
86.54.7%0
18d3 nano vLLM Semantic Router87.382.0–92.7
86.65.4%0
18Drex v2.0 Nace AI87.382.0–92.7
87.16.3%0
18Gemini Decision 1 Google87.382.0–92.7
87.13.4%0
23Jev 1.13 TypeSafe86.080.4–91.6
85.78.2%0
23Jev 1.14 TypeSafe86.080.4–91.6
85.77.0%0
23d1-3B Liquid AI86.080.4–91.6
85.53.1%0
26Clef Flash Cloudflare84.778.9–90.4
84.65.8%0
27d3 edge vLLM Semantic Router84.078.1–89.9
83.63.6%0
27Joey 1 Baseten84.078.1–89.9
83.75.7%0
29Span-02 Respan83.377.4–89.3
83.54.4%0
30Clef Omni Cloudflare82.776.6–88.7
82.010.8%0
31d1-omni-600M Liquid AI82.075.9–88.1
81.711.6%0
32Kev 0.8B Jared Palmer81.375.1–87.6
80.73.8%0
33Decider 1 meraGPT80.774.4–87.0
79.83.6%0
34Naluri-308M Jericho Siahayain-domain76.069.2–82.8
75.24.7%0
35Solar Decide Upstage74.767.7–81.6
75.120.8%0
36GPT-6 Luna Decisions OpenAI66.759.1–74.2
67.520.7%0
37Solar Decide Flash Upstage65.357.7–72.9
65.921.0%0
38Naluri-279M Jericho Siahayain-domain30.723.3–38.0
16.140.6%0
38Classify-1 ScaleDown30.723.3–38.0
15.72.7%150

The bar is accuracy; the bracket on it is the 95% interval for 150 cases. Tied for 1st marks every model whose gap to the leader is within sampling noise (1.96 standard errors of the difference, treating the runs as independent). A failed call counts as wrong. In-domain models trained on another split of this dataset, by their maker's own account.

About this benchmark

Question
Does one sentence follow from another, 3 options
Source
facebook/xnli, test split
Sample
150 random pairs, the same items as XNLI French and Arabic
Pinned
Every source is pinned to an exact commit, and the cases are frozen by hash in data.lock.json.

It counts towards the System One Index and its English sub-index. Every model gets the same cases and the same request, under the benchmark rules.