system-one-bench

~/system-one-bench / benchmarks/sst-2

Text track · English · All benchmarks

SST-2

Movie review sentiment, yes/no. d1.1 has the top score, 1.0 points ahead of Clef, but Clef and 20 more are within sampling noise: a statistical tie for first.

Top score
d1.1
98.0, tied with 21 within noise
Lead over 2nd
1.0
Points of accuracy
Models
39
Charted on the leaderboard
Cases
200
Best constant answer
50.0
Always giving the most common label
Random guess
50.0
Uniform over the options

Leaderboard

RankModelAccuracy95% intervalMacro-F1Calibration errorFailed
1d1.1 Liquid AILeader98.096.1–99.9
98.06.4%0
2Clef CloudflareTied for 1st97.094.6–99.4
97.04.2%0
2Clef Omni CloudflareTied for 1st97.094.6–99.4
97.01.4%0
4Gemini Decision 1 GoogleTied for 1st96.594.0–99.0
96.55.0%0
5Drex Max Nace AITied for 1st96.093.3–98.7
96.02.9%0
5Mammoth 1 BasetenTied for 1st96.093.3–98.7
96.04.1%0
5GLiDE FastinoTied for 1st96.093.3–98.7
96.03.5%0
8d1 Liquid AITied for 1st95.592.6–98.4
95.52.0%0
8Mercury Decide InceptionTied for 1st95.592.6–98.4
95.51.6%0
8Microsoft-Decision-1 MicrosoftTied for 1st95.592.6–98.4
95.52.1%0
8d1-3B Liquid AITied for 1st95.592.6–98.4
95.54.4%0
8Decider V1.2 PerplexityTied for 1st95.592.6–98.4
95.52.6%0
8d1-ultra Liquid AITied for 1st95.592.6–98.4
95.52.1%0
8Span-02 RespanTied for 1st95.592.6–98.4
95.52.5%0
8Celeris-1 Decision CelerisTied for 1st95.592.6–98.4
95.54.1%0
8Dexy-Decision-9B Patrick-18Tied for 1st95.592.6–98.4
95.54.3%0
17Jev 1.14 TypeSafeTied for 1st95.092.0–98.0
95.07.7%0
17Clef Flash CloudflareTied for 1st95.092.0–98.0
95.03.0%0
17Decider Flash V1.2 PerplexityTied for 1st95.092.0–98.0
95.03.7%0
20Jev 1.13 TypeSafeTied for 1st94.591.3–97.7
94.56.4%0
20Drex v1.5 Nace AITied for 1st94.591.3–97.7
94.52.5%0
20Elephant 1 BasetenTied for 1st94.591.3–97.7
94.52.9%0
23Tev1 4B Experimental Together AI94.090.7–97.3
94.03.7%0
23Drex v2.0 Nace AI94.090.7–97.3
94.011.7%0
25GPT-6 Luna Decisions OpenAI93.590.1–96.9
93.54.8%0
26Decider 1 meraGPT93.089.5–96.5
93.013.2%0
27Kev 4B Jared Palmer92.588.8–96.2
92.54.5%0
28Solar Decide Upstage91.587.6–95.4
91.55.9%0
29Solar Decide Flash Upstage90.586.4–94.6
90.57.1%0
30Kev 0.8B Jared Palmer90.085.8–94.2
90.04.6%0
31d3 nano vLLM Semantic Router89.585.3–93.7
89.53.1%0
32Laya Router Convai Innovations88.584.1–92.9
88.54.9%0
33d1-omni-600M Liquid AI86.581.8–91.2
86.55.7%0
34d3 lite vLLM Semantic Router86.081.2–90.8
86.05.4%0
34Joey 1 Baseten86.081.2–90.8
85.97.0%0
36d3 edge vLLM Semantic Router80.575.0–86.0
80.58.5%0
37Naluri-279M Jericho Siahaya77.071.2–82.8
76.75.3%0
38Naluri-308M Jericho Siahaya68.562.1–74.9
68.43.8%0
39Classify-1 ScaleDown50.043.1–56.9
33.30.0%200

The bar is accuracy; the bracket on it is the 95% interval for 200 cases. Tied for 1st marks every model whose gap to the leader is within sampling noise (1.96 standard errors of the difference, treating the runs as independent). A failed call counts as wrong.

About this benchmark

Question
Movie review sentiment, yes/no
Source
stanfordnlp/sst2, validation split
Sample
200 random reviews
Pinned
Every source is pinned to an exact commit, and the cases are frozen by hash in data.lock.json.

It counts towards the System One Index and its English sub-index. Every model gets the same cases and the same request, under the benchmark rules.