system-one-bench

~/system-one-bench / benchmarks/sst-5

Text track · English · All benchmarks

SST-5

Sentiment on a 5-level score. Microsoft-Decision-1 has the top score, 0.5 points ahead of Jev 1.13, but Jev 1.13 and 22 more are within sampling noise: a statistical tie for first.

Top score
Microsoft-Decision-1
62.5, tied with 23 within noise
Lead over 2nd
0.5
Points of accuracy
Models
39
Charted on the leaderboard
Cases
200
Best constant answer
31.0
Always giving the most common label
Random guess
20.0
Uniform over the options

Leaderboard

RankModelAccuracy95% intervalMacro-F1Calibration errorFailed
1Microsoft-Decision-1 MicrosoftLeader62.555.8–69.2
61.711.4%0
2Jev 1.13 TypeSafeTied for 1st62.055.3–68.7
60.716.1%0
3GPT-6 Luna Decisions OpenAITied for 1st61.554.8–68.2
58.716.0%0
3d1-ultra Liquid AITied for 1st61.554.8–68.2
58.18.2%0
3GLiDE FastinoTied for 1st61.554.8–68.2
55.77.2%0
6Celeris-1 Decision CelerisTied for 1st61.054.2–67.8
59.46.3%0
7Jev 1.14 TypeSafeTied for 1st60.053.2–66.8
58.618.1%0
7Decider V1.2 PerplexityTied for 1st60.053.2–66.8
55.47.0%0
7d1.1 Liquid AITied for 1st60.053.2–66.8
55.44.5%0
7Dexy-Decision-9B Patrick-18Tied for 1st60.053.2–66.8
57.73.9%0
11Drex Max Nace AITied for 1st59.052.2–65.8
54.89.6%0
11Mammoth 1 BasetenTied for 1st59.052.2–65.8
57.49.7%0
11Gemini Decision 1 GoogleTied for 1st59.052.2–65.8
55.08.1%1
14Elephant 1 BasetenTied for 1st58.551.7–65.3
56.712.9%0
15Span-02 RespanTied for 1st57.550.6–64.4
57.921.8%0
16Clef Flash CloudflareTied for 1st57.050.1–63.9
55.413.8%0
17d1 Liquid AITied for 1st56.549.6–63.4
52.68.0%0
18d1-3B Liquid AITied for 1st56.049.1–62.9
55.05.7%0
18Drex v2.0 Nace AITied for 1st56.049.1–62.9
45.78.7%0
20d3 nano vLLM Semantic RouterTied for 1st55.548.6–62.4
45.26.2%0
20Decider 1 meraGPTTied for 1st55.548.6–62.4
52.79.4%0
22Decider Flash V1.2 PerplexityTied for 1st54.047.1–60.9
54.46.8%0
23Solar Decide UpstageTied for 1st53.546.6–60.4
53.442.3%0
23Drex v1.5 Nace AITied for 1st53.546.6–60.4
46.410.3%0
25Mercury Decide Inception52.045.1–58.9
50.922.8%0
25Kev 4B Jared Palmerin-domain52.045.1–58.9
50.213.9%0
27Clef Cloudflare51.544.6–58.4
48.910.9%0
28Clef Omni Cloudflare51.044.1–57.9
48.014.4%0
29d3 lite vLLM Semantic Router50.543.6–57.4
43.64.9%0
30d1-omni-600M Liquid AI47.040.1–53.9
43.711.1%0
30Kev 0.8B Jared Palmerin-domain47.040.1–53.9
44.016.2%0
32Solar Decide Flash Upstage44.537.6–51.4
44.131.5%0
33Tev1 4B Experimental Together AI43.036.1–49.9
44.726.7%0
34Joey 1 Baseten40.533.7–47.3
40.28.1%0
35d3 edge vLLM Semantic Router39.532.7–46.3
33.312.4%0
36Laya Router Convai Innovations34.027.4–40.6
21.438.1%0
36Naluri-279M Jericho Siahaya34.027.4–40.6
33.09.1%0
38Naluri-308M Jericho Siahaya28.021.8–34.2
27.932.5%0
39Classify-1 ScaleDown13.08.3–17.7
4.67.0%200

The bar is accuracy; the bracket on it is the 95% interval for 200 cases. Tied for 1st marks every model whose gap to the leader is within sampling noise (1.96 standard errors of the difference, treating the runs as independent). A failed call counts as wrong. In-domain models trained on another split of this dataset, by their maker's own account.

About this benchmark

Question
Sentiment on a 5-level score
Source
SetFit/sst5, test split
Sample
200 random reviews
Pinned
Every source is pinned to an exact commit, and the cases are frozen by hash in data.lock.json.

It counts towards the System One Index and its English sub-index. Every model gets the same cases and the same request, under the benchmark rules.