~/system-one-bench / benchmarks/sst-2
Text track · English · All benchmarks
Movie review sentiment, yes/no. d1.1 has the top score, 1.0 points ahead of Clef, but Clef and 20 more are within sampling noise: a statistical tie for first.
| Rank | Model | Accuracy | 95% interval | Macro-F1 | Calibration error | Failed | |
|---|---|---|---|---|---|---|---|
| 1 | d1.1 Liquid AILeader | 98.0 | 96.1–99.9 | 98.0 | 6.4% | 0 | |
| 2 | 97.0 | 94.6–99.4 | 97.0 | 4.2% | 0 | ||
| 2 | 97.0 | 94.6–99.4 | 97.0 | 1.4% | 0 | ||
| 4 | 96.5 | 94.0–99.0 | 96.5 | 5.0% | 0 | ||
| 5 | 96.0 | 93.3–98.7 | 96.0 | 2.9% | 0 | ||
| 5 | Mammoth 1 BasetenTied for 1st | 96.0 | 93.3–98.7 | 96.0 | 4.1% | 0 | |
| 5 | GLiDE FastinoTied for 1st | 96.0 | 93.3–98.7 | 96.0 | 3.5% | 0 | |
| 8 | d1 Liquid AITied for 1st | 95.5 | 92.6–98.4 | 95.5 | 2.0% | 0 | |
| 8 | Mercury Decide InceptionTied for 1st | 95.5 | 92.6–98.4 | 95.5 | 1.6% | 0 | |
| 8 | 95.5 | 92.6–98.4 | 95.5 | 2.1% | 0 | ||
| 8 | d1-3B Liquid AITied for 1st | 95.5 | 92.6–98.4 | 95.5 | 4.4% | 0 | |
| 8 | 95.5 | 92.6–98.4 | 95.5 | 2.6% | 0 | ||
| 8 | d1-ultra Liquid AITied for 1st | 95.5 | 92.6–98.4 | 95.5 | 2.1% | 0 | |
| 8 | Span-02 RespanTied for 1st | 95.5 | 92.6–98.4 | 95.5 | 2.5% | 0 | |
| 8 | Celeris-1 Decision CelerisTied for 1st | 95.5 | 92.6–98.4 | 95.5 | 4.1% | 0 | |
| 8 | Dexy-Decision-9B Patrick-18Tied for 1st | 95.5 | 92.6–98.4 | 95.5 | 4.3% | 0 | |
| 17 | Jev 1.14 TypeSafeTied for 1st | 95.0 | 92.0–98.0 | 95.0 | 7.7% | 0 | |
| 17 | 95.0 | 92.0–98.0 | 95.0 | 3.0% | 0 | ||
| 17 | 95.0 | 92.0–98.0 | 95.0 | 3.7% | 0 | ||
| 20 | Jev 1.13 TypeSafeTied for 1st | 94.5 | 91.3–97.7 | 94.5 | 6.4% | 0 | |
| 20 | 94.5 | 91.3–97.7 | 94.5 | 2.5% | 0 | ||
| 20 | Elephant 1 BasetenTied for 1st | 94.5 | 91.3–97.7 | 94.5 | 2.9% | 0 | |
| 23 | 94.0 | 90.7–97.3 | 94.0 | 3.7% | 0 | ||
| 23 | 94.0 | 90.7–97.3 | 94.0 | 11.7% | 0 | ||
| 25 | 93.5 | 90.1–96.9 | 93.5 | 4.8% | 0 | ||
| 26 | 93.0 | 89.5–96.5 | 93.0 | 13.2% | 0 | ||
| 27 | Kev 4B Jared Palmer | 92.5 | 88.8–96.2 | 92.5 | 4.5% | 0 | |
| 28 | 91.5 | 87.6–95.4 | 91.5 | 5.9% | 0 | ||
| 29 | 90.5 | 86.4–94.6 | 90.5 | 7.1% | 0 | ||
| 30 | Kev 0.8B Jared Palmer | 90.0 | 85.8–94.2 | 90.0 | 4.6% | 0 | |
| 31 | 89.5 | 85.3–93.7 | 89.5 | 3.1% | 0 | ||
| 32 | Laya Router Convai Innovations | 88.5 | 84.1–92.9 | 88.5 | 4.9% | 0 | |
| 33 | d1-omni-600M Liquid AI | 86.5 | 81.8–91.2 | 86.5 | 5.7% | 0 | |
| 34 | 86.0 | 81.2–90.8 | 86.0 | 5.4% | 0 | ||
| 34 | Joey 1 Baseten | 86.0 | 81.2–90.8 | 85.9 | 7.0% | 0 | |
| 36 | 80.5 | 75.0–86.0 | 80.5 | 8.5% | 0 | ||
| 37 | Naluri-279M Jericho Siahaya | 77.0 | 71.2–82.8 | 76.7 | 5.3% | 0 | |
| 38 | Naluri-308M Jericho Siahaya | 68.5 | 62.1–74.9 | 68.4 | 3.8% | 0 | |
| 39 | Classify-1 ScaleDown | 50.0 | 43.1–56.9 | 33.3 | 0.0% | 200 |
The bar is accuracy; the bracket on it is the 95% interval for 200 cases. Tied for 1st marks every model whose gap to the leader is within sampling noise (1.96 standard errors of the difference, treating the runs as independent). A failed call counts as wrong.
It counts towards the System One Index and its English sub-index. Every model gets the same cases and the same request, under the benchmark rules.