~/system-one-bench / benchmarks/sst-5
Text track · English · All benchmarks
Sentiment on a 5-level score. Microsoft-Decision-1 has the top score, 0.5 points ahead of Jev 1.13, but Jev 1.13 and 22 more are within sampling noise: a statistical tie for first.
| Rank | Model | Accuracy | 95% interval | Macro-F1 | Calibration error | Failed | |
|---|---|---|---|---|---|---|---|
| 1 | 62.5 | 55.8–69.2 | 61.7 | 11.4% | 0 | ||
| 2 | Jev 1.13 TypeSafeTied for 1st | 62.0 | 55.3–68.7 | 60.7 | 16.1% | 0 | |
| 3 | 61.5 | 54.8–68.2 | 58.7 | 16.0% | 0 | ||
| 3 | d1-ultra Liquid AITied for 1st | 61.5 | 54.8–68.2 | 58.1 | 8.2% | 0 | |
| 3 | GLiDE FastinoTied for 1st | 61.5 | 54.8–68.2 | 55.7 | 7.2% | 0 | |
| 6 | Celeris-1 Decision CelerisTied for 1st | 61.0 | 54.2–67.8 | 59.4 | 6.3% | 0 | |
| 7 | Jev 1.14 TypeSafeTied for 1st | 60.0 | 53.2–66.8 | 58.6 | 18.1% | 0 | |
| 7 | 60.0 | 53.2–66.8 | 55.4 | 7.0% | 0 | ||
| 7 | d1.1 Liquid AITied for 1st | 60.0 | 53.2–66.8 | 55.4 | 4.5% | 0 | |
| 7 | Dexy-Decision-9B Patrick-18Tied for 1st | 60.0 | 53.2–66.8 | 57.7 | 3.9% | 0 | |
| 11 | 59.0 | 52.2–65.8 | 54.8 | 9.6% | 0 | ||
| 11 | Mammoth 1 BasetenTied for 1st | 59.0 | 52.2–65.8 | 57.4 | 9.7% | 0 | |
| 11 | 59.0 | 52.2–65.8 | 55.0 | 8.1% | 1 | ||
| 14 | Elephant 1 BasetenTied for 1st | 58.5 | 51.7–65.3 | 56.7 | 12.9% | 0 | |
| 15 | Span-02 RespanTied for 1st | 57.5 | 50.6–64.4 | 57.9 | 21.8% | 0 | |
| 16 | 57.0 | 50.1–63.9 | 55.4 | 13.8% | 0 | ||
| 17 | d1 Liquid AITied for 1st | 56.5 | 49.6–63.4 | 52.6 | 8.0% | 0 | |
| 18 | d1-3B Liquid AITied for 1st | 56.0 | 49.1–62.9 | 55.0 | 5.7% | 0 | |
| 18 | 56.0 | 49.1–62.9 | 45.7 | 8.7% | 0 | ||
| 20 | 55.5 | 48.6–62.4 | 45.2 | 6.2% | 0 | ||
| 20 | 55.5 | 48.6–62.4 | 52.7 | 9.4% | 0 | ||
| 22 | 54.0 | 47.1–60.9 | 54.4 | 6.8% | 0 | ||
| 23 | 53.5 | 46.6–60.4 | 53.4 | 42.3% | 0 | ||
| 23 | 53.5 | 46.6–60.4 | 46.4 | 10.3% | 0 | ||
| 25 | Mercury Decide Inception | 52.0 | 45.1–58.9 | 50.9 | 22.8% | 0 | |
| 25 | Kev 4B Jared Palmerin-domain | 52.0 | 45.1–58.9 | 50.2 | 13.9% | 0 | |
| 27 | 51.5 | 44.6–58.4 | 48.9 | 10.9% | 0 | ||
| 28 | 51.0 | 44.1–57.9 | 48.0 | 14.4% | 0 | ||
| 29 | 50.5 | 43.6–57.4 | 43.6 | 4.9% | 0 | ||
| 30 | d1-omni-600M Liquid AI | 47.0 | 40.1–53.9 | 43.7 | 11.1% | 0 | |
| 30 | Kev 0.8B Jared Palmerin-domain | 47.0 | 40.1–53.9 | 44.0 | 16.2% | 0 | |
| 32 | 44.5 | 37.6–51.4 | 44.1 | 31.5% | 0 | ||
| 33 | 43.0 | 36.1–49.9 | 44.7 | 26.7% | 0 | ||
| 34 | Joey 1 Baseten | 40.5 | 33.7–47.3 | 40.2 | 8.1% | 0 | |
| 35 | 39.5 | 32.7–46.3 | 33.3 | 12.4% | 0 | ||
| 36 | Laya Router Convai Innovations | 34.0 | 27.4–40.6 | 21.4 | 38.1% | 0 | |
| 36 | Naluri-279M Jericho Siahaya | 34.0 | 27.4–40.6 | 33.0 | 9.1% | 0 | |
| 38 | Naluri-308M Jericho Siahaya | 28.0 | 21.8–34.2 | 27.9 | 32.5% | 0 | |
| 39 | Classify-1 ScaleDown | 13.0 | 8.3–17.7 | 4.6 | 7.0% | 200 |
The bar is accuracy; the bracket on it is the 95% interval for 200 cases. Tied for 1st marks every model whose gap to the leader is within sampling noise (1.96 standard errors of the difference, treating the runs as independent). A failed call counts as wrong. In-domain models trained on another split of this dataset, by their maker's own account.
It counts towards the System One Index and its English sub-index. Every model gets the same cases and the same request, under the benchmark rules.