~/system-one-bench / benchmarks/dair-emotion
Text track · English · All benchmarks
Emotion, 6 options. Mercury Decide leads by 6.5 points over d1-omni-600M, more than sampling noise. But its score is unusually high for Mercury Decide: 91.5%, 30 points above the 62% its other datasets predict.
| Rank | Model | Accuracy | 95% interval | Macro-F1 | Calibration error | Failed | |
|---|---|---|---|---|---|---|---|
| 1 | Mercury Decide InceptionLeaderunusually high | 91.5 | 87.6–95.4 | 87.0 | 4.9% | 0 | |
| 2 | d1-omni-600M Liquid AIunusually high | 85.0 | 80.1–89.9 | 79.2 | 11.6% | 0 | |
| 3 | Naluri-308M Jericho Siahayain-domainunusually high | 84.0 | 78.9–89.1 | 78.8 | 15.5% | 0 | |
| 4 | Celeris-1 Decision Celeris | 72.5 | 66.3–78.7 | 64.7 | 7.0% | 0 | |
| 5 | Naluri-279M Jericho Siahayain-domainunusually high | 68.5 | 62.1–74.9 | 61.8 | 20.9% | 0 | |
| 6 | 67.5 | 61.0–74.0 | 60.7 | 3.9% | 0 | ||
| 7 | 65.0 | 58.4–71.6 | 55.9 | 8.2% | 0 | ||
| 8 | d1 Liquid AI | 63.5 | 56.8–70.2 | 56.3 | 11.9% | 0 | |
| 8 | d1-3B Liquid AI | 63.5 | 56.8–70.2 | 52.9 | 13.8% | 0 | |
| 10 | d1-ultra Liquid AI | 63.0 | 56.3–69.7 | 53.8 | 14.2% | 0 | |
| 11 | GLiDE Fastino | 61.5 | 54.8–68.2 | 54.6 | 11.3% | 0 | |
| 12 | d1.1 Liquid AI | 60.5 | 53.7–67.3 | 51.7 | 11.9% | 0 | |
| 13 | 60.0 | 53.2–66.8 | 51.8 | 8.5% | 0 | ||
| 14 | 59.5 | 52.7–66.3 | 50.8 | 16.4% | 0 | ||
| 14 | 59.5 | 52.7–66.3 | 52.2 | 15.1% | 1 | ||
| 16 | Classify-1 ScaleDown | 59.0 | 52.2–65.8 | 52.6 | 7.8% | 0 | |
| 16 | Mammoth 1 Baseten | 59.0 | 52.2–65.8 | 52.4 | 17.0% | 0 | |
| 18 | 58.5 | 51.7–65.3 | 49.4 | 26.1% | 0 | ||
| 18 | 58.5 | 51.7–65.3 | 48.9 | 18.3% | 0 | ||
| 20 | Jev 1.14 TypeSafe | 57.5 | 50.6–64.4 | 50.2 | 27.0% | 0 | |
| 21 | 57.0 | 50.1–63.9 | 50.9 | 22.3% | 0 | ||
| 21 | 57.0 | 50.1–63.9 | 49.4 | 16.4% | 0 | ||
| 21 | 57.0 | 50.1–63.9 | 49.8 | 18.2% | 0 | ||
| 21 | 57.0 | 50.1–63.9 | 51.8 | 20.8% | 0 | ||
| 21 | 57.0 | 50.1–63.9 | 50.4 | 14.1% | 0 | ||
| 26 | 56.5 | 49.6–63.4 | 47.1 | 20.6% | 0 | ||
| 27 | Jev 1.13 TypeSafe | 56.0 | 49.1–62.9 | 47.6 | 30.4% | 0 | |
| 27 | 56.0 | 49.1–62.9 | 46.9 | 21.0% | 0 | ||
| 27 | Kev 0.8B Jared Palmer | 56.0 | 49.1–62.9 | 48.9 | 8.6% | 0 | |
| 27 | Span-02 Respan | 56.0 | 49.1–62.9 | 46.5 | 31.3% | 0 | |
| 31 | 55.5 | 48.6–62.4 | 46.5 | 15.2% | 0 | ||
| 32 | Kev 4B Jared Palmer | 54.5 | 47.6–61.4 | 47.7 | 12.3% | 0 | |
| 32 | 54.5 | 47.6–61.4 | 47.3 | 32.2% | 0 | ||
| 32 | Elephant 1 Baseten | 54.5 | 47.6–61.4 | 48.6 | 22.0% | 0 | |
| 35 | 54.0 | 47.1–60.9 | 46.7 | 40.5% | 0 | ||
| 35 | Dexy-Decision-9B Patrick-18 | 54.0 | 47.1–60.9 | 44.1 | 16.1% | 0 | |
| 37 | Laya Router Convai Innovations | 53.5 | 46.6–60.4 | 41.0 | 35.1% | 0 | |
| 37 | 53.5 | 46.6–60.4 | 45.3 | 24.2% | 0 | ||
| 39 | Joey 1 Baseten | 52.5 | 45.6–59.4 | 44.9 | 17.0% | 0 |
The bar is accuracy; the bracket on it is the 95% interval for 200 cases. Tied for 1st marks every model whose gap to the leader is within sampling noise (1.96 standard errors of the difference, treating the runs as independent). A failed call counts as wrong. In-domain models trained on another split of this dataset, by their maker's own account. Unusually high marks a score far above what that model's other datasets, and every other model's score here, predict (hover for the numbers). It often means the model trained on this dataset or one like it; it isn't proof, and the score still counts.
It counts towards the System One Index and its English sub-index. Every model gets the same cases and the same request, under the benchmark rules.