system-one-bench

~/system-one-bench / benchmarks/dair-emotion

Text track · English · All benchmarks

DAIR Emotion

Emotion, 6 options. Mercury Decide leads by 6.5 points over d1-omni-600M, more than sampling noise. But its score is unusually high for Mercury Decide: 91.5%, 30 points above the 62% its other datasets predict.

Leader
Mercury Decide
91.5
Lead over 2nd
6.5
Points of accuracy
Models
39
Charted on the leaderboard
Cases
200
Best constant answer
30.0
Always giving the most common label
Random guess
16.7
Uniform over the options

Leaderboard

RankModelAccuracy95% intervalMacro-F1Calibration errorFailed
1Mercury Decide InceptionLeaderunusually high91.587.6–95.4
87.04.9%0
2d1-omni-600M Liquid AIunusually high85.080.1–89.9
79.211.6%0
3Naluri-308M Jericho Siahayain-domainunusually high84.078.9–89.1
78.815.5%0
4Celeris-1 Decision Celeris72.566.3–78.7
64.77.0%0
5Naluri-279M Jericho Siahayain-domainunusually high68.562.1–74.9
61.820.9%0
6Decider 1 meraGPT67.561.0–74.0
60.73.9%0
7Drex Max Nace AI65.058.4–71.6
55.98.2%0
8d1 Liquid AI63.556.8–70.2
56.311.9%0
8d1-3B Liquid AI63.556.8–70.2
52.913.8%0
10d1-ultra Liquid AI63.056.3–69.7
53.814.2%0
11GLiDE Fastino61.554.8–68.2
54.611.3%0
12d1.1 Liquid AI60.553.7–67.3
51.711.9%0
13Decider V1.2 Perplexity60.053.2–66.8
51.88.5%0
14d3 nano vLLM Semantic Router59.552.7–66.3
50.816.4%0
14Gemini Decision 1 Google59.552.7–66.3
52.215.1%1
16Classify-1 ScaleDown59.052.2–65.8
52.67.8%0
16Mammoth 1 Baseten59.052.2–65.8
52.417.0%0
18GPT-6 Luna Decisions OpenAI58.551.7–65.3
49.426.1%0
18Clef Cloudflare58.551.7–65.3
48.918.3%0
20Jev 1.14 TypeSafe57.550.6–64.4
50.227.0%0
21Clef Flash Cloudflare57.050.1–63.9
50.922.3%0
21Microsoft-Decision-1 Microsoft57.050.1–63.9
49.416.4%0
21d3 lite vLLM Semantic Router57.050.1–63.9
49.818.2%0
21d3 edge vLLM Semantic Router57.050.1–63.9
51.820.8%0
21Decider Flash V1.2 Perplexity57.050.1–63.9
50.414.1%0
26Tev1 4B Experimental Together AI56.549.6–63.4
47.120.6%0
27Jev 1.13 TypeSafe56.049.1–62.9
47.630.4%0
27Clef Omni Cloudflare56.049.1–62.9
46.921.0%0
27Kev 0.8B Jared Palmer56.049.1–62.9
48.98.6%0
27Span-02 Respan56.049.1–62.9
46.531.3%0
31Drex v2.0 Nace AI55.548.6–62.4
46.515.2%0
32Kev 4B Jared Palmer54.547.6–61.4
47.712.3%0
32Solar Decide Flash Upstage54.547.6–61.4
47.332.2%0
32Elephant 1 Baseten54.547.6–61.4
48.622.0%0
35Solar Decide Upstage54.047.1–60.9
46.740.5%0
35Dexy-Decision-9B Patrick-1854.047.1–60.9
44.116.1%0
37Laya Router Convai Innovations53.546.6–60.4
41.035.1%0
37Drex v1.5 Nace AI53.546.6–60.4
45.324.2%0
39Joey 1 Baseten52.545.6–59.4
44.917.0%0

The bar is accuracy; the bracket on it is the 95% interval for 200 cases. Tied for 1st marks every model whose gap to the leader is within sampling noise (1.96 standard errors of the difference, treating the runs as independent). A failed call counts as wrong. In-domain models trained on another split of this dataset, by their maker's own account. Unusually high marks a score far above what that model's other datasets, and every other model's score here, predict (hover for the numbers). It often means the model trained on this dataset or one like it; it isn't proof, and the score still counts.

About this benchmark

Question
Emotion, 6 options
Source
dair-ai/emotion, test split
Sample
200 random texts
Pinned
Every source is pinned to an exact commit, and the cases are frozen by hash in data.lock.json.

It counts towards the System One Index and its English sub-index. Every model gets the same cases and the same request, under the benchmark rules.