system-one-bench

~/system-one-bench / benchmarks/ag-news

Text track · English · All benchmarks

AG News

News topic, 4 options. Laya Router has the top score, 0.5 points ahead of Mercury Decide, but Mercury Decide and 28 more are within sampling noise: a statistical tie for first. But its score is unusually high for Laya Router: 92.5%, 13 points above the 80% its other datasets predict.

Top score
Laya Router
92.5, tied with 29 within noise
Lead over 2nd
0.5
Points of accuracy
Models
39
Charted on the leaderboard
Cases
200
Best constant answer
26.5
Always giving the most common label
Random guess
25.0
Uniform over the options

Leaderboard

RankModelAccuracy95% intervalMacro-F1Calibration errorFailed
1Laya Router Convai InnovationsLeaderunusually high92.588.8–96.2
92.32.8%0
2Mercury Decide InceptionTied for 1st92.088.2–95.8
91.95.6%0
2Mammoth 1 BasetenTied for 1st92.088.2–95.8
91.85.8%0
2GLiDE FastinoTied for 1st92.088.2–95.8
91.96.2%0
5Tev1 4B Experimental Together AITied for 1st91.587.6–95.4
91.34.0%0
5Decider V1.2 PerplexityTied for 1st91.587.6–95.4
91.34.9%0
5Dexy-Decision-9B Patrick-18Tied for 1st91.587.6–95.4
91.45.0%0
8Kev 0.8B Jared PalmerTied for 1stin-domain91.087.0–95.0
90.84.0%0
8d1-ultra Liquid AITied for 1st91.087.0–95.0
90.83.2%0
10Clef CloudflareTied for 1st90.586.4–94.6
90.35.7%0
10Kev 4B Jared PalmerTied for 1stin-domain90.586.4–94.6
90.33.4%0
10Drex Max Nace AITied for 1st90.586.4–94.6
90.33.9%0
10d1.1 Liquid AITied for 1st90.586.4–94.6
90.35.6%0
10Celeris-1 Decision CelerisTied for 1st90.586.4–94.6
90.33.5%0
15d1 Liquid AITied for 1st90.085.8–94.2
89.76.1%0
15Clef Flash CloudflareTied for 1st90.085.8–94.2
89.84.0%0
15Drex v1.5 Nace AITied for 1st90.085.8–94.2
89.77.0%0
18Jev 1.14 TypeSafeTied for 1st89.084.7–93.3
88.76.9%0
18Microsoft-Decision-1 MicrosoftTied for 1st89.084.7–93.3
88.76.7%0
18Clef Omni CloudflareTied for 1st89.084.7–93.3
88.73.4%0
18Decider Flash V1.2 PerplexityTied for 1st89.084.7–93.3
88.73.1%0
18Elephant 1 BasetenTied for 1st89.084.7–93.3
88.65.1%0
23Jev 1.13 TypeSafeTied for 1st88.584.1–92.9
88.18.6%0
23d1-3B Liquid AITied for 1st88.584.1–92.9
88.27.1%0
23Gemini Decision 1 GoogleTied for 1st88.584.1–92.9
88.15.3%0
26d3 nano vLLM Semantic RouterTied for 1st88.083.5–92.5
87.810.1%0
26Drex v2.0 Nace AITied for 1st88.083.5–92.5
87.64.3%0
28GPT-6 Luna Decisions OpenAITied for 1st87.582.9–92.1
87.011.2%0
28d1-omni-600M Liquid AITied for 1st87.582.9–92.1
87.16.5%0
28Decider 1 meraGPTTied for 1st87.582.9–92.1
87.13.4%0
31Span-02 Respan86.581.8–91.2
86.09.9%0
32Solar Decide Upstage86.081.2–90.8
85.512.6%0
32Solar Decide Flash Upstage86.081.2–90.8
85.510.2%0
34d3 lite vLLM Semantic Router85.580.6–90.4
85.110.1%0
35Joey 1 Baseten85.080.1–89.9
84.54.9%0
36d3 edge vLLM Semantic Router80.074.5–85.5
79.09.9%0
37Naluri-308M Jericho Siahaya79.573.9–85.1
79.18.3%0
38Naluri-279M Jericho Siahaya77.571.7–83.3
74.29.9%0
39Classify-1 ScaleDown26.520.4–32.6
10.51.5%200

The bar is accuracy; the bracket on it is the 95% interval for 200 cases. Tied for 1st marks every model whose gap to the leader is within sampling noise (1.96 standard errors of the difference, treating the runs as independent). A failed call counts as wrong. In-domain models trained on another split of this dataset, by their maker's own account. Unusually high marks a score far above what that model's other datasets, and every other model's score here, predict (hover for the numbers). It often means the model trained on this dataset or one like it; it isn't proof, and the score still counts.

About this benchmark

Question
News topic, 4 options
Source
fancyzhx/ag_news, test split
Sample
200 random articles
Pinned
Every source is pinned to an exact commit, and the cases are frozen by hash in data.lock.json.

It counts towards the System One Index and its English sub-index. Every model gets the same cases and the same request, under the benchmark rules.