system-one-bench

~/system-one-bench / benchmarks/arabic-massive-scenario

Text track · Arabic · All benchmarks

MASSIVE scenario

Assistant request scenario, 18 options. Clef Flash and Dexy-Decision-9B share the top score, and 3 more are within sampling noise. But its score is unusually high for Clef Flash: 89.5%, 20 points above the 69% its other datasets predict.

Top score
Clef Flash
89.5, tied with 4 within noise
Lead over 2nd
0.0
Points of accuracy
Models
41
Charted on the leaderboard
Cases
200
Best constant answer
17.5
Always giving the most common label
Random guess
5.6
Uniform over the options

Leaderboard

RankModelAccuracy95% intervalMacro-F1Calibration errorFailed
1Clef Flash CloudflareLeaderunusually high89.585.3–93.7
82.64.9%0
1Dexy-Decision-9B Patrick-18Tied for 1stin-domainunusually high89.585.3–93.7
88.05.2%0
3Clef Omni CloudflareTied for 1stunusually high89.084.7–93.3
84.86.3%0
4Clef CloudflareTied for 1stunusually high88.083.5–92.5
83.17.5%0
5Microsoft-Decision-1 MicrosoftTied for 1st83.578.4–88.6
77.37.0%0
6Decider V1.2 Perplexity79.073.4–84.6
74.25.3%0
7d1-ultra Liquid AI76.070.1–81.9
74.96.9%0
8Mammoth 1 Baseten75.069.0–81.0
77.811.9%0
9GLiDE Fastino73.567.4–79.6
68.113.6%0
9Celeris-1 Decision Celeris73.567.4–79.6
66.06.3%0
11d1 Liquid AI73.066.8–79.2
68.78.2%0
11Drex Max Nace AI73.066.8–79.2
72.011.8%0
11d1.1 Liquid AI73.066.8–79.2
72.710.0%0
14Gemini Decision 1 Google72.566.3–78.7
73.410.1%0
15Mercury Decide Inception72.065.8–78.2
74.416.5%0
16d1-3B Liquid AI71.565.2–77.8
62.47.7%0
16Joey 1 Baseten71.565.2–77.8
66.99.6%0
18Span-02 Respan71.064.7–77.3
71.813.3%0
19Decider Flash V1.2 Perplexity70.564.2–76.8
64.18.3%0
20laya-ara Wouzein-domain69.563.1–75.9
64.621.8%0
21Elephant 1 Baseten69.062.6–75.4
64.114.5%0
22GPT-6 Luna Decisions OpenAI66.059.4–72.6
64.316.8%0
23Jev 1.14 TypeSafe65.058.4–71.6
59.816.3%0
24Naluri-308M Jericho Siahayain-domain63.556.8–70.2
55.311.1%0
24Drex v2.0 Nace AI63.556.8–70.2
61.29.7%0
26Jev 1.13 TypeSafe63.056.3–69.7
57.321.0%0
27d3 nano vLLM Semantic Router61.554.8–68.2
56.016.2%0
28laya-ara-rag Wouzein-domain60.553.7–67.3
55.622.3%0
29Drex v1.5 Nace AI58.551.7–65.3
54.125.0%0
30d3 lite vLLM Semantic Router58.051.2–64.8
53.510.4%0
30Naluri-279M Jericho Siahayain-domainunusually high58.051.2–64.8
52.510.1%0
32Kev 4B Jared Palmer57.050.1–63.9
54.712.9%0
33Tev1 4B Experimental Together AI56.549.6–63.4
56.113.9%0
34Solar Decide Upstage55.548.6–62.4
47.637.2%0
35Solar Decide Flash Upstage46.039.1–52.9
40.432.7%0
36d3 edge vLLM Semantic Router41.034.2–47.8
30.821.1%0
37Laya Router Convai Innovations39.032.2–45.8
37.340.6%0
38Kev 0.8B Jared Palmer38.531.8–45.2
39.512.8%0
39d1-omni-600M Liquid AI37.530.8–44.2
30.729.1%0
40Classify-1 ScaleDown3.00.6–5.4
0.32.6%200
40Decider 1 meraGPT3.00.6–5.4
0.32.6%200

The bar is accuracy; the bracket on it is the 95% interval for 200 cases. Tied for 1st marks every model whose gap to the leader is within sampling noise (1.96 standard errors of the difference, treating the runs as independent). A failed call counts as wrong. In-domain models trained on another split of this dataset, by their maker's own account. Unusually high marks a score far above what that model's other datasets, and every other model's score here, predict (hover for the numbers). It often means the model trained on this dataset or one like it; it isn't proof, and the score still counts.

About this benchmark

Question
Assistant request scenario, 18 options
Source
SetFit/amazon_massive_scenario_ar-SA, test split
Sample
200 random requests, all 18 scenarios as options
Pinned
Every source is pinned to an exact commit, and the cases are frozen by hash in data.lock.json.

It counts towards the System One Index and its Arabic sub-index. Every model gets the same cases and the same request, under the benchmark rules.