system-one-bench

~/system-one-bench / benchmarks/arabic-massive-intent

Text track · Arabic · All benchmarks

MASSIVE intent

Assistant request intent, 20 options. Clef Omni and Dexy-Decision-9B share the top score, and 15 more are within sampling noise.

Top score
Clef Omni
91.5, tied with 16 within noise
Lead over 2nd
0.0
Points of accuracy
Models
41
Charted on the leaderboard
Cases
200
Best constant answer
11.0
Always giving the most common label
Random guess
5.0
Uniform over the options

Leaderboard

RankModelAccuracy95% intervalMacro-F1Calibration errorFailed
1Clef Omni CloudflareLeader91.587.6–95.4
91.77.2%0
1Dexy-Decision-9B Patrick-18Tied for 1stin-domain91.587.6–95.4
91.45.2%0
3Decider V1.2 PerplexityTied for 1st91.087.0–95.0
91.14.9%0
3Mammoth 1 BasetenTied for 1st91.087.0–95.0
90.73.6%0
3GLiDE FastinoTied for 1st91.087.0–95.0
91.12.9%0
6Clef Flash CloudflareTied for 1st90.586.4–94.6
90.84.4%0
7d1 Liquid AITied for 1st90.085.8–94.2
90.64.1%0
8Clef CloudflareTied for 1st89.585.3–93.7
89.85.8%0
8Elephant 1 BasetenTied for 1st89.585.3–93.7
89.63.9%0
10Microsoft-Decision-1 MicrosoftTied for 1st89.084.7–93.3
89.73.5%0
11Drex Max Nace AITied for 1st88.083.5–92.5
88.45.1%0
11d1.1 Liquid AITied for 1st88.083.5–92.5
88.44.9%0
11Celeris-1 Decision CelerisTied for 1st88.083.5–92.5
88.16.7%0
14d1-ultra Liquid AITied for 1st87.082.3–91.7
87.06.2%0
14Gemini Decision 1 GoogleTied for 1st87.082.3–91.7
87.24.2%0
16Decider Flash V1.2 PerplexityTied for 1st86.081.2–90.8
85.84.1%0
16Span-02 RespanTied for 1st86.081.2–90.8
86.72.7%0
18Jev 1.14 TypeSafe85.080.1–89.9
85.42.8%0
18Mercury Decide Inception85.080.1–89.9
85.57.8%0
20GPT-6 Luna Decisions OpenAI84.579.5–89.5
84.45.1%1
20Joey 1 Baseten84.579.5–89.5
84.93.4%0
22Jev 1.13 TypeSafe84.078.9–89.1
83.63.5%0
23d1-3B Liquid AI83.578.4–88.6
84.46.1%0
24Drex v2.0 Nace AI82.577.2–87.8
82.58.1%0
25d3 nano vLLM Semantic Router81.075.6–86.4
81.28.0%0
26laya-ara Wouzein-domain77.571.7–83.3
77.316.1%0
26Drex v1.5 Nace AI77.571.7–83.3
78.111.5%0
26d3 lite vLLM Semantic Router77.571.7–83.3
77.03.4%0
29Kev 4B Jared Palmer76.570.6–82.4
76.422.5%0
30Solar Decide Upstage75.069.0–81.0
74.718.9%0
31Tev1 4B Experimental Together AI70.063.6–76.4
68.69.6%0
32Naluri-308M Jericho Siahayain-domain65.558.9–72.1
64.532.2%0
33Solar Decide Flash Upstage63.056.3–69.7
63.218.3%0
34laya-ara-rag Wouzein-domain55.048.1–61.9
54.223.5%0
34d3 edge vLLM Semantic Router55.048.1–61.9
54.910.6%0
36d1-omni-600M Liquid AI52.045.1–58.9
51.725.1%0
36Naluri-279M Jericho Siahayain-domain52.045.1–58.9
52.112.6%0
38Kev 0.8B Jared Palmer51.544.6–58.4
50.48.4%0
39Laya Router Convai Innovations32.526.0–39.0
32.242.4%0
40Classify-1 ScaleDown3.51.0–6.0
0.31.5%200
40Decider 1 meraGPT3.51.0–6.0
0.31.5%200

The bar is accuracy; the bracket on it is the 95% interval for 200 cases. Tied for 1st marks every model whose gap to the leader is within sampling noise (1.96 standard errors of the difference, treating the runs as independent). A failed call counts as wrong. In-domain models trained on another split of this dataset, by their maker's own account.

About this benchmark

Question
Assistant request intent, 20 options
Source
SetFit/amazon_massive_intent_ar-SA, test split
Sample
200 random requests, the right intent and 19 random others as options
Pinned
Every source is pinned to an exact commit, and the cases are frozen by hash in data.lock.json.

It counts towards the System One Index and its Arabic sub-index. Every model gets the same cases and the same request, under the benchmark rules.