system-one-bench

~/system-one-bench / benchmarks/arabic-miracl-pairwise

Text track · Arabic · All benchmarks

MIRACL pairwise

Is this passage relevant, yes/no. Decider V1.2 and Span-02 share the top score, and 27 more are within sampling noise.

Top score
Decider V1.2
91.5, tied with 28 within noise
Lead over 2nd
0.0
Points of accuracy
Models
41
Charted on the leaderboard
Cases
200
Best constant answer
50.0
Always giving the most common label
Random guess
50.0
Uniform over the options

Leaderboard

RankModelAccuracy95% intervalMacro-F1Calibration errorFailed
1Decider V1.2 PerplexityLeader91.587.6–95.4
91.55.5%0
1Span-02 RespanTied for 1st91.587.6–95.4
91.54.7%0
3d1 Liquid AITied for 1st91.087.0–95.0
91.03.5%0
3Microsoft-Decision-1 MicrosoftTied for 1st91.087.0–95.0
91.01.6%0
3d1-ultra Liquid AITied for 1st91.087.0–95.0
91.02.1%0
3Elephant 1 BasetenTied for 1st91.087.0–95.0
91.01.6%0
3Dexy-Decision-9B Patrick-18Tied for 1st91.087.0–95.0
91.04.8%0
3Gemini Decision 1 GoogleTied for 1st91.087.0–95.0
91.03.2%0
9Jev 1.13 TypeSafeTied for 1st90.586.4–94.6
90.52.7%0
9Decider Flash V1.2 PerplexityTied for 1st90.586.4–94.6
90.53.1%0
9GLiDE FastinoTied for 1st90.586.4–94.6
90.52.4%0
12Jev 1.14 TypeSafeTied for 1st90.085.8–94.2
90.02.1%0
12Clef Flash CloudflareTied for 1st90.085.8–94.2
90.02.7%0
12Mercury Decide InceptionTied for 1st90.085.8–94.2
90.07.6%0
12Mammoth 1 BasetenTied for 1st90.085.8–94.2
90.04.2%0
12Celeris-1 Decision CelerisTied for 1st90.085.8–94.2
90.04.0%0
17Drex Max Nace AITied for 1st89.585.3–93.7
89.53.3%0
17d1.1 Liquid AITied for 1st89.585.3–93.7
89.53.8%0
19Clef CloudflareTied for 1st89.084.7–93.3
89.06.4%0
19d1-3B Liquid AITied for 1st89.084.7–93.3
89.05.2%0
19Drex v2.0 Nace AITied for 1st89.084.7–93.3
89.02.9%0
22GPT-6 Luna Decisions OpenAITied for 1st87.582.9–92.1
87.510.7%0
22Tev1 4B Experimental Together AITied for 1st87.582.9–92.1
87.53.6%0
24laya-ara-rag WouzeTied for 1stin-domain86.581.8–91.2
86.43.7%0
24Kev 4B Jared PalmerTied for 1st86.581.8–91.2
86.54.3%0
24Clef Omni CloudflareTied for 1st86.581.8–91.2
86.510.0%0
24d3 nano vLLM Semantic RouterTied for 1st86.581.8–91.2
86.54.4%0
28Drex v1.5 Nace AITied for 1st86.081.2–90.8
85.99.1%0
29Solar Decide UpstageTied for 1st85.580.6–90.4
85.510.6%0
30d3 lite vLLM Semantic Router84.579.5–89.5
84.55.7%0
31Solar Decide Flash Upstage83.077.8–88.2
83.07.2%0
31Joey 1 Baseten83.077.8–88.2
83.010.3%0
33Laya Router Convai Innovations82.076.7–87.3
82.05.0%0
34Decider 1 meraGPT78.072.3–83.7
77.95.9%0
35d3 edge vLLM Semantic Router76.070.1–81.9
75.65.0%0
35Naluri-308M Jericho Siahaya76.070.1–81.9
75.52.4%0
37laya-ara Wouze74.568.5–80.5
74.09.2%0
38Kev 0.8B Jared Palmer67.060.5–73.5
66.94.4%0
39d1-omni-600M Liquid AI66.560.0–73.0
65.27.1%0
40Naluri-279M Jericho Siahaya50.543.6–57.4
35.342.8%0
41Classify-1 ScaleDown50.043.1–56.9
33.30.0%200

The bar is accuracy; the bracket on it is the 95% interval for 200 cases. Tied for 1st marks every model whose gap to the leader is within sampling noise (1.96 standard errors of the difference, treating the runs as independent). A failed call counts as wrong. In-domain models trained on another split of this dataset, by their maker's own account.

About this benchmark

Question
Is this passage relevant, yes/no
Source
mteb/MIRACLReranking, dev split
Sample
200 other Arabic queries, half with a relevant passage
Pinned
Every source is pinned to an exact commit, and the cases are frozen by hash in data.lock.json.

It counts towards the System One Index and its Arabic sub-index. Every model gets the same cases and the same request, under the benchmark rules.