system-one-bench

~/system-one-bench / benchmarks/arabic-miracl-rerank

Text track · Arabic · All benchmarks

MIRACL rerank

Pick the relevant passage out of 8. d1 has the top score, 0.5 points ahead of Microsoft-Decision-1, but Microsoft-Decision-1 and 22 more are within sampling noise: a statistical tie for first.

Top score
d1
90.5, tied with 23 within noise
Lead over 2nd
0.5
Points of accuracy
Models
41
Charted on the leaderboard
Cases
200
Best constant answer
15.5
Always giving the most common label
Random guess
12.5
Uniform over the options

Leaderboard

RankModelAccuracy95% intervalMacro-F1Calibration errorFailed
1d1 Liquid AILeader90.586.4–94.6
90.23.0%0
2Microsoft-Decision-1 MicrosoftTied for 1st90.085.8–94.2
89.74.7%0
3Drex Max Nace AITied for 1st89.585.3–93.7
89.54.5%0
3Gemini Decision 1 GoogleTied for 1st89.585.3–93.7
89.34.9%0
5Clef CloudflareTied for 1st89.084.7–93.3
88.913.4%0
5Drex v1.5 Nace AITied for 1st89.084.7–93.3
88.95.3%0
5d1-ultra Liquid AITied for 1st89.084.7–93.3
88.84.6%0
5Mammoth 1 BasetenTied for 1st89.084.7–93.3
88.86.2%0
9Decider Flash V1.2 PerplexityTied for 1st88.584.1–92.9
88.14.4%0
9Span-02 RespanTied for 1st88.584.1–92.9
88.37.9%0
9d1.1 Liquid AITied for 1st88.584.1–92.9
88.12.2%0
12Clef Flash CloudflareTied for 1st88.083.5–92.5
88.212.1%0
12Mercury Decide InceptionTied for 1st88.083.5–92.5
88.08.2%0
12Elephant 1 BasetenTied for 1st88.083.5–92.5
88.05.0%0
12Dexy-Decision-9B Patrick-18Tied for 1st88.083.5–92.5
87.65.4%0
16Jev 1.14 TypeSafeTied for 1st87.582.9–92.1
87.16.6%0
17Drex v2.0 Nace AITied for 1st87.082.3–91.7
86.64.5%0
17GLiDE FastinoTied for 1st87.082.3–91.7
86.85.4%0
19Jev 1.13 TypeSafeTied for 1st86.581.8–91.2
85.86.1%0
19Decider V1.2 PerplexityTied for 1st86.581.8–91.2
86.44.7%0
19Celeris-1 Decision CelerisTied for 1st86.581.8–91.2
86.23.0%0
22GPT-6 Luna Decisions OpenAITied for 1st86.081.2–90.8
85.64.5%1
23Tev1 4B Experimental Together AITied for 1st85.580.6–90.4
85.38.7%0
24Solar Decide UpstageTied for 1st84.579.5–89.5
84.211.0%0
25Clef Omni Cloudflare83.077.8–88.2
82.617.7%0
26d3 lite vLLM Semantic Router82.577.2–87.8
82.34.3%0
26d1-3B Liquid AI82.577.2–87.8
82.16.5%0
28Decider 1 meraGPT81.075.6–86.4
80.913.5%0
29Kev 4B Jared Palmer79.573.9–85.1
79.613.0%0
30d3 nano vLLM Semantic Router77.571.7–83.3
77.38.4%0
31laya-ara-rag Wouzein-domain71.565.2–77.8
71.411.3%0
31Solar Decide Flash Upstage71.565.2–77.8
71.75.7%0
33d3 edge vLLM Semantic Router70.063.6–76.4
69.25.6%0
34Kev 0.8B Jared Palmer62.555.8–69.2
62.124.8%0
35Joey 1 Baseten62.055.3–68.7
62.17.3%0
36d1-omni-600M Liquid AI55.048.1–61.9
55.010.7%0
37Naluri-308M Jericho Siahaya34.027.4–40.6
32.58.4%0
38laya-ara Wouze32.025.5–38.5
32.326.4%0
39Laya Router Convai Innovations31.525.1–37.9
30.918.9%0
40Naluri-279M Jericho Siahaya28.021.8–34.2
24.410.4%0
41Classify-1 ScaleDown12.07.5–16.5
2.70.5%200

The bar is accuracy; the bracket on it is the 95% interval for 200 cases. Tied for 1st marks every model whose gap to the leader is within sampling noise (1.96 standard errors of the difference, treating the runs as independent). A failed call counts as wrong. In-domain models trained on another split of this dataset, by their maker's own account.

About this benchmark

Question
Pick the relevant passage out of 8
Source
mteb/MIRACLReranking, dev split
Sample
200 Arabic queries, 1 relevant passage and 7 not, shuffled
Pinned
Every source is pinned to an exact commit, and the cases are frozen by hash in data.lock.json.

It counts towards the System One Index and its Arabic sub-index. Every model gets the same cases and the same request, under the benchmark rules.