system-one-bench

~/system-one-bench / benchmarks/boolq

Text track · English · All benchmarks

BoolQ

Reading comprehension, yes/no. Drex Max has the top score, 0.5 points ahead of Drex v2.0, but Drex v2.0 and 23 more are within sampling noise: a statistical tie for first.

Top score
Drex Max
93.0, tied with 24 within noise
Lead over 2nd
0.5
Points of accuracy
Models
39
Charted on the leaderboard
Cases
200
Best constant answer
62.0
Always giving the most common label
Random guess
50.0
Uniform over the options

Leaderboard

RankModelAccuracy95% intervalMacro-F1Calibration errorFailed
1Drex Max Nace AILeader93.089.5–96.5
92.51.9%0
2Drex v2.0 Nace AITied for 1st92.588.8–96.2
92.13.3%0
3d1.1 Liquid AITied for 1st92.088.2–95.8
91.52.6%0
3Mammoth 1 BasetenTied for 1st92.088.2–95.8
91.44.0%0
3GLiDE FastinoTied for 1st92.088.2–95.8
91.62.8%0
6Kev 4B Jared PalmerTied for 1stin-domain91.587.6–95.4
90.92.1%0
6Decider V1.2 PerplexityTied for 1st91.587.6–95.4
91.02.4%0
6d1-ultra Liquid AITied for 1st91.587.6–95.4
91.03.2%0
9Clef CloudflareTied for 1st91.087.0–95.0
90.56.1%0
9Tev1 4B Experimental Together AITied for 1st91.087.0–95.0
90.75.8%0
9Elephant 1 BasetenTied for 1st91.087.0–95.0
90.44.4%0
12Mercury Decide InceptionTied for 1st90.586.4–94.6
90.15.3%0
13Jev 1.14 TypeSafeTied for 1st90.085.8–94.2
89.62.7%0
13d1 Liquid AITied for 1st90.085.8–94.2
89.44.2%0
13Clef Omni CloudflareTied for 1st90.085.8–94.2
89.47.2%0
13Span-02 RespanTied for 1st90.085.8–94.2
89.46.1%0
17Jev 1.13 TypeSafeTied for 1st89.585.3–93.7
89.13.9%0
17Celeris-1 Decision CelerisTied for 1st89.585.3–93.7
88.83.1%0
17Dexy-Decision-9B Patrick-18Tied for 1st89.585.3–93.7
88.95.4%0
20Drex v1.5 Nace AITied for 1st89.084.7–93.3
88.36.6%0
20Decider Flash V1.2 PerplexityTied for 1st89.084.7–93.3
88.33.2%0
20Gemini Decision 1 GoogleTied for 1st89.084.7–93.3
88.53.6%0
23Clef Flash CloudflareTied for 1st88.584.1–92.9
87.96.6%0
23Microsoft-Decision-1 MicrosoftTied for 1st88.584.1–92.9
87.96.7%0
25d3 nano vLLM Semantic RouterTied for 1st88.083.5–92.5
87.44.5%0
26Solar Decide Upstage87.082.3–91.7
86.59.9%0
27Solar Decide Flash Upstage86.081.2–90.8
85.47.5%0
28GPT-6 Luna Decisions OpenAI85.580.6–90.4
84.88.6%0
29d1-3B Liquid AI85.080.1–89.9
84.16.1%0
30Laya Router Convai Innovations81.576.1–86.9
80.09.3%0
31d3 lite vLLM Semantic Router81.075.6–86.4
79.78.5%0
32Joey 1 Baseten80.074.5–85.5
78.44.7%0
32Decider 1 meraGPT80.074.5–85.5
78.98.6%0
34d3 edge vLLM Semantic Router79.573.9–85.1
78.55.8%0
35Kev 0.8B Jared Palmerin-domain76.070.1–81.9
72.912.0%0
36d1-omni-600M Liquid AI74.067.9–80.1
73.113.1%0
37Naluri-308M Jericho Siahaya56.549.6–63.4
55.319.1%0
38Naluri-279M Jericho Siahaya54.547.6–61.4
47.513.6%0
39Classify-1 ScaleDown38.031.3–44.7
27.512.0%200

The bar is accuracy; the bracket on it is the 95% interval for 200 cases. Tied for 1st marks every model whose gap to the leader is within sampling noise (1.96 standard errors of the difference, treating the runs as independent). A failed call counts as wrong. In-domain models trained on another split of this dataset, by their maker's own account.

About this benchmark

Question
Reading comprehension, yes/no
Source
google/boolq, validation split
Sample
200 random passage and question pairs
Pinned
Every source is pinned to an exact commit, and the cases are frozen by hash in data.lock.json.

It counts towards the System One Index and its English sub-index. Every model gets the same cases and the same request, under the benchmark rules.