system-one-bench

~/system-one-bench / benchmarks/typed-decisions

Text track · English · All benchmarks

Typed decisions

Agent and product states, 5 questions per case. Dexy-Decision-9B has the top score, 1.0 points ahead of Microsoft-Decision-1, but Microsoft-Decision-1 and 8 more are within sampling noise: a statistical tie for first.

Top score
Dexy-Decision-9B
80.0, tied with 9 within noise
Lead over 2nd
1.0
Points of accuracy
Models
39
Charted on the leaderboard
Questions
2,000
400 cases
Best constant answer
46.2
Always giving the most common label
Random guess
31.8
Uniform over the options

Leaderboard

RankModelAccuracy95% intervalMacro-F1Calibration errorFailed
1Dexy-Decision-9B Patrick-18Leaderin-domain80.076.1–83.9
78.811.3%0
2Microsoft-Decision-1 MicrosoftTied for 1st79.075.0–83.0
77.92.6%0
3Celeris-1 Decision CelerisTied for 1st76.472.3–80.6
74.74.8%0
4Mammoth 1 BasetenTied for 1st76.472.2–80.6
76.211.0%0
5Decider V1.2 PerplexityTied for 1st75.170.9–79.4
74.31.6%0
6GLiDE FastinoTied for 1st75.070.8–79.3
73.66.3%0
7d1-ultra Liquid AITied for 1st75.070.8–79.2
74.53.1%0
8d1 Liquid AITied for 1st74.570.2–78.7
73.53.2%0
9Elephant 1 BasetenTied for 1st74.269.9–78.5
72.510.5%0
9Decider 1 meraGPTTied for 1st74.269.9–78.5
74.815.2%0
11Jev 1.14 TypeSafe74.269.9–78.4
74.55.0%0
12Gemini Decision 1 Google74.069.7–78.3
72.48.4%0
13Decider Flash V1.2 Perplexity73.068.6–77.4
72.84.3%0
13Drex Max Nace AI73.068.6–77.4
72.812.0%0
15Jev 1.13 TypeSafe72.768.3–77.0
72.94.0%0
16Clef Cloudflare72.267.9–76.6
69.62.6%0
17Span-02 Respan72.167.7–76.5
70.913.0%0
18d1.1 Liquid AI72.067.6–76.4
72.05.9%0
19Mercury Decide Inception71.166.7–75.5
69.518.8%0
20GPT-6 Luna Decisions OpenAI71.066.6–75.5
71.68.4%0
21Clef Flash Cloudflare70.566.1–75.0
67.91.2%0
22Solar Decide Upstage69.965.4–74.4
70.121.0%0
23Drex v2.0 Nace AI68.764.1–73.2
66.74.5%0
24Tev1 4B Experimental Together AI67.062.4–71.7
66.34.7%0
25Clef Omni Cloudflare67.062.4–71.6
67.12.7%0
26Kev 4B Jared Palmer66.361.7–71.0
65.49.0%0
27d1-3B Liquid AI64.860.2–69.5
62.43.1%0
28Drex v1.5 Nace AI64.860.1–69.4
63.013.6%0
29d3 nano vLLM Semantic Router62.657.9–67.4
61.611.5%0
30Solar Decide Flash Upstage62.557.8–67.3
63.012.5%0
31Joey 1 Baseten62.457.6–67.1
60.26.0%0
32d1-omni-600M Liquid AI57.552.6–62.3
56.05.7%0
33Naluri-308M Jericho Siahaya54.749.8–59.6
43.66.5%0
34d3 lite vLLM Semantic Router52.647.7–57.5
50.613.4%0
35Naluri-279M Jericho Siahaya47.342.4–52.2
45.07.8%0
36Kev 0.8B Jared Palmer46.341.4–51.2
45.74.6%0
37d3 edge vLLM Semantic Router42.337.5–47.1
38.334.4%0
38Laya Router Convai Innovations36.431.6–41.1
32.717.3%0
39Classify-1 ScaleDown27.222.8–31.5
8.54.6%2000

The bar is accuracy; the bracket on it is the 95% interval for 2,000 questions over 400 cases, counted as 400 samples. Tied for 1st marks every model whose gap to the leader is within sampling noise (1.96 standard errors of the difference, treating the runs as independent). A failed call counts as wrong. In-domain models trained on another split of this dataset, by their maker's own account.

About this benchmark

Question
Agent and product states, 5 questions per case
Source
LocalLLaMA/typed-decisions, test split
Sample
The whole test split: 400 cases, 5 questions each, with soft labels
Pinned
Every source is pinned to an exact commit, and the cases are frozen by hash in data.lock.json.

It counts towards the System One Index and its English sub-index. Every model gets the same cases and the same request, under the benchmark rules.