system-one-bench

~/system-one-bench / models/patrick-18-dexy-decision-9b

Patrick-18 · Open weights, local

Dexy-Decision-9B

Dexy-Decision-9B by Patrick-18 scores 84.7 on the System One Index (1st of 39). Calibration error 3.5%. 4 of the datasets it ran are in-domain: it trained on another split of them.

System One Index
84.7
1st of 39
English
82.3
Arabic
86.8
Vision Index
–
No image input
Without in-domain datasets
83.4
7th of 39 on the same 12 datasets
Calibration error
3.5%
Lower is better
Median latency
62 ms
Local, not comparable
Cost per 1,000 calls
Local
As billed

Text datasets

DatasetAccuracyRankBest on the boardCalibration error
AG News English91.55th of 3992.55.0%
SST-2 English95.58th of 3998.04.3%
SST-5 English60.07th of 3962.53.9%
DAIR Emotion English54.035th of 3991.516.1%
Banking77 English, in-domain93.06th of 3998.53.1%
BoolQ English89.517th of 3993.05.4%
XNLI English English94.72nd of 3995.33.0%
Typed decisions English, in-domain80.01st of 3980.011.3%
XNLI French French89.35th of 3991.36.2%
XNLI Arabic Arabic89.32nd of 4190.04.4%
MASSIVE scenario Arabic, in-domain, unusually high89.51st of 4189.55.2%
MASSIVE intent Arabic, in-domain91.51st of 4191.55.2%
OSACT4 offensive Arabic91.513th of 4197.04.1%
Tweet sentiment Arabic67.019th of 4180.013.1%
MIRACL rerank Arabic88.012th of 4190.55.4%
MIRACL pairwise Arabic91.03rd of 4191.54.8%

This run

Model id
Patrick-18/Dexy-Decision-9B
Runs on
Open weights, local
Run date
2026-11-11
Latency
Cloud A100 80 GB GPU, the maker's vLLM adapter, one call at a time (90 calls)
Questions
4,850, 0 failed (counted wrong)

In-domain: Typed decisions, Banking77, MASSIVE scenario, MASSIVE intent. Its developer reported fine-tuning on the train splits of typed-decisions and Banking77, and on MASSIVE's English and Chinese train splits. We test a different split. The System One Index counts all 16 datasets for every model.

Independent fine-tune of Qwen3.5-9B, Apache 2.0, run at revision afee4ba through the maker's vLLM /v1/systemone adapter. Its developer reported its training data before the run: the train splits of typed-decisions and Banking77, and MASSIVE's English and Chinese train splits, so those datasets and the Arabic MASSIVE ones are in-domain. None of our test cases appear in those train splits. Text only.

Unusually high: MASSIVE scenario, 89.5%, 16 points above the 74% its other datasets predict. A score this far above a model’s other results often means it trained on that dataset or one like it. It isn’t proof, and the score still counts.

Compare

Ranks count the models charted on the leaderboard. Every model gets the same cases and the same request, under the benchmark rules.