~/system-one-bench / models/patrick-18-dexy-decision-9b
Patrick-18 · Open weights, local
Dexy-Decision-9B by Patrick-18 scores 84.7 on the System One Index (1st of 39). Calibration error 3.5%. 4 of the datasets it ran are in-domain: it trained on another split of them.
| Dataset | Accuracy | Rank | Best on the board | Calibration error |
|---|---|---|---|---|
| AG News English | 91.5 | 5th of 39 | 92.5 | 5.0% |
| SST-2 English | 95.5 | 8th of 39 | 98.0 | 4.3% |
| SST-5 English | 60.0 | 7th of 39 | 62.5 | 3.9% |
| DAIR Emotion English | 54.0 | 35th of 39 | 91.5 | 16.1% |
| Banking77 English, in-domain | 93.0 | 6th of 39 | 98.5 | 3.1% |
| BoolQ English | 89.5 | 17th of 39 | 93.0 | 5.4% |
| XNLI English English | 94.7 | 2nd of 39 | 95.3 | 3.0% |
| Typed decisions English, in-domain | 80.0 | 1st of 39 | 80.0 | 11.3% |
| XNLI French French | 89.3 | 5th of 39 | 91.3 | 6.2% |
| XNLI Arabic Arabic | 89.3 | 2nd of 41 | 90.0 | 4.4% |
| MASSIVE scenario Arabic, in-domain, unusually high | 89.5 | 1st of 41 | 89.5 | 5.2% |
| MASSIVE intent Arabic, in-domain | 91.5 | 1st of 41 | 91.5 | 5.2% |
| OSACT4 offensive Arabic | 91.5 | 13th of 41 | 97.0 | 4.1% |
| Tweet sentiment Arabic | 67.0 | 19th of 41 | 80.0 | 13.1% |
| MIRACL rerank Arabic | 88.0 | 12th of 41 | 90.5 | 5.4% |
| MIRACL pairwise Arabic | 91.0 | 3rd of 41 | 91.5 | 4.8% |
Patrick-18/Dexy-Decision-9BIn-domain: Typed decisions, Banking77, MASSIVE scenario, MASSIVE intent. Its developer reported fine-tuning on the train splits of typed-decisions and Banking77, and on MASSIVE's English and Chinese train splits. We test a different split. The System One Index counts all 16 datasets for every model.
Independent fine-tune of Qwen3.5-9B, Apache 2.0, run at revision afee4ba through the maker's vLLM /v1/systemone adapter. Its developer reported its training data before the run: the train splits of typed-decisions and Banking77, and MASSIVE's English and Chinese train splits, so those datasets and the Arabic MASSIVE ones are in-domain. None of our test cases appear in those train splits. Text only.
Unusually high: MASSIVE scenario, 89.5%, 16 points above the 74% its other datasets predict. A score this far above a model’s other results often means it trained on that dataset or one like it. It isn’t proof, and the score still counts.
Ranks count the models charted on the leaderboard. Every model gets the same cases and the same request, under the benchmark rules.