Upstage · OpenRouter API
Solar Decide by Upstage scores 68.8 on the System One Index (17th of 24). Calibration error 20.3%, median latency 715 ms.
| Dataset | Accuracy | Rank | Best on the board | Calibration error |
|---|---|---|---|---|
| AG News English | 86.0 | 19th of 24 | 92.5 | 12.6% |
| SST-2 English | 91.5 | 15th of 24 | 97.0 | 5.9% |
| SST-5 English | 53.5 | 10th of 24 | 62.5 | 42.3% |
| DAIR Emotion English | 54.0 | 22nd of 24 | 91.5 | 40.5% |
| Banking77 English | 1.0 | 22nd of 24 | 98.5 | 0.3% |
| BoolQ English | 87.0 | 14th of 24 | 92.5 | 9.9% |
| XNLI English English | 74.7 | 21st of 24 | 93.3 | 20.8% |
| Typed decisions English | 69.9 | 10th of 24 | 79.0 | 21.0% |
| XNLI French French | 67.3 | 18th of 24 | 90.7 | 27.8% |
| XNLI Arabic Arabic | 58.0 | 21st of 26 | 89.3 | 36.8% |
| MASSIVE scenario Arabic | 55.5 | 21st of 26 | 89.5 | 37.2% |
| MASSIVE intent Arabic | 75.0 | 17th of 26 | 91.5 | 18.9% |
| OSACT4 offensive Arabic | 86.0 | 14th of 26 | 95.5 | 8.8% |
| Tweet sentiment Arabic | 71.5 | 5th of 26 | 80.0 | 24.8% |
| MIRACL rerank Arabic | 84.5 | 12th of 26 | 90.5 | 11.0% |
| MIRACL pairwise Arabic | 85.5 | 17th of 26 | 91.0 | 10.6% |
upstage/solar-decideAccepts at most 26 options per question, so all 200 Banking77 cases (77 options) failed and count as wrong.
Ranks count the models charted on the leaderboard. Every model gets the same cases and the same request, under the benchmark rules.