ScaleDown · OpenRouter API
Classify-1 by ScaleDown scores 30.6 on the System One Index (30th of 30). Calibration error 3.2%, median latency 254 ms.
| Dataset | Accuracy | Rank | Best on the board | Calibration error |
|---|---|---|---|---|
| AG News English | 26.5 | 30th of 30 | 92.5 | 1.5% |
| SST-2 English | 50.0 | 30th of 30 | 97.0 | 0.0% |
| SST-5 English | 13.0 | 30th of 30 | 62.5 | 7.0% |
| DAIR Emotion English | 59.0 | 11th of 30 | 91.5 | 7.8% |
| Banking77 English | 1.0 | 27th of 30 | 98.5 | 0.3% |
| BoolQ English | 38.0 | 30th of 30 | 93.0 | 12.0% |
| XNLI English English | 30.7 | 29th of 30 | 95.3 | 2.7% |
| Typed decisions English | 27.2 | 30th of 30 | 79.0 | 4.6% |
| XNLI French French | 30.7 | 30th of 30 | 91.3 | 2.7% |
| XNLI Arabic Arabic | 30.7 | 31st of 32 | 90.0 | 2.7% |
| MASSIVE scenario Arabic | 3.0 | 32nd of 32 | 89.5 | 2.6% |
| MASSIVE intent Arabic | 3.5 | 32nd of 32 | 91.5 | 1.5% |
| OSACT4 offensive Arabic | 80.0 | 27th of 32 | 97.0 | 30.0% |
| Tweet sentiment Arabic | 35.0 | 31st of 32 | 80.0 | 1.7% |
| MIRACL rerank Arabic | 12.0 | 32nd of 32 | 90.5 | 0.5% |
| MIRACL pairwise Arabic | 50.0 | 32nd of 32 | 91.5 | 0.0% |
scaledown/classify-1Only accepts a state that is a string or an object with text or document fields, so it rejected every case except the 200 DAIR Emotion cases, whose state is {"text": ...}. The 3,050 rejected cases count as wrong. It also rejects image_url parts, so it has no vision score. Its latency and cost come from the 200 calls it answered.
Ranks count the models charted on the leaderboard. Every model gets the same cases and the same request, under the benchmark rules.