Perplexity ยท OpenRouter API
Decider V1.1 27B by Perplexity scores 83.4 on the System One Index (1st of 24) and 83.0 on the Vision Index (3rd of 8). Calibration error 5.7%, median latency 263 ms.
| Dataset | Accuracy | Rank | Best on the board | Calibration error |
|---|---|---|---|---|
| AG News English | 92.0 | 2nd of 24 | 92.5 | 7.4% |
| SST-2 English | 96.0 | 3rd of 24 | 97.0 | 1.3% |
| SST-5 English | 60.5 | 4th of 24 | 62.5 | 5.5% |
| DAIR Emotion English | 60.0 | 7th of 24 | 91.5 | 13.0% |
| Banking77 English | 81.0 | 14th of 24 | 98.5 | 7.4% |
| BoolQ English | 92.5 | 1st of 24 | 92.5 | 2.3% |
| XNLI English English | 93.3 | 1st of 24 | 93.3 | 5.3% |
| Typed decisions English | 73.4 | 4th of 24 | 79.0 | 9.4% |
| XNLI French French | 90.7 | 1st of 24 | 90.7 | 5.4% |
| XNLI Arabic Arabic | 89.3 | 1st of 26 | 89.3 | 4.6% |
| MASSIVE scenario Arabic | 72.5 | 6th of 26 | 89.5 | 13.0% |
| MASSIVE intent Arabic | 90.0 | 3rd of 26 | 91.5 | 4.5% |
| OSACT4 offensive Arabic | 95.5 | 1st of 26 | 95.5 | 2.9% |
| Tweet sentiment Arabic | 72.0 | 4th of 26 | 80.0 | 6.1% |
| MIRACL rerank Arabic | 87.0 | 8th of 26 | 90.5 | 7.0% |
| MIRACL pairwise Arabic | 88.5 | 9th of 26 | 91.0 | 5.9% |
| Dataset | Accuracy | Rank | Best on the board | Calibration error |
|---|---|---|---|---|
| SST-2, as an image | 96.0 | 1st of 8 | 96.0 | 4.1% |
| Arabic sentiment, as an image | 51.0 | 4th of 8 | 67.0 | 17.6% |
| Oxford-IIIT Pet | 84.0 | 6th of 8 | 88.5 | 8.8% |
| VQAv2 yes/no | 89.5 | 1st of 8 | 89.5 | 3.8% |
| CountBench | 94.5 | 1st of 8 | 94.5 | 4.8% |
perplexity/pplx-decider-v1.1-27bRanks count the models charted on the leaderboard. Every model gets the same cases and the same request, under the benchmark rules.