Inception ยท OpenRouter API (free tier)
Mercury Decide by Inception scores 82.4 on the System One Index (3rd of 24) and 79.4 on the Vision Index (5th of 8). Calibration error 13.1%, median latency 276 ms.
| Dataset | Accuracy | Rank | Best on the board | Calibration error |
|---|---|---|---|---|
| AG News English | 92.0 | 2nd of 24 | 92.5 | 5.6% |
| SST-2 English | 95.5 | 4th of 24 | 97.0 | 1.6% |
| SST-5 English | 52.0 | 12th of 24 | 62.5 | 22.8% |
| DAIR Emotion English | 91.5 | 1st of 24 | 91.5 | 4.9% |
| Banking77 English | 86.0 | 12th of 24 | 98.5 | 8.9% |
| BoolQ English | 90.5 | 5th of 24 | 92.5 | 5.3% |
| XNLI English English | 91.3 | 2nd of 24 | 93.3 | 4.6% |
| Typed decisions English | 71.1 | 7th of 24 | 79.0 | 18.8% |
| XNLI French French | 82.7 | 4th of 24 | 90.7 | 10.5% |
| XNLI Arabic Arabic | 76.0 | 5th of 26 | 89.3 | 13.8% |
| MASSIVE scenario Arabic | 72.0 | 7th of 26 | 89.5 | 16.5% |
| MASSIVE intent Arabic | 85.0 | 7th of 26 | 91.5 | 7.8% |
| OSACT4 offensive Arabic | 92.0 | 4th of 26 | 95.5 | 3.5% |
| Tweet sentiment Arabic | 62.5 | 15th of 26 | 80.0 | 22.0% |
| MIRACL rerank Arabic | 88.0 | 5th of 26 | 90.5 | 8.2% |
| MIRACL pairwise Arabic | 90.0 | 4th of 26 | 91.0 | 7.6% |
| Dataset | Accuracy | Rank | Best on the board | Calibration error |
|---|---|---|---|---|
| SST-2, as an image | 93.0 | 6th of 8 | 96.0 | 2.8% |
| Arabic sentiment, as an image | 39.0 | 5th of 8 | 67.0 | 37.5% |
| Oxford-IIIT Pet | 84.5 | 5th of 8 | 88.5 | 5.7% |
| VQAv2 yes/no | 87.0 | 5th of 8 | 89.5 | 7.9% |
| CountBench | 93.5 | 3rd of 8 | 94.5 | 1.4% |
inception/mercury-decide:freeRun on OpenRouter's free tier, so its cost is $0.
Ranks count the models charted on the leaderboard. Every model gets the same cases and the same request, under the benchmark rules.