~/system-one-bench / models/meragpt-decider-1
meraGPT · meraGPT API
Decider 1 by meraGPT scores 61.8 on the System One Index (35th of 39). Calibration error 7.9%.
| Dataset | Accuracy | Rank | Best on the board | Calibration error |
|---|---|---|---|---|
| AG News English | 87.5 | 28th of 39 | 92.5 | 3.4% |
| SST-2 English | 93.0 | 26th of 39 | 98.0 | 13.2% |
| SST-5 English | 55.5 | 20th of 39 | 62.5 | 9.4% |
| DAIR Emotion English | 67.5 | 6th of 39 | 91.5 | 3.9% |
| Banking77 English | 1.0 | 35th of 39 | 98.5 | 0.3% |
| BoolQ English | 80.0 | 32nd of 39 | 93.0 | 8.6% |
| XNLI English English | 80.7 | 33rd of 39 | 95.3 | 3.6% |
| Typed decisions English | 74.2 | 9th of 39 | 80.0 | 15.2% |
| XNLI French French | 67.3 | 31st of 39 | 91.3 | 8.1% |
| XNLI Arabic Arabic | 67.3 | 30th of 41 | 90.0 | 2.4% |
| MASSIVE scenario Arabic | 3.0 | 40th of 41 | 89.5 | 2.6% |
| MASSIVE intent Arabic | 3.5 | 40th of 41 | 91.5 | 1.5% |
| OSACT4 offensive Arabic | 86.0 | 26th of 41 | 97.0 | 14.7% |
| Tweet sentiment Arabic | 64.0 | 25th of 41 | 80.0 | 9.6% |
| MIRACL rerank Arabic | 81.0 | 28th of 41 | 90.5 | 13.5% |
| MIRACL pairwise Arabic | 78.0 | 34th of 41 | 91.5 | 5.9% |
decider-1Rejects any question with more than 10 options, so all 200 cases each of Banking77 (77 options), Arabic MASSIVE scenario (18) and Arabic MASSIVE intent (20) count as wrong; on the other 13 datasets it averages 75.5. Its 4,096-token cap rejects every image, so it has no vision score. The API takes state-decider-1 or decider-1 and reports decider-1. Cost is as billed, rounded to the millionth of a dollar per call. Its latency, not timed for the board, was about 10 seconds a call.
Ranks count the models charted on the leaderboard. Every model gets the same cases and the same request, under the benchmark rules.