system-one-bench

~/system-one-bench / models/meragpt-decider-1

meraGPT · meraGPT API

Decider 1

Decider 1 by meraGPT scores 61.8 on the System One Index (35th of 39). Calibration error 7.9%.

System One Index
61.8
35th of 39
English
67.4
Arabic
54.7
Vision Index
–
No image input
Calibration error
7.9%
Lower is better
Median latency
–
Cost per 1,000 calls
$0.012
As billed

Text datasets

DatasetAccuracyRankBest on the boardCalibration error
AG News English87.528th of 3992.53.4%
SST-2 English93.026th of 3998.013.2%
SST-5 English55.520th of 3962.59.4%
DAIR Emotion English67.56th of 3991.53.9%
Banking77 English1.035th of 3998.50.3%
BoolQ English80.032nd of 3993.08.6%
XNLI English English80.733rd of 3995.33.6%
Typed decisions English74.29th of 3980.015.2%
XNLI French French67.331st of 3991.38.1%
XNLI Arabic Arabic67.330th of 4190.02.4%
MASSIVE scenario Arabic3.040th of 4189.52.6%
MASSIVE intent Arabic3.540th of 4191.51.5%
OSACT4 offensive Arabic86.026th of 4197.014.7%
Tweet sentiment Arabic64.025th of 4180.09.6%
MIRACL rerank Arabic81.028th of 4190.513.5%
MIRACL pairwise Arabic78.034th of 4191.55.9%

This run

Model id
decider-1
Runs on
meraGPT API
Run date
2026-11-08
Latency
API; not timed (round 11 ran on meraGPT's own API from a machine whose round trips don't compare with the board's)
Questions
4,850, 600 failed (counted wrong)

Rejects any question with more than 10 options, so all 200 cases each of Banking77 (77 options), Arabic MASSIVE scenario (18) and Arabic MASSIVE intent (20) count as wrong; on the other 13 datasets it averages 75.5. Its 4,096-token cap rejects every image, so it has no vision score. The API takes state-decider-1 or decider-1 and reports decider-1. Cost is as billed, rounded to the millionth of a dollar per call. Its latency, not timed for the board, was about 10 seconds a call.

Compare

Ranks count the models charted on the leaderboard. Every model gets the same cases and the same request, under the benchmark rules.