system-one-bench

~/system-one-bench / models/google-gemini-decision-1

Google · Gemini API

Gemini Decision 1

Gemini Decision 1 by Google scores 81.6 on the System One Index (14th of 39) and 89.2 on the Vision Index (1st of 14). Calibration error 3.6%.

System One Index
81.6
14th of 39
English
79.2
Arabic
84.5
Vision Index
89.2
1st of 14
Calibration error
3.6%
Lower is better
Median latency
–
Cost per 1,000 calls
$0.036
At list price

Text datasets

DatasetAccuracyRankBest on the boardCalibration error
AG News English88.523rd of 3992.55.3%
SST-2 English96.54th of 3998.05.0%
SST-5 English59.011th of 3962.58.1%
DAIR Emotion English59.514th of 3991.515.1%
Banking77 English79.528th of 3998.58.3%
BoolQ English89.020th of 3993.03.6%
XNLI English English87.318th of 3995.33.4%
Typed decisions English74.012th of 3980.08.4%
XNLI French French81.318th of 3991.35.6%
XNLI Arabic Arabic78.012th of 4190.011.7%
MASSIVE scenario Arabic72.514th of 4189.510.1%
MASSIVE intent Arabic87.014th of 4191.54.2%
OSACT4 offensive Arabic95.52nd of 4197.04.7%
Tweet sentiment Arabic78.02nd of 4180.04.7%
MIRACL rerank Arabic89.53rd of 4190.54.9%
MIRACL pairwise Arabic91.03rd of 4191.53.2%

Image datasets

DatasetAccuracyRankBest on the boardCalibration error
SST-2, as an image94.55th of 1497.014.2%
Arabic sentiment, as an image75.01st of 1475.06.6%
Oxford-IIIT Pet98.01st of 1498.03.0%
VQAv2 yes/no86.013th of 1489.512.1%
CountBench92.56th of 1495.59.4%

This run

Model id
gemini-decision-1
Runs on
Gemini API
Run date
2026-11-11
Latency
API; not timed (round 13 ran on Google's own API from a machine whose round trips don't compare with the board's)
Questions
4,850, 4 failed (counted wrong)

Closed weights, on Google's Decisions API, which takes a different request body: run_decisions.py --provider google sends the same state and questions, with the state as JSON text in `input`, noul criteria renamed true_description and false_description, and images as image blocks. Google's safety filter refused 4 text cases (2 OSACT4 offensive, 1 SST-5, 1 DAIR Emotion), which count as wrong. Cost is input tokens at the $0.10 per 1M list price, since the API reports no cost. Ran on a paid-tier key, so Google doesn't train on the requests. Doesn't answer Box questions.

Compare

Ranks count the models charted on the leaderboard. Every model gets the same cases and the same request, under the benchmark rules.