~/system-one-bench / models/google-gemini-decision-1
Google · Gemini API
Gemini Decision 1 by Google scores 81.6 on the System One Index (14th of 39) and 89.2 on the Vision Index (1st of 14). Calibration error 3.6%.
| Dataset | Accuracy | Rank | Best on the board | Calibration error |
|---|---|---|---|---|
| AG News English | 88.5 | 23rd of 39 | 92.5 | 5.3% |
| SST-2 English | 96.5 | 4th of 39 | 98.0 | 5.0% |
| SST-5 English | 59.0 | 11th of 39 | 62.5 | 8.1% |
| DAIR Emotion English | 59.5 | 14th of 39 | 91.5 | 15.1% |
| Banking77 English | 79.5 | 28th of 39 | 98.5 | 8.3% |
| BoolQ English | 89.0 | 20th of 39 | 93.0 | 3.6% |
| XNLI English English | 87.3 | 18th of 39 | 95.3 | 3.4% |
| Typed decisions English | 74.0 | 12th of 39 | 80.0 | 8.4% |
| XNLI French French | 81.3 | 18th of 39 | 91.3 | 5.6% |
| XNLI Arabic Arabic | 78.0 | 12th of 41 | 90.0 | 11.7% |
| MASSIVE scenario Arabic | 72.5 | 14th of 41 | 89.5 | 10.1% |
| MASSIVE intent Arabic | 87.0 | 14th of 41 | 91.5 | 4.2% |
| OSACT4 offensive Arabic | 95.5 | 2nd of 41 | 97.0 | 4.7% |
| Tweet sentiment Arabic | 78.0 | 2nd of 41 | 80.0 | 4.7% |
| MIRACL rerank Arabic | 89.5 | 3rd of 41 | 90.5 | 4.9% |
| MIRACL pairwise Arabic | 91.0 | 3rd of 41 | 91.5 | 3.2% |
| Dataset | Accuracy | Rank | Best on the board | Calibration error |
|---|---|---|---|---|
| SST-2, as an image | 94.5 | 5th of 14 | 97.0 | 14.2% |
| Arabic sentiment, as an image | 75.0 | 1st of 14 | 75.0 | 6.6% |
| Oxford-IIIT Pet | 98.0 | 1st of 14 | 98.0 | 3.0% |
| VQAv2 yes/no | 86.0 | 13th of 14 | 89.5 | 12.1% |
| CountBench | 92.5 | 6th of 14 | 95.5 | 9.4% |
gemini-decision-1Closed weights, on Google's Decisions API, which takes a different request body: run_decisions.py --provider google sends the same state and questions, with the state as JSON text in `input`, noul criteria renamed true_description and false_description, and images as image blocks. Google's safety filter refused 4 text cases (2 OSACT4 offensive, 1 SST-5, 1 DAIR Emotion), which count as wrong. Cost is input tokens at the $0.10 per 1M list price, since the API reports no cost. Ran on a paid-tier key, so Google doesn't train on the requests. Doesn't answer Box questions.
Ranks count the models charted on the leaderboard. Every model gets the same cases and the same request, under the benchmark rules.