system-one-bench

~/system-one-bench / compare/cloudflare-clef-omni-vs-google-gemini-decision-1

Head to head

Clef Omni vs Gemini Decision 1

Clef Omni scores 79.7 and Gemini Decision 1 81.6 on the System One Index. Clef Omni is more accurate on 7 of 16 text datasets, Gemini Decision 1 on 9. Same cases, same request, same scorer.

Overall

Clef OmniGemini Decision 1
System One Index79.781.6
English78.679.2
Arabic80.584.5
Vision Index72.289.2
Calibration error3.4%3.6%
Median latency208 ms–
Cost per 1,000 calls$0.074$0.036

Text datasets, accuracy

Clef OmniGemini Decision 1
AG News English89.088.5
SST-2 English97.096.5
SST-5 English51.059.0
DAIR Emotion English56.059.5
Banking77 English96.079.5
BoolQ English90.089.0
XNLI English English82.787.3
Typed decisions English67.074.0
XNLI French French82.781.3
XNLI Arabic Arabic73.378.0
MASSIVE scenario Arabic89.072.5
MASSIVE intent Arabic91.587.0
OSACT4 offensive Arabic90.595.5
Tweet sentiment Arabic50.078.0
MIRACL rerank Arabic83.089.5
MIRACL pairwise Arabic86.591.0

Image datasets, accuracy

Clef OmniGemini Decision 1
SST-2, as an image89.594.5
Arabic sentiment, as an image28.575.0
Oxford-IIIT Pet88.098.0
VQAv2 yes/no87.086.0
CountBench68.092.5

Model pages: Clef Omni · Gemini Decision 1. Latency for API models is the median per call through OpenRouter, timed from a cloud server with 8 calls in flight.