system-one-bench

~/system-one-bench / compare/cloudflare-clef-flash-vs-google-gemini-decision-1

Head to head

Clef Flash vs Gemini Decision 1

Clef Flash scores 80.3 and Gemini Decision 1 81.6 on the System One Index. Clef Flash is more accurate on 4 of 16 text datasets, Gemini Decision 1 on 12. Same cases, same request, same scorer.

Overall

Clef FlashGemini Decision 1
System One Index80.381.6
English80.279.2
Arabic80.584.5
Vision Index78.089.2
Calibration error3.5%3.6%
Median latency267 ms–
Cost per 1,000 calls$0.044$0.036

Text datasets, accuracy

Clef FlashGemini Decision 1
AG News English90.088.5
SST-2 English95.096.5
SST-5 English57.059.0
DAIR Emotion English57.059.5
Banking77 English98.579.5
BoolQ English88.589.0
XNLI English English84.787.3
Typed decisions English70.574.0
XNLI French French80.081.3
XNLI Arabic Arabic68.778.0
MASSIVE scenario Arabic89.572.5
MASSIVE intent Arabic90.587.0
OSACT4 offensive Arabic89.595.5
Tweet sentiment Arabic47.078.0
MIRACL rerank Arabic88.089.5
MIRACL pairwise Arabic90.091.0

Image datasets, accuracy

Clef FlashGemini Decision 1
SST-2, as an image95.594.5
Arabic sentiment, as an image33.575.0
Oxford-IIIT Pet85.098.0
VQAv2 yes/no88.086.0
CountBench88.092.5

Model pages: Clef Flash · Gemini Decision 1. Latency for API models is the median per call through OpenRouter, timed from a cloud server with 8 calls in flight.