system-one-bench

~/system-one-bench / compare/convai-innovations-laya-router-vs-google-gemini-decision-1

Head to head

Laya Router vs Gemini Decision 1

Laya Router scores 60.4 and Gemini Decision 1 81.6 on the System One Index. Laya Router is more accurate on 1 of 16 text datasets, Gemini Decision 1 on 14, and 1 are tied. Same cases, same request, same scorer.

Overall

Laya RouterGemini Decision 1
System One Index60.481.6
English64.779.2
Arabic53.284.5
Vision Index–89.2
Calibration error20.0%3.6%
Median latency––
Cost per 1,000 calls–$0.036

Text datasets, accuracy

Laya RouterGemini Decision 1
AG News English92.588.5
SST-2 English88.596.5
SST-5 English34.059.0
DAIR Emotion English53.559.5
Banking77 English44.079.5
BoolQ English81.589.0
XNLI English English87.387.3
Typed decisions English36.474.0
XNLI French French76.081.3
XNLI Arabic Arabic68.078.0
MASSIVE scenario Arabic39.072.5
MASSIVE intent Arabic32.587.0
OSACT4 offensive Arabic82.595.5
Tweet sentiment Arabic37.078.0
MIRACL rerank Arabic31.589.5
MIRACL pairwise Arabic82.091.0

Image datasets, accuracy

Laya RouterGemini Decision 1
SST-2, as an image–94.5
Arabic sentiment, as an image–75.0
Oxford-IIIT Pet–98.0
VQAv2 yes/no–86.0
CountBench–92.5

Model pages: Laya Router · Gemini Decision 1. Latency for API models is the median per call through OpenRouter, timed from a cloud server with 8 calls in flight.