System One Bench

Head to head

d3 lite vs d3 nano

d3 lite scores 74.9 and d3 nano 78.0 on the System One Index. d3 lite is more accurate on 3 of 16 text datasets, d3 nano on 13. Same cases, same request, same scorer.

Overall

d3 lited3 nano
System One Index74.978.0
English74.377.9
Arabic74.776.9
Vision Index––
Calibration error7.1%7.0%
Median latency––
Cost per 1,000 calls––

Text datasets, accuracy

d3 lited3 nano
AG News English85.588.0
SST-2 English86.089.5
SST-5 English50.555.5
DAIR Emotion English57.059.5
Banking77 English91.093.0
BoolQ English81.088.0
XNLI English English90.787.3
Typed decisions English52.662.6
XNLI French French81.386.0
XNLI Arabic Arabic72.782.0
MASSIVE scenario Arabic58.061.5
MASSIVE intent Arabic77.581.0
OSACT4 offensive Arabic80.579.0
Tweet sentiment Arabic67.071.0
MIRACL rerank Arabic82.577.5
MIRACL pairwise Arabic84.586.5

Model pages: d3 lite · d3 nano. Latency for API models is the median per call through OpenRouter, timed from one machine.