system-one-bench

~/system-one-bench / compare/inception-mercury-decide-vs-patrick-18-dexy-decision-9b

Head to head

Mercury Decide vs Dexy-Decision-9B

Mercury Decide scores 82.4 and Dexy-Decision-9B 84.7 on the System One Index. Mercury Decide is more accurate on 4 of 16 text datasets, Dexy-Decision-9B on 10, and 2 are tied. Same cases, same request, same scorer.

Overall

Mercury DecideDexy-Decision-9B
System One Index82.484.7
English83.782.3
Arabic80.886.8
Vision Index79.4–
Calibration error13.1%3.5%
Median latency276 ms–
Cost per 1,000 callsFree–

Text datasets, accuracy

Mercury DecideDexy-Decision-9B
AG News English92.091.5
SST-2 English95.595.5
SST-5 English52.060.0
DAIR Emotion English91.554.0
Banking77 English86.093.0
BoolQ English90.589.5
XNLI English English91.394.7
Typed decisions English71.180.0
XNLI French French82.789.3
XNLI Arabic Arabic76.089.3
MASSIVE scenario Arabic72.089.5
MASSIVE intent Arabic85.091.5
OSACT4 offensive Arabic92.091.5
Tweet sentiment Arabic62.567.0
MIRACL rerank Arabic88.088.0
MIRACL pairwise Arabic90.091.0

Image datasets, accuracy

Mercury DecideDexy-Decision-9B
SST-2, as an image93.0–
Arabic sentiment, as an image39.0–
Oxford-IIIT Pet84.5–
VQAv2 yes/no87.0–
CountBench93.5–

Model pages: Mercury Decide · Dexy-Decision-9B. Latency for API models is the median per call through OpenRouter, timed from a cloud server with 8 calls in flight.