system-one-bench

~/system-one-bench / models/baseten-joey-1

Baseten · OpenRouter API

Joey 1

Joey 1 by Baseten scores 73.3 on the System One Index (28th of 39). Calibration error 2.1%.

System One Index
73.3
28th of 39
English
72.9
Arabic
73.5
Vision Index
–
No image input
Calibration error
2.1%
Lower is better
Median latency
–
Cost per 1,000 calls
$0.004
As billed

Text datasets

DatasetAccuracyRankBest on the boardCalibration error
AG News English85.035th of 3992.54.9%
SST-2 English86.034th of 3998.07.0%
SST-5 English40.534th of 3962.58.1%
DAIR Emotion English52.539th of 3991.517.0%
Banking77 English93.06th of 3998.53.2%
BoolQ English80.032nd of 3993.04.7%
XNLI English English84.027th of 3995.35.7%
Typed decisions English62.431st of 3980.06.0%
XNLI French French74.728th of 3991.311.8%
XNLI Arabic Arabic64.732nd of 4190.012.1%
MASSIVE scenario Arabic71.516th of 4189.59.6%
MASSIVE intent Arabic84.520th of 4191.53.4%
OSACT4 offensive Arabic83.032nd of 4197.04.4%
Tweet sentiment Arabic66.021st of 4180.07.3%
MIRACL rerank Arabic62.035th of 4190.57.3%
MIRACL pairwise Arabic83.031st of 4191.510.3%

This run

Model id
baseten/joey-1
Runs on
OpenRouter API
Run date
2026-10-30
Latency
API; not timed (round 9 ran from a machine whose round trips don't compare with the board's)
Questions
4,850, 0 failed (counted wrong)

Built on Qwen3.5-2B by Base Labs, Baseten's research team. MIT weights (baselabs/joey-1). Text only.

Compare

Ranks count the models charted on the leaderboard. Every model gets the same cases and the same request, under the benchmark rules.