system-one-bench

~/system-one-bench / models/baseten-elephant-1

Baseten · OpenRouter API

Elephant 1

Elephant 1 by Baseten scores 82.1 on the System One Index (11th of 39). Calibration error 3.7%.

System One Index
82.1
11th of 39
English
81.3
Arabic
82.0
Vision Index
–
No image input
Calibration error
3.7%
Lower is better
Median latency
–
Cost per 1,000 calls
$0.013
As billed

Text datasets

DatasetAccuracyRankBest on the boardCalibration error
AG News English89.018th of 3992.55.1%
SST-2 English94.520th of 3998.02.9%
SST-5 English58.514th of 3962.512.9%
DAIR Emotion English54.532nd of 3991.522.0%
Banking77 English94.54th of 3998.52.9%
BoolQ English91.09th of 3993.04.4%
XNLI English English94.04th of 3995.34.0%
Typed decisions English74.29th of 3980.010.5%
XNLI French French90.02nd of 3991.35.4%
XNLI Arabic Arabic79.39th of 4190.07.4%
MASSIVE scenario Arabic69.021st of 4189.514.5%
MASSIVE intent Arabic89.58th of 4191.53.9%
OSACT4 offensive Arabic91.015th of 4197.03.5%
Tweet sentiment Arabic66.021st of 4180.013.7%
MIRACL rerank Arabic88.012th of 4190.55.0%
MIRACL pairwise Arabic91.03rd of 4191.51.6%

This run

Model id
baseten/elephant-1
Runs on
OpenRouter API
Run date
2026-10-30
Latency
API; not timed (round 9 ran from a machine whose round trips don't compare with the board's)
Questions
4,850, 0 failed (counted wrong)

9B, built on Qwen3.5-9B by Base Labs, Baseten's research team. MIT weights (baselabs/elephant-1). Text only.

Compare

Ranks count the models charted on the leaderboard. Every model gets the same cases and the same request, under the benchmark rules.