system-one-bench

~/system-one-bench / models/baseten-mammoth-1

Baseten · OpenRouter API

Mammoth 1

Mammoth 1 by Baseten scores 82.9 on the System One Index (7th of 39). Calibration error 4.9%.

System One Index
82.9
7th of 39
English
82.5
Arabic
82.8
Vision Index
–
No image input
Calibration error
4.9%
Lower is better
Median latency
–
Cost per 1,000 calls
$0.037
As billed

Text datasets

DatasetAccuracyRankBest on the boardCalibration error
AG News English92.02nd of 3992.55.8%
SST-2 English96.05th of 3998.04.1%
SST-5 English59.011th of 3962.59.7%
DAIR Emotion English59.016th of 3991.517.0%
Banking77 English94.54th of 3998.52.8%
BoolQ English92.03rd of 3993.04.0%
XNLI English English91.38th of 3995.33.1%
Typed decisions English76.44th of 3980.011.0%
XNLI French French86.08th of 3991.34.3%
XNLI Arabic Arabic78.711th of 4190.04.9%
MASSIVE scenario Arabic75.08th of 4189.511.9%
MASSIVE intent Arabic91.03rd of 4191.53.6%
OSACT4 offensive Arabic91.513th of 4197.01.9%
Tweet sentiment Arabic64.524th of 4180.015.1%
MIRACL rerank Arabic89.05th of 4190.56.2%
MIRACL pairwise Arabic90.012th of 4191.54.2%

This run

Model id
baseten/mammoth-1
Runs on
OpenRouter API
Run date
2026-10-30
Latency
API; not timed (round 9 ran from a machine whose round trips don't compare with the board's)
Questions
4,850, 0 failed (counted wrong)

27B, built on Qwen3.5-27B by Base Labs, Baseten's research team. MIT weights (baselabs/mammoth-1). Text only.

Compare

Ranks count the models charted on the leaderboard. Every model gets the same cases and the same request, under the benchmark rules.