System One Bench

Independent decision-model benchmark

Loading the latest results.

System One models read a situation and answer typed questions in one fast call: yes or no, pick one option, or score a level, each with probabilities. Every model gets the same cases and the same request body, scored against each dataset's labels. Text and images are scored as two separate tracks.

System One Bench is a living benchmark: we keep adding models as they're released, and every new model runs the same cases. See the roadmap for what's next.

Text track highlights

System One Index

Mean accuracy across all 16 text datasets, top 5. Higher is better.

Latency

Median time per call in milliseconds, fastest 5 API models. Lower is better.

Cost

USD per 1,000 calls, as billed by the API, cheapest 5. Lower is better.

Both tracks

Leaderboard

Every model run, with its text and vision results side by side. Failed calls count as wrong and are flagged under the model's name. Click a column to sort, and a model to see every dataset score. The search box and the maker filter apply to every chart and table on this page.

Text track

System One Index

The index is the mean accuracy across datasets, with every dataset weighted equally, on a 0 to 100 scale. A model gets an index only if it ran every dataset in the set, so the Arabic fine-tunes appear in the Arabic index alone.

All datasets

Text track

Accuracy against speed and cost

The shaded corner is the most attractive: more accurate, and faster or cheaper. The line joins the models that nothing else beats on both measures.

Index against latency

System One Index (all datasets) against median latency per call, API models.

Index against cost

System One Index (all datasets) against USD per 1,000 calls. Local models have no per-call price.

Text track

Calibration, speed and cost

Calibration error is the average gap between a model's stated confidence and how often it's right, in percentage points. A well-calibrated model's 80% answers are right about 80% of the time.

Calibration error

Top-label ECE over every question, all datasets. Lower is better.

Latency, median and 95th percentile

Milliseconds per call, API models. The bar is the median; the tick marks the 95th percentile, shown on hover. Lower is better.

Cost to run System One Bench

USD for all 3,250 cases, as billed. Lower is better.

Input tokens per call

Mean billed input tokens. Each tokenizer counts differently, so this explains cost rather than ranks models.

Text track

By dataset

One chart per dataset. Bars are colored by who makes the model.

    Show every dataset as a table

    Roadmap

    Models we plan to add. Vendors publish their own scores for several of these; none appear on this page until they've run on System One Bench like every other model.

    Method

    What's measured

    • Each case is a state plus one to five named questions, sent unchanged to every model.
    • Accuracy compares the most likely answer with the dataset's label. A failed call counts as wrong.
    • Text and images are two tracks with two indexes: the System One Index over 16 text datasets, and the Vision Index over 5 image datasets. They rank models separately, so a model can lead one and not the other.
    • Calibration error is top-label ECE over 10 bins. Log loss, Brier score and more are in results.json.
    • Cost is the summed usage.cost the API reports.
    • Latency depends on the network between caller and API, so every API model was timed again on 15 October from one cloud server, all 3,250 cases through OpenRouter with 8 requests in flight. Only latency comes from that run; accuracy, calibration and cost come from each model's original run. Open-weight models ran locally, one call at a time: Laya on an Apple M5 GPU, and the open-weight d1, d3, Kev and Naluri sizes on a 4-vCPU cloud CPU. Their timings aren't comparable with API round trips, so they stay out of the latency charts.
    • The vision track ran on 15 October from a cloud server, 8 requests in flight per model. It sends each image once, as a base64 PNG or JPEG in state, resized to fit every model's limits. Box answers count as right when they overlap the labelled box by at least half (IoU of 0.5 or more).
    • The data is frozen: data.lock.json holds a SHA-256 for every dataset and image, every source is pinned to a fixed version, and runs are refused on data that doesn't match.
    • Nothing was tuned on test data. Some open-weight fine-tunes trained on related data; the repo README lists which.

    Run it yourself

    export OPENROUTER_API_KEY=<your key>
python run_decisions.py --model openai/gpt-6-luna-decisions --name luna
python run_decisions.py --model typesafe/jev-1.13 --name or_jev
python score.py preds_or_jev.jsonl preds_luna.jsonl

    To race two models live on the same cases, run python server.py and open localhost:8061. The race needs API keys, so it runs on your machine rather than on this page.