system-one-bench

~/system-one-bench / benchmarks/sst-2-as-an-image

Vision track · All benchmarks

SST-2, as an image

The 200 SST-2 reviews rendered as a picture, yes/no. d1-ultra has the top score, 0.5 points ahead of d1.1, but d1.1 and 9 more are within sampling noise: a statistical tie for first.

Top score
d1-ultra
97.0, tied with 10 within noise
Lead over 2nd
0.5
Points of accuracy
Models
14
Charted on the leaderboard
Cases
200
Best constant answer
50.0
Always giving the most common label
Random guess
50.0
Uniform over the options

Leaderboard

RankModelAccuracy95% intervalMacro-F1Calibration errorFailed
1d1-ultra Liquid AILeader97.094.6–99.4
97.04.3%0
2d1.1 Liquid AITied for 1st96.594.0–99.0
96.55.6%0
3Clef CloudflareTied for 1st95.592.6–98.4
95.53.9%0
3Clef Flash CloudflareTied for 1st95.592.6–98.4
95.55.1%0
5Jev 1.13 TypeSafeTied for 1st94.591.3–97.7
94.56.0%0
5Decider V1.2 PerplexityTied for 1st94.591.3–97.7
94.54.9%0
5Gemini Decision 1 GoogleTied for 1st94.591.3–97.7
94.514.2%0
8GLiDE FastinoTied for 1st94.090.7–97.3
94.05.1%0
9Jev 1.14 TypeSafeTied for 1st93.590.1–96.9
93.56.0%0
9Decider Flash V1.2 PerplexityTied for 1st93.590.1–96.9
93.54.1%0
11Mercury Decide InceptionTied for 1st93.089.5–96.5
93.02.8%0
12Span-02 Respan92.588.8–96.2
92.51.8%0
13Clef Omni Cloudflare89.585.3–93.7
89.45.5%0
14GPT-6 Luna Decisions OpenAI88.584.1–92.9
88.45.5%1

The bar is accuracy; the bracket on it is the 95% interval for 200 cases. Tied for 1st marks every model whose gap to the leader is within sampling noise (1.96 standard errors of the difference, treating the runs as independent). A failed call counts as wrong.

About this benchmark

Question
The 200 SST-2 reviews rendered as a picture, yes/no
Source
stanfordnlp/sst2, validation split
Sample
The 200 SST-2 reviews from the text track, rendered in Noto Sans
Pinned
Every source is pinned to an exact commit, and the cases are frozen by hash in data.lock.json.

It counts towards the Vision Index. Every model gets the same cases and the same request, under the benchmark rules.