~/system-one-bench / benchmarks/sst-2-as-an-image
Vision track · All benchmarks
The 200 SST-2 reviews rendered as a picture, yes/no. d1-ultra has the top score, 0.5 points ahead of d1.1, but d1.1 and 9 more are within sampling noise: a statistical tie for first.
| Rank | Model | Accuracy | 95% interval | Macro-F1 | Calibration error | Failed | |
|---|---|---|---|---|---|---|---|
| 1 | d1-ultra Liquid AILeader | 97.0 | 94.6–99.4 | 97.0 | 4.3% | 0 | |
| 2 | d1.1 Liquid AITied for 1st | 96.5 | 94.0–99.0 | 96.5 | 5.6% | 0 | |
| 3 | 95.5 | 92.6–98.4 | 95.5 | 3.9% | 0 | ||
| 3 | 95.5 | 92.6–98.4 | 95.5 | 5.1% | 0 | ||
| 5 | Jev 1.13 TypeSafeTied for 1st | 94.5 | 91.3–97.7 | 94.5 | 6.0% | 0 | |
| 5 | 94.5 | 91.3–97.7 | 94.5 | 4.9% | 0 | ||
| 5 | 94.5 | 91.3–97.7 | 94.5 | 14.2% | 0 | ||
| 8 | GLiDE FastinoTied for 1st | 94.0 | 90.7–97.3 | 94.0 | 5.1% | 0 | |
| 9 | Jev 1.14 TypeSafeTied for 1st | 93.5 | 90.1–96.9 | 93.5 | 6.0% | 0 | |
| 9 | 93.5 | 90.1–96.9 | 93.5 | 4.1% | 0 | ||
| 11 | Mercury Decide InceptionTied for 1st | 93.0 | 89.5–96.5 | 93.0 | 2.8% | 0 | |
| 12 | Span-02 Respan | 92.5 | 88.8–96.2 | 92.5 | 1.8% | 0 | |
| 13 | 89.5 | 85.3–93.7 | 89.4 | 5.5% | 0 | ||
| 14 | 88.5 | 84.1–92.9 | 88.4 | 5.5% | 1 |
The bar is accuracy; the bracket on it is the 95% interval for 200 cases. Tied for 1st marks every model whose gap to the leader is within sampling noise (1.96 standard errors of the difference, treating the runs as independent). A failed call counts as wrong.
It counts towards the Vision Index. Every model gets the same cases and the same request, under the benchmark rules.