~/system-one-bench / benchmarks/vqav2-yes-no
Vision track · All benchmarks
A yes/no question about a photo. d1-ultra and d1.1 share the top score, and 11 more are within sampling noise.
| Rank | Model | Accuracy | 95% interval | Macro-F1 | Calibration error | Failed | |
|---|---|---|---|---|---|---|---|
| 1 | d1-ultra Liquid AILeader | 89.5 | 85.3–93.7 | 89.5 | 3.2% | 0 | |
| 1 | d1.1 Liquid AITied for 1st | 89.5 | 85.3–93.7 | 89.5 | 3.2% | 0 | |
| 3 | GLiDE FastinoTied for 1st | 89.0 | 84.7–93.3 | 89.0 | 2.7% | 0 | |
| 4 | Jev 1.13 TypeSafeTied for 1st | 88.0 | 83.5–92.5 | 88.0 | 3.0% | 0 | |
| 4 | 88.0 | 83.5–92.5 | 88.0 | 1.9% | 0 | ||
| 6 | 87.5 | 82.9–92.1 | 87.5 | 2.5% | 0 | ||
| 6 | 87.5 | 82.9–92.1 | 87.5 | 4.5% | 0 | ||
| 8 | Mercury Decide InceptionTied for 1st | 87.0 | 82.3–91.7 | 87.0 | 7.9% | 0 | |
| 8 | 87.0 | 82.3–91.7 | 87.0 | 5.4% | 0 | ||
| 8 | Span-02 RespanTied for 1st | 87.0 | 82.3–91.7 | 87.0 | 6.7% | 0 | |
| 11 | Jev 1.14 TypeSafeTied for 1st | 86.5 | 81.8–91.2 | 86.5 | 5.8% | 0 | |
| 11 | 86.5 | 81.8–91.2 | 86.5 | 3.1% | 0 | ||
| 13 | 86.0 | 81.2–90.8 | 86.0 | 12.1% | 0 | ||
| 14 | 80.5 | 75.0–86.0 | 80.5 | 8.2% | 1 |
The bar is accuracy; the bracket on it is the 95% interval for 200 cases. Tied for 1st marks every model whose gap to the leader is within sampling noise (1.96 standard errors of the difference, treating the runs as independent). A failed call counts as wrong.
It counts towards the Vision Index. Every model gets the same cases and the same request, under the benchmark rules.