~/system-one-bench / benchmarks/countbench
Vision track · All benchmarks
How many objects, 2 to 10, as a 9-level score. Decider V1.2 has the top score, 1.0 points ahead of Jev 1.13, but Jev 1.13 and 6 more are within sampling noise: a statistical tie for first.
| Rank | Model | Accuracy | 95% interval | Macro-F1 | Calibration error | Failed | |
|---|---|---|---|---|---|---|---|
| 1 | 95.5 | 92.6–98.4 | 95.3 | 4.4% | 0 | ||
| 2 | Jev 1.13 TypeSafeTied for 1st | 94.5 | 91.3–97.7 | 94.3 | 4.6% | 0 | |
| 3 | Jev 1.14 TypeSafeTied for 1st | 93.5 | 90.1–96.9 | 93.5 | 4.7% | 0 | |
| 3 | Mercury Decide InceptionTied for 1st | 93.5 | 90.1–96.9 | 93.3 | 1.4% | 0 | |
| 3 | GLiDE FastinoTied for 1st | 93.5 | 90.1–96.9 | 93.3 | 3.2% | 0 | |
| 6 | 92.5 | 88.8–96.2 | 92.3 | 9.4% | 0 | ||
| 7 | d1-ultra Liquid AITied for 1st | 91.5 | 87.6–95.4 | 91.8 | 7.6% | 0 | |
| 7 | d1.1 Liquid AITied for 1st | 91.5 | 87.6–95.4 | 91.8 | 6.9% | 0 | |
| 9 | 90.0 | 85.8–94.2 | 89.3 | 13.3% | 0 | ||
| 10 | Span-02 Respan | 88.5 | 84.1–92.9 | 87.7 | 8.7% | 0 | |
| 11 | 88.0 | 83.5–92.5 | 87.9 | 29.3% | 0 | ||
| 12 | 87.5 | 82.9–92.1 | 87.3 | 5.2% | 0 | ||
| 13 | 85.0 | 80.1–89.9 | 85.1 | 9.0% | 0 | ||
| 14 | 68.0 | 61.5–74.5 | 67.7 | 4.7% | 0 |
The bar is accuracy; the bracket on it is the 95% interval for 200 cases. Tied for 1st marks every model whose gap to the leader is within sampling noise (1.96 standard errors of the difference, treating the runs as independent). A failed call counts as wrong.
It counts towards the Vision Index. Every model gets the same cases and the same request, under the benchmark rules.