~/system-one-bench / benchmarks/oxford-iiit-pet
Vision track · All benchmarks
Cat or dog breed from a photo, 37 options. Gemini Decision 1 leads by 5.5 points over Span-02, more than sampling noise.
| Rank | Model | Accuracy | 95% interval | Macro-F1 | Calibration error | Failed | |
|---|---|---|---|---|---|---|---|
| 1 | 98.0 | 96.1–99.9 | 97.5 | 3.0% | 0 | ||
| 2 | Span-02 Respan | 92.5 | 88.8–96.2 | 91.9 | 8.3% | 0 | |
| 3 | d1.1 Liquid AI | 90.5 | 86.4–94.6 | 91.6 | 6.7% | 0 | |
| 4 | Jev 1.14 TypeSafe | 88.5 | 84.1–92.9 | 88.8 | 5.0% | 0 | |
| 5 | 88.0 | 83.5–92.5 | 85.7 | 35.8% | 0 | ||
| 6 | d1-ultra Liquid AI | 87.5 | 82.9–92.1 | 88.6 | 6.9% | 0 | |
| 7 | 87.0 | 82.3–91.7 | 86.8 | 4.2% | 0 | ||
| 8 | 85.5 | 80.6–90.4 | 85.3 | 17.8% | 0 | ||
| 9 | 85.0 | 80.1–89.9 | 83.9 | 23.2% | 0 | ||
| 10 | Mercury Decide Inception | 84.5 | 79.5–89.5 | 82.9 | 5.7% | 0 | |
| 10 | GLiDE Fastino | 84.5 | 79.5–89.5 | 83.2 | 8.5% | 0 | |
| 12 | 84.0 | 78.9–89.1 | 79.9 | 13.0% | 0 | ||
| 13 | Jev 1.13 TypeSafe | 83.5 | 78.4–88.6 | 84.7 | 5.1% | 0 | |
| 14 | 82.5 | 77.2–87.8 | 81.3 | 14.8% | 0 |
The bar is accuracy; the bracket on it is the 95% interval for 200 cases. Tied for 1st marks every model whose gap to the leader is within sampling noise (1.96 standard errors of the difference, treating the runs as independent). A failed call counts as wrong.
It counts towards the Vision Index. Every model gets the same cases and the same request, under the benchmark rules.