system-one-bench

~/system-one-bench / benchmarks/oxford-iiit-pet

Vision track · All benchmarks

Oxford-IIIT Pet

Cat or dog breed from a photo, 37 options. Gemini Decision 1 leads by 5.5 points over Span-02, more than sampling noise.

Leader
Gemini Decision 1
98.0
Lead over 2nd
5.5
Points of accuracy
Models
14
Charted on the leaderboard
Cases
200
Best constant answer
5.5
Always giving the most common label
Random guess
2.7
Uniform over the options

Leaderboard

RankModelAccuracy95% intervalMacro-F1Calibration errorFailed
1Gemini Decision 1 GoogleLeader98.096.1–99.9
97.53.0%0
2Span-02 Respan92.588.8–96.2
91.98.3%0
3d1.1 Liquid AI90.586.4–94.6
91.66.7%0
4Jev 1.14 TypeSafe88.584.1–92.9
88.85.0%0
5Clef Omni Cloudflare88.083.5–92.5
85.735.8%0
6d1-ultra Liquid AI87.582.9–92.1
88.66.9%0
7GPT-6 Luna Decisions OpenAI87.082.3–91.7
86.84.2%0
8Decider V1.2 Perplexity85.580.6–90.4
85.317.8%0
9Clef Flash Cloudflare85.080.1–89.9
83.923.2%0
10Mercury Decide Inception84.579.5–89.5
82.95.7%0
10GLiDE Fastino84.579.5–89.5
83.28.5%0
12Decider Flash V1.2 Perplexity84.078.9–89.1
79.913.0%0
13Jev 1.13 TypeSafe83.578.4–88.6
84.75.1%0
14Clef Cloudflare82.577.2–87.8
81.314.8%0

The bar is accuracy; the bracket on it is the 95% interval for 200 cases. Tied for 1st marks every model whose gap to the leader is within sampling noise (1.96 standard errors of the difference, treating the runs as independent). A failed call counts as wrong.

About this benchmark

Question
Cat or dog breed from a photo, 37 options
Source
timm/oxford-iiit-pet, test split
Sample
200 random photos, all 37 breeds as options
Pinned
Every source is pinned to an exact commit, and the cases are frozen by hash in data.lock.json.

It counts towards the Vision Index. Every model gets the same cases and the same request, under the benchmark rules.