system-one-bench

~/system-one-bench / benchmarks/vqav2-yes-no

Vision track · All benchmarks

VQAv2 yes/no

A yes/no question about a photo. d1-ultra and d1.1 share the top score, and 11 more are within sampling noise.

Top score
d1-ultra
89.5, tied with 12 within noise
Lead over 2nd
0.0
Points of accuracy
Models
14
Charted on the leaderboard
Cases
200
Best constant answer
52.0
Always giving the most common label
Random guess
50.0
Uniform over the options

Leaderboard

RankModelAccuracy95% intervalMacro-F1Calibration errorFailed
1d1-ultra Liquid AILeader89.585.3–93.7
89.53.2%0
1d1.1 Liquid AITied for 1st89.585.3–93.7
89.53.2%0
3GLiDE FastinoTied for 1st89.084.7–93.3
89.02.7%0
4Jev 1.13 TypeSafeTied for 1st88.083.5–92.5
88.03.0%0
4Clef Flash CloudflareTied for 1st88.083.5–92.5
88.01.9%0
6Clef CloudflareTied for 1st87.582.9–92.1
87.52.5%0
6Decider Flash V1.2 PerplexityTied for 1st87.582.9–92.1
87.54.5%0
8Mercury Decide InceptionTied for 1st87.082.3–91.7
87.07.9%0
8Clef Omni CloudflareTied for 1st87.082.3–91.7
87.05.4%0
8Span-02 RespanTied for 1st87.082.3–91.7
87.06.7%0
11Jev 1.14 TypeSafeTied for 1st86.581.8–91.2
86.55.8%0
11Decider V1.2 PerplexityTied for 1st86.581.8–91.2
86.53.1%0
13Gemini Decision 1 GoogleTied for 1st86.081.2–90.8
86.012.1%0
14GPT-6 Luna Decisions OpenAI80.575.0–86.0
80.58.2%1

The bar is accuracy; the bracket on it is the 95% interval for 200 cases. Tied for 1st marks every model whose gap to the leader is within sampling noise (1.96 standard errors of the difference, treating the runs as independent). A failed call counts as wrong.

About this benchmark

Question
A yes/no question about a photo
Source
lmms-lab-encoder/VQAv2, validation split
Sample
200 photos from the first shard, one yes/no question each
Pinned
Every source is pinned to an exact commit, and the cases are frozen by hash in data.lock.json.

It counts towards the Vision Index. Every model gets the same cases and the same request, under the benchmark rules.