system-one-bench

~/system-one-bench / benchmarks/countbench

Vision track · All benchmarks

CountBench

How many objects, 2 to 10, as a 9-level score. Decider V1.2 has the top score, 1.0 points ahead of Jev 1.13, but Jev 1.13 and 6 more are within sampling noise: a statistical tie for first.

Top score
Decider V1.2
95.5, tied with 7 within noise
Lead over 2nd
1.0
Points of accuracy
Models
14
Charted on the leaderboard
Cases
200
Best constant answer
13.0
Always giving the most common label
Random guess
11.1
Uniform over the options

Leaderboard

RankModelAccuracy95% intervalMacro-F1Calibration errorFailed
1Decider V1.2 PerplexityLeader95.592.6–98.4
95.34.4%0
2Jev 1.13 TypeSafeTied for 1st94.591.3–97.7
94.34.6%0
3Jev 1.14 TypeSafeTied for 1st93.590.1–96.9
93.54.7%0
3Mercury Decide InceptionTied for 1st93.590.1–96.9
93.31.4%0
3GLiDE FastinoTied for 1st93.590.1–96.9
93.33.2%0
6Gemini Decision 1 GoogleTied for 1st92.588.8–96.2
92.39.4%0
7d1-ultra Liquid AITied for 1st91.587.6–95.4
91.87.6%0
7d1.1 Liquid AITied for 1st91.587.6–95.4
91.86.9%0
9Clef Cloudflare90.085.8–94.2
89.313.3%0
10Span-02 Respan88.584.1–92.9
87.78.7%0
11Clef Flash Cloudflare88.083.5–92.5
87.929.3%0
12Decider Flash V1.2 Perplexity87.582.9–92.1
87.35.2%0
13GPT-6 Luna Decisions OpenAI85.080.1–89.9
85.19.0%0
14Clef Omni Cloudflare68.061.5–74.5
67.74.7%0

The bar is accuracy; the bracket on it is the 95% interval for 200 cases. Tied for 1st marks every model whose gap to the leader is within sampling noise (1.96 standard errors of the difference, treating the runs as independent). A failed call counts as wrong.

About this benchmark

Question
How many objects, 2 to 10, as a 9-level score
Source
vikhyatk/CountBenchQA, test split
Sample
200 photos, counts from 2 to 10
Pinned
Every source is pinned to an exact commit, and the cases are frozen by hash in data.lock.json.

It counts towards the Vision Index. Every model gets the same cases and the same request, under the benchmark rules.