system-one-bench

~/system-one-bench / benchmarks/refcoco

Vision track · Box questions · All benchmarks

RefCOCO

Box the person or object a phrase describes, 100 people and 100 objects. Jev 1.14 is the only model on this benchmark so far.

Leader
Jev 1.14
78.5
Lead over 2nd
–
Points of accuracy
Models
1
Charted on the leaderboard
Cases
200

Leaderboard

RankModelRight (IoU ≥ 0.5)95% intervalMean IoUCalibration errorFailed
1Jev 1.14 TypeSafeLeader78.572.8–84.2
0.7030.3%0

The bar is accuracy; the bracket on it is the 95% interval for 200 cases. Tied for 1st marks every model whose gap to the leader is within sampling noise (1.96 standard errors of the difference, treating the runs as independent). A failed call counts as wrong.

About this benchmark

Question
Box the person or object a phrase describes, 100 people and 100 objects
Source
lmms-lab-encoder/RefCOCO, testA and testB split
Sample
100 people (testA) and 100 objects (testB), one phrase per image
Pinned
Every source is pinned to an exact commit, and the cases are frozen by hash in data.lock.json.

Box questions are scored apart and stay out of the Vision Index. Every model gets the same cases and the same request, under the benchmark rules.