system-one-bench

~/system-one-bench / benchmarks/arabic-osact4-offensive

Text track · Arabic · All benchmarks

OSACT4 offensive

Offensive tweet, yes/no. Span-02 has the top score, 1.5 points ahead of d1, but d1 and 8 more are within sampling noise: a statistical tie for first.

Top score
Span-02
97.0, tied with 9 within noise
Lead over 2nd
1.5
Points of accuracy
Models
41
Charted on the leaderboard
Cases
200
Best constant answer
80.0
Always giving the most common label
Random guess
50.0
Uniform over the options

Leaderboard

RankModelAccuracy95% intervalMacro-F1Calibration errorFailed
1Span-02 RespanLeader97.094.6–99.4
95.10.9%0
2d1 Liquid AITied for 1st95.592.6–98.4
92.85.1%0
2Gemini Decision 1 GoogleTied for 1st95.592.6–98.4
92.84.7%2
4Decider V1.2 PerplexityTied for 1st95.092.0–98.0
92.54.4%0
4GLiDE FastinoTied for 1st95.092.0–98.0
92.33.3%0
6d1-ultra Liquid AITied for 1st94.591.3–97.7
91.55.4%0
6Drex v2.0 Nace AITied for 1st94.591.3–97.7
90.64.7%0
6Celeris-1 Decision CelerisTied for 1st94.591.3–97.7
91.15.1%0
9GPT-6 Luna Decisions OpenAITied for 1st93.590.1–96.9
90.82.2%0
9Drex Max Nace AITied for 1st93.590.1–96.9
89.12.7%0
11d1.1 Liquid AI92.588.8–96.2
89.57.8%0
12Mercury Decide Inception92.088.2–95.8
85.53.5%0
13Mammoth 1 Baseten91.587.6–95.4
84.81.9%0
13Dexy-Decision-9B Patrick-1891.587.6–95.4
86.14.1%0
15Elephant 1 Baseten91.087.0–95.0
84.43.5%0
16Clef Omni Cloudflare90.586.4–94.6
83.45.2%0
16Drex v1.5 Nace AI90.586.4–94.6
85.33.3%0
16Decider Flash V1.2 Perplexity90.586.4–94.6
86.39.3%0
19Jev 1.13 TypeSafe90.085.8–94.2
86.35.5%0
20Clef Cloudflare89.585.3–93.7
85.32.8%0
20Clef Flash Cloudflare89.585.3–93.7
84.33.2%0
22laya-ara Wouzein-domain88.584.1–92.9
83.19.1%0
22Kev 4B Jared Palmer88.584.1–92.9
83.46.8%0
22Microsoft-Decision-1 Microsoft88.584.1–92.9
82.84.5%0
25Jev 1.14 TypeSafe87.082.3–91.7
82.94.8%0
26Solar Decide Upstage86.081.2–90.8
76.38.8%0
26Decider 1 meraGPT86.081.2–90.8
71.814.7%0
28laya-ara-rag Wouze84.579.5–89.5
69.25.0%0
28Tev1 4B Experimental Together AI84.579.5–89.5
79.74.6%0
28Solar Decide Flash Upstage84.579.5–89.5
75.18.2%0
28d1-3B Liquid AI84.579.5–89.5
78.63.8%0
32Joey 1 Baseten83.077.8–88.2
58.24.4%0
33Laya Router Convai Innovations82.577.2–87.8
63.18.3%0
33Naluri-279M Jericho Siahayaunusually high82.577.2–87.8
74.74.1%0
35d3 lite vLLM Semantic Router80.575.0–86.0
67.35.1%0
36Classify-1 ScaleDown80.074.5–85.5
44.430.0%200
37Naluri-308M Jericho Siahaya79.573.9–85.1
74.25.8%0
38d3 nano vLLM Semantic Router79.073.4–84.6
73.03.9%0
39d1-omni-600M Liquid AI77.571.7–83.3
43.713.2%0
40d3 edge vLLM Semantic Router72.566.3–78.7
64.92.9%0
41Kev 0.8B Jared Palmer49.042.1–55.9
48.015.1%0

The bar is accuracy; the bracket on it is the 95% interval for 200 cases. Tied for 1st marks every model whose gap to the leader is within sampling noise (1.96 standard errors of the difference, treating the runs as independent). A failed call counts as wrong. In-domain models trained on another split of this dataset, by their maker's own account. Unusually high marks a score far above what that model's other datasets, and every other model's score here, predict (hover for the numbers). It often means the model trained on this dataset or one like it; it isn't proof, and the score still counts.

About this benchmark

Question
Offensive tweet, yes/no
Source
arbml/OSACT4_hatespeech, validation split
Sample
200 random tweets
Pinned
Every source is pinned to an exact commit, and the cases are frozen by hash in data.lock.json.

It counts towards the System One Index and its Arabic sub-index. Every model gets the same cases and the same request, under the benchmark rules.