system-one-bench

~/system-one-bench / benchmarks/banking77

Text track · English · All benchmarks

Banking77

Banking intent, 77 options. Clef Flash has the top score, 2.5 points ahead of Clef Omni, but Clef Omni and 1 more are within sampling noise: a statistical tie for first. But its score is unusually high for Clef Flash: 98.5%, 10 points above the 88% its other datasets predict.

Top score
Clef Flash
98.5, tied with 2 within noise
Lead over 2nd
2.5
Points of accuracy
Models
39
Charted on the leaderboard
Cases
200
Best constant answer
3.5
Always giving the most common label
Random guess
1.3
Uniform over the options

Leaderboard

RankModelAccuracy95% intervalMacro-F1Calibration errorFailed
1Clef Flash CloudflareLeaderunusually high98.596.8–100.0
98.06.2%0
2Clef Omni CloudflareTied for 1st96.093.3–98.7
95.819.9%0
3Clef CloudflareTied for 1st95.592.6–98.4
95.24.6%0
4Mammoth 1 Baseten94.591.3–97.7
94.22.8%0
4Elephant 1 Baseten94.591.3–97.7
93.22.9%0
6d3 nano vLLM Semantic Router93.089.5–96.5
89.35.0%0
6Joey 1 Baseten93.089.5–96.5
92.33.2%0
6Dexy-Decision-9B Patrick-18in-domain93.089.5–96.5
92.63.1%0
9GLiDE Fastino91.587.6–95.4
90.23.8%0
10d3 lite vLLM Semantic Router91.087.0–95.0
90.07.1%0
11Decider Flash V1.2 Perplexity90.586.4–94.6
89.64.7%0
11Drex Max Nace AI90.586.4–94.6
88.94.4%0
13d3 edge vLLM Semantic Router90.085.8–94.2
86.51.8%0
14d1 Liquid AI89.084.7–93.3
86.13.6%0
14Celeris-1 Decision Celeris89.084.7–93.3
85.53.3%0
16Kev 4B Jared Palmerin-domain88.083.5–92.5
84.612.2%0
16d1-3B Liquid AI88.083.5–92.5
84.73.1%0
16Decider V1.2 Perplexity88.083.5–92.5
85.63.7%0
19Microsoft-Decision-1 Microsoft87.082.3–91.7
84.43.8%0
19Drex v1.5 Nace AI87.082.3–91.7
83.76.7%0
19d1.1 Liquid AI87.082.3–91.7
84.67.1%0
22Mercury Decide Inception86.081.2–90.8
80.48.9%0
23d1-omni-600M Liquid AI84.579.5–89.5
80.34.0%0
24d1-ultra Liquid AI83.578.4–88.6
80.76.6%0
25Span-02 Respan82.076.7–87.3
77.65.3%0
26Kev 0.8B Jared Palmerin-domain80.575.0–86.0
76.011.4%0
26Drex v2.0 Nace AI80.575.0–86.0
77.13.2%0
28Gemini Decision 1 Google79.573.9–85.1
78.08.3%0
29Jev 1.13 TypeSafe79.073.4–84.6
75.110.1%0
30Jev 1.14 TypeSafe78.572.8–84.2
76.610.4%0
31GPT-6 Luna Decisions OpenAI77.571.7–83.3
73.68.7%0
32Laya Router Convai Innovations44.037.1–50.9
35.849.0%0
33Naluri-279M Jericho Siahaya31.024.6–37.4
24.611.1%0
34Naluri-308M Jericho Siahaya27.020.8–33.2
21.39.5%0
35Solar Decide Upstage1.00.0–2.4
0.00.3%200
35Tev1 4B Experimental Together AI1.00.0–2.4
0.00.3%200
35Solar Decide Flash Upstage1.00.0–2.4
0.00.3%200
35Classify-1 ScaleDown1.00.0–2.4
0.00.3%200
35Decider 1 meraGPT1.00.0–2.4
0.00.3%200

The bar is accuracy; the bracket on it is the 95% interval for 200 cases. Tied for 1st marks every model whose gap to the leader is within sampling noise (1.96 standard errors of the difference, treating the runs as independent). A failed call counts as wrong. In-domain models trained on another split of this dataset, by their maker's own account. Unusually high marks a score far above what that model's other datasets, and every other model's score here, predict (hover for the numbers). It often means the model trained on this dataset or one like it; it isn't proof, and the score still counts.

About this benchmark

Question
Banking intent, 77 options
Source
mteb/banking77, test split
Sample
200 random messages, all 77 intents as options
Pinned
Every source is pinned to an exact commit, and the cases are frozen by hash in data.lock.json.

It counts towards the System One Index and its English sub-index. Every model gets the same cases and the same request, under the benchmark rules.