TL;DR: SO-Arena is a benchmark that measures System One models, which return a decision in a single forward pass, on both capability and reliability. A more capable model is not necessarily a more stable one.
SO-Arena: Benchmarking the Capability Spectrum and Reliability of System One Models
Key points
ใป๐ 2,919 tasks across 11 domains, built from 40 public datasets
ใปโก Open-weight System One models answer within 0.3 s per task, tens of times faster than System Two
ใป๐ Even the best System One trails GPT-6 Luna (79.7) by about 10 points
ใป๐ Order stability: does the answer survive reshuffling the options?
ใป๐ซ Null discrimination: can the model tell when the correct option has been removed?
ใป๐ The most reliable model, JevAny-27B, ranks 11th of 21 in capability
Simple semantic judgments are fine for small models, but multi-step derivation like math benefits greatly from thinking. If you use these for routing or verification, look at stability, not just accuracy.
#
LLMBenchmark# #
SystemOne#