Register and share your invite link to earn from video plays and referrals.

cv usk
@cv_usk
AI / Software Research Notes AI Agent, LLMOps, MLOps, Software Architecture ๆŠ•็จฟใฏๅ€‹ไบบใฎๆ„่ฆ‹ใงใ™ใ€‚
Joined May 2026
286 Following    471 Followers
TL;DR: SO-Arena is a benchmark that measures System One models, which return a decision in a single forward pass, on both capability and reliability. A more capable model is not necessarily a more stable one. SO-Arena: Benchmarking the Capability Spectrum and Reliability of System One Models Key points ใƒป๐Ÿ“š 2,919 tasks across 11 domains, built from 40 public datasets ใƒปโšก Open-weight System One models answer within 0.3 s per task, tens of times faster than System Two ใƒป๐Ÿ“‰ Even the best System One trails GPT-6 Luna (79.7) by about 10 points ใƒป๐Ÿ”€ Order stability: does the answer survive reshuffling the options? ใƒป๐Ÿšซ Null discrimination: can the model tell when the correct option has been removed? ใƒป๐Ÿ” The most reliable model, JevAny-27B, ranks 11th of 21 in capability Simple semantic judgments are fine for small models, but multi-step derivation like math benefits greatly from thinking. If you use these for routing or verification, look at stability, not just accuracy. #LLMBenchmark# #SystemOne#
Show more