TL;DR: Classifier-style "decision models" can replace LLM judges at 16 to 357 times lower cost, but they get things wrong in the same places the LLMs do. Decision Models vs. LLMs as Rubric Judges: Cheaper, Faster, and Wrong in the Same Places Key points ・⚖️ Compares Jev and Perplexity's Decider against 3 flash-tier LLM judges on 7 benchmarks, 9 panels, 5,003 items ・💸 LLM judges cost 16 to 357x more and take 5 to 350x longer ・✅ Accuracy: significantly worse in at most 6 of 27 comparisons, and often stronger on binary panels ・📉 Weaker on ordinal scales, scoring lower than human raters ・🔁 When Jev is confidently wrong, 96.0% of LLM verdicts repeat the error (50.3% expected by chance) ・🪜 Cascades gain at most +2.7 points, and just +1.7 even with higher-tier fallbacks Swap them in to save cost; "defer to an LLM when unsure" is a weak accuracy safety net. #LLMasaJudge# #LLMEvaluation#