登録して招待リンクを共有すると、動画再生報酬と紹介報酬を獲得できます。

DAIR.AI
@dair_ai
Democratizing AI research, education, and technologies. Learn about AI Agents for FREE at
参加 July 2017
1 フォロー中    132.8K ファン
Banger paper introducing Jev-as-a-Judge. The overall finding is that you want to use a cheap judge for most of your evals and send only the uncertain calls to a frontier model. This paper measures how well that works with JEV, TypeSafe AI's decision-only judge. On 510 held-out preference pairs, a cascade that accepted JEV's confident verdicts and escalated the rest to GPT-6 Astra kept 99% of GPT-6's accuracy at about 57% of its fee. JEV returns a verdict and label probabilities with no reasoning text. It costs $0.044 per 1,000 judgments at a median latency of 0.152 seconds, against $12.182 and 1.885 seconds for GPT-6, about 277 times cheaper. On ordinary preference and evidence-grounded factuality it stays within 3 points of GPT-6 (92.2% against 93.5% on RewardBench, 87.5% against 86.7% on HaluEval). The gap grows to 9 to 20 points on tasks that require checking a derivation or rejecting an elaborately written wrong answer, such as JudgeBench (78.6% against 93.1%). On several benchmarks, JEV's gap to GPT-6 is concentrated in its low-confidence decisions, which is why the cascade works. The escalation threshold did not transfer for every fallback model, so the authors recommend setting it on your own data. Paper:
もっと見る