登録して招待リンクを共有すると、動画再生報酬と紹介報酬を獲得できます。

Aakash Sabharwal
@aakashsabharwal
prev: VP of Eng @scale_ai @ScaleAILabs
参加 September 2009
3.9K フォロー中    1.7K ファン
Excellent analysis from the @preferencemodel team. Just shows how hard it is to build truly hard yet fair benchmarks. Four takeaways for me: 1. Domain expertise ≠ verifier expertise. Creating meaningful tasks and designing reliable graders are different skills. Data teams need both and failure is to index on one vs the other. 2. The agent just needs one loophole but the grader has to cover them all. Like cybersecurity, agents are both the problem and the solution. use them to continuously attack and harden your graders/verifiers. 3. just look at the data / actual runs. Did the model fail, or did the grader reject a valid solution? Did it solve the task, or just pass the tests? Catching both still takes expert judgment. Agents can help but no substitute for looking at raw traces. 4. one thing I am not sold on 100%. Ambiguity isn’t always a flaw. You don’t specify every detail to your coworkers, you expect judgment and them to make reasonable assumptions. Enterprise agents need that too. Diff between that and arbitrary hidden requirements.
もっと見る
We looked at TerminalBench 4.0 and found a lot of issues. After fixing the issues Sonnet 5.5 scores 89.2%. So, TerminalBench is already mostly saturated. Along the way we found a lot of crazy stuff, including reward hacks in the published transcripts.
もっと見る