Register and share your invite link to earn from video plays and referrals.

Aakash Sabharwal
@aakashsabharwal
prev: VP of Eng @scale_ai @ScaleAILabs
Joined September 2009
3.9K Following    1.7K Followers
Excellent analysis from the @preferencemodel team. Just shows how hard it is to build truly hard yet fair benchmarks. Four takeaways for me: 1. Domain expertise ≠ verifier expertise. Creating meaningful tasks and designing reliable graders are different skills. Data teams need both and failure is to index on one vs the other. 2. The agent just needs one loophole but the grader has to cover them all. Like cybersecurity, agents are both the problem and the solution. use them to continuously attack and harden your graders/verifiers. 3. just look at the data / actual runs. Did the model fail, or did the grader reject a valid solution? Did it solve the task, or just pass the tests? Catching both still takes expert judgment. Agents can help but no substitute for looking at raw traces. 4. one thing I am not sold on 100%. Ambiguity isn’t always a flaw. You don’t specify every detail to your coworkers, you expect judgment and them to make reasonable assumptions. Enterprise agents need that too. Diff between that and arbitrary hidden requirements.
Show more
We looked at TerminalBench 4.0 and found a lot of issues. After fixing the issues Sonnet 5.5 scores 89.2%. So, TerminalBench is already mostly saturated. Along the way we found a lot of crazy stuff, including reward hacks in the published transcripts.
Show more