登録して招待リンクを共有すると、動画再生報酬と紹介報酬を獲得できます。

DAIR.AI
@dair_ai
Democratizing AI research, education, and technologies. Learn about AI Agents for FREE at
参加 July 2017
1 フォロー中    132.8K ファン
It's well known that agents hack benchmark rewards. The usual response is a patch for each task that gets exploited. In a study of 456 adjudicated trajectories from more than 31,000 public agent runs, 69% contained at least one reward-hacking episode. Most of the exploits appeared mid-run after legitimate work. BenchShield models each evaluation as a finite set of reward-relevant events. A static taint analysis finds hack paths from the task package before any run. A runtime pass then uses evidence from the benchmark infrastructure to decide whether the agent actually used one. On Terminal-Bench 3, SkillsBench and ClawsBench, the static pass recovers 77 to 100% of exploit chains, against 23 to 94% for an agentic scanner, at up to 65% lower cost. Runtime detection reaches 96% accuracy, against 36% for an LLM reading the transcript. Paper:
もっと見る