注册并分享邀请链接,可获得视频播放与邀请奖励。

Anka Reuel | @ankareuel.bsky.social
@AnkaReuel
Incoming Assistant Prof @ Harvard (Public Policy & Computer Science) | CS PhD Candidate @ Stanford | Former Vice-Chair GPAI CoP @ European Commission
加入 March 2020
1.5K 正在关注    5.4K 粉丝
I‘m with @jungofthewon here. Especially for agents, there are so many moving pieces in deployment (tools, envs, other agents, humans, …) that an empirical predeployment evaluation has barely any validity for any downstream use case.
显示更多
i disagree. i think we're hitting diminishing returns on static benchmarks. we need more algorithmic approaches and guarantees of quality. it's getting impossible to evaluate the scale of work agents are capable of now. in many cases, if you can create the benchmark you've solved the task.
显示更多