Anka Reuel | @ankareuel.bsky.social(@AnkaReuel):I‘m with @jungofthewon here. Especially for agents, there are so many moving pieces in deployment (tools, envs, other agents, humans, …) that an empirical predeployment evaluation has barely any validity for any downstream use case.

Anka Reuel | @ankareuel.bsky.social

@AnkaReuel

Incoming Assistant Prof @ Harvard (Public Policy & Computer Science) | CS PhD Candidate @ Stanford | Former Vice-Chair GPAI CoP @ European Commission

加入 March 2020

1.5K 正在關注 5.4K 粉絲

Anka Reuel | @ankareuel.bsky.social@AnkaReuel

2026.09.25 00:14

I‘m with @jungofthewon here. Especially for agents, there are so many moving pieces in deployment (tools, envs, other agents, humans, …) that an empirical predeployment evaluation has barely any validity for any downstream use case.

Jungwon@jungofthewon

2026.09.25 00:07

i disagree. i think we're hitting diminishing returns on static benchmarks. we need more algorithmic approaches and guarantees of quality. it's getting impossible to evaluate the scale of work agents are capable of now. in many cases, if you can create the benchmark you've solved the task.