Anka Reuel | @ankareuel.bsky.social(@AnkaReuel):I‘m with @jungofthewon here. Especially for agents, there are so many moving pieces in deployment (tools, envs, other agents, humans, …) that an empirical predeployment evaluation has barely any validity for any downstream use case.

Anka Reuel | @ankareuel.bsky.social@AnkaReuel

2026.09.25 00:14

I‘m with @jungofthewon here. Especially for agents, there are so many moving pieces in deployment (tools, envs, other agents, humans, …) that an empirical predeployment evaluation has barely any validity for any downstream use case.

Jungwon@jungofthewon

2026.09.25 00:07

i disagree. i think we're hitting diminishing returns on static benchmarks. we need more algorithmic approaches and guarantees of quality. it's getting impossible to evaluate the scale of work agents are capable of now. in many cases, if you can create the benchmark you've solved the task.

Forward to community