eric provencher(@pvncher):While you’re absolutely correct that these routers don’t make sense, I’ve fully soured on running benchmarks that just test a bunch of tiny tasks in isolation to evaluate ideas like this. In the real world users run prompts that can take an hour+ to run. The model has to decipher your ambiguous prompt, navigate your messy repo, and then iterate until it’s satisfied. This takes compactions, and sometimes steers from you. Compare that to a deepswe task that’s pretty clear upfront, and that’s so trivial it costs the model a buck fifty in inference to complete.

2026.09.26 14:48

While you’re absolutely correct that these routers don’t make sense, I’ve fully soured on running benchmarks that just test a bunch of tiny tasks in isolation to evaluate ideas like this. In the real world users run prompts that can take an hour+ to run. The model has to decipher your ambiguous prompt, navigate your messy repo, and then iterate until it’s satisfied. This takes compactions, and sometimes steers from you. Compare that to a deepswe task that’s pretty clear upfront, and that’s so trivial it costs the model a buck fifty in inference to complete.

显示更多

145

转发到社区

热门用户