eric provencher
@pvncher
加入 June 2013
4.9K 正在關注 45.3K 粉絲
Model benchmarks are completely broken.
They don't reflect real world use at all. Stop putting so much trust in them
While you’re absolutely correct that these routers don’t make sense, I’ve fully soured on running benchmarks that just test a bunch of tiny tasks in isolation to evaluate ideas like this.
In the real world users run prompts that can take an hour+ to run. The model has to decipher your ambiguous prompt, navigate your messy repo, and then iterate until it’s satisfied. This takes compactions, and sometimes steers from you.
Compare that to a deepswe task that’s pretty clear upfront, and that’s so trivial it costs the model a buck fifty in inference to complete.
顯示更多
