Benjamin Marie(@bnjmn_marie):A proof that the benchmarks we are all using, are used to directly validate LLM training. In this case of Xiaomi live streaming, they can stop the training when the performance on deepswe, and all the benchmarks they target, is greater than some other baselines they have. This is not how benchmarks are supposed to be used. And that's why they are quickly "saturated" once released. It's obvious that most providers do something similar.

Benjamin Marie

@bnjmn_marie

Independent AI researcher (LLM, NLP). My blog, The Kaitchup - AI on a Budget:

加入 June 2019

221 正在关注 6.9K 粉丝

Benjamin Marie@bnjmn_marie

2026.09.17 16:30

A proof that the benchmarks we are all using, are used to directly validate LLM training. In this case of Xiaomi live streaming, they can stop the training when the performance on deepswe, and all the benchmarks they target, is greater than some other baselines they have. This is not how benchmarks are supposed to be used. And that's why they are quickly "saturated" once released. It's obvious that most providers do something similar.

显示更多