Benjamin Marie(@bnjmn_marie):One of the most interesting tables in the DeepSeek V4.1 model card. Same model, same benchmark, wildly different scores depending on the agent harness. One more proof that comparing your new model’s score with previously published numbers is close to meaningless unless the harness and config are matched.

Benjamin Marie

@bnjmn_marie

Independent AI researcher (LLM, NLP). My blog, The Kaitchup - AI on a Budget:

Joined June 2019

221 Following 6.9K Followers

Benjamin Marie@bnjmn_marie

2026.09.11 22:04

One of the most interesting tables in the DeepSeek V4.1 model card. Same model, same benchmark, wildly different scores depending on the agent harness. One more proof that comparing your new model’s score with previously published numbers is close to meaningless unless the harness and config are matched.