Benjamin Marie(@bnjmn_marie):The more I think about it, the worse it gets. We’re not just evaluating a harness–model pair. We’re evaluating an adapter–harness–model triplet. The adapter is what you can easily benchmaxx. Take DeepSWE: Pi running through a basic Pi-to-Pier adapter would perform much worse than the same harness and model running through a carefully tuned adapter. Same model, same harness, different integration, potentially very different results. The adapter is rarely published. We really do have a benchmarking problem.

2026.09.12 02:05

The more I think about it, the worse it gets. We’re not just evaluating a harness–model pair. We’re evaluating an adapter–harness–model triplet. The adapter is what you can easily benchmaxx. Take DeepSWE: Pi running through a basic Pi-to-Pier adapter would perform much worse than the same harness and model running through a carefully tuned adapter. Same model, same harness, different integration, potentially very different results. The adapter is rarely published. We really do have a benchmarking problem.

Benjamin Marie@bnjmn_marie

2026.09.11 22:04

One of the most interesting tables in the DeepSeek V4.1 model card. Same model, same benchmark, wildly different scores depending on the agent harness. One more proof that comparing your new model’s score with previously published numbers is close to meaningless unless the harness and config are matched.