Today's benchmarks serve a very different purpose compared to language modelling benchmarks in years prior. The reality is that no one serious is choosing to use model A over B simply because A is higher than B on some benchmark leaderboard. Models are so good now that model preference is often subjective, and it is virtually impossible to get a total ordering. As such, each benchmark simply serves as a measure of a very specific very narrow capability: - DeepSWE: Given some spec, can a model implement this feature? - FrontierCode: Are models able to come up with the spec for a good solution and then implement it themselves? - FrontierSWE: Are models able to do long horizon tasks? None of these single-handedly cover the space of coding ability. In fact, the space of what the models are tasked to do in real customers changes for each customer.