During the run, we wanted a clearer picture of model progress and monitored both partial credit and the gain on only the fair tasks of DeepSWE.
We often saw pass
@1 gains trailing behind pass
@8. And it turns out that the pass
@8 gains were the model simply lucking into a correct solution on the unfair tasks.
We found that both these approaches were more representative of capabilities beyond binary reward.