as these potential hacks are well discussed in their tech report, it is a bit hard to believe at first that they left the answer recoverable in 2/3 of the coding tasks.
but after going back through their tech report, there might really be a few things that can be improved in their RL environments.
The first is how they removed the reference fix. Their tech report says each task repo is set back to right before the fix, and the later commits are removed. However, according to the audit, only the branches got deleted. And in Git a branch is just a name, so the commit with the full fix was still sitting in the repo's storage. The MiMo team's own check only looks at the names, so it came back clean when the repo was not. Deleting the leftover commits and having the check look at the storage itself would close this.
The second is the "below 2%" hack rate from their tech report. That is the share of training runs where a grader caught the model copying an answer. Runs that copied without getting caught just counted as passes. So a clearer explanation of how the grader audits each run would help explain the gap between the 2% hack rate and the 2/3 leak rate.
The third is the cleanup list. Files touched by the fix had a later "last modified" time than the rest of the repo, and the audit shows the model using that to figure out where to edit. Timestamps are not on their cleanup list, so adding them would help.
Xiaomi open-sourced the RL environments behind MiMo v2.6. In two-thirds of the coding tasks, the answer is still in the task's Git history, and MiMo finds it.