Liam Fedus says an AI model that memorized the answer in pretraining can "fake work" in training, and what it learns won't generalize. "If the pre-trained model has already memorized some of this data, and then you try to do reinforcement learning on that, if it already knows the answer and you construct reinforcement learning tasks that depend on that answer, it can fake work." "It already knows the answer, so it doesn't actually have to do the hard physics reasoning. It doesn't have to actually do the calculation simulations, and it gets to the right answer." "And then you reinforce that policy and you upweight those reasoning strategies. Those reasoning strategies will not generalize to novel systems." His fix is a timestamped record of lab evidence: "You could say at this date, this was our experimental evidence up until that point." - Liam Fedus, cofounder of Periodic Labs, on Latent Space with @swyx