Did softmax and cross-entropy have numerical bugs nobody actually fixed? A new paper from Google DeepMind says yes. Title: A Fix for Softmax and Cross-Entropy URL: ❓ What is the problem? Subtracting the max logit prevents overflow, but two bugs remain. In Softmax Collapse, the target gradient rounds to zero once the correct logit runs far ahead. In Softmax Cancellation, distillation subtracts nearly equal probabilities. ❓ How is it fixed? By rewriting in log space. The max logit is set to 0, the loss is softplus(-m), and the target gradient is defined as the negated sum of the others, so gradients always sum to zero. The distillation side uses expm1 and log1p to avoid the subtraction. Short JAX code is a drop-in replacement. ❓ Does it help? On a²+b² mod 97 grokking, standard test accuracy drops from 61% to 23% at the precision cliff, while the fix reaches 85%. In soft-target distillation, the fix keeps KL at exactly 0 in every precision. ❓ What did we learn? Slingshot spikes and grokking failures have a numerical cause, and the margin ceiling is about log(1/ε) from Adam's ε and weight decay. The experiments are small-scale, though, and large-scale impact is untested. #DeepLearning# #MachineLearning#