Larry Dial(@classiclarryd):New NanoGPT Speedrun WR at 74.6s (−0.8s), by Jan Varho (jvarho on GitHub). The clever idea: if the target is the token " bananas", give partial credit for predicting " banana". More generally, during early training, add an auxiliary loss on the longest token-prefix of each target token.

Larry Dial

@classiclarryd

Technical Staff at Open Athena, working on Marin

Joined May 2024

52 Following 2.8K Followers

Larry Dial@classiclarryd

2026.08.02 16:59

New NanoGPT Speedrun WR at 74.6s (−0.8s), by Jan Varho (jvarho on GitHub). The clever idea: if the target is the token " bananas", give partial credit for predicting " banana". More generally, during early training, add an auxiliary loss on the longest token-prefix of each target token.