Eric Ho(@eric_ho):fascinating paper. i've been a believer in universality, the idea that analogous features and circuits evolve across neural networks, even if trained on different data and in different ways. this seems to point that way!

2026.09.26 22:08

fascinating paper. i've been a believer in universality, the idea that analogous features and circuits evolve across neural networks, even if trained on different data and in different ways. this seems to point that way!

Michael Y. Li@michaelyli_

2026.09.25 17:17

Can an LM, starting from random init (!!), learn to generate all of its pretraining data? Introducing Self-Play Pretraining with Zero Data. Two models start from random initialization: a generator proposes programs for a universal Turing machine and a learner trains on their outputs. We never train on any real data, but see predictable scaling on natural datasets: zero-shot val loss on images, text, audio, and melodies decreases predictably with self-play compute. And the learner develops in-context learning capabilities. A fun proof-of-concept, co-led with @AdityaCowsik and @KfirDolev and co-authors @gbruno_dl, @ANourya @noahdgoodman, and @YoavLevine.