注册并分享邀请链接,可获得视频播放与邀请奖励。

Daniel Fein
@DanielFein7
researcher @valsai | stanford
加入 August 2020
1K 正在关注    601 粉丝
Beautiful demonstration of the persistent value of academia
Can an LM, starting from random init (!!), learn to generate all of its pretraining data? Introducing Self-Play Pretraining with Zero Data. Two models start from random initialization: a generator proposes programs for a universal Turing machine and a learner trains on their outputs. We never train on any real data, but see predictable scaling on natural datasets: zero-shot val loss on images, text, audio, and melodies decreases predictably with self-play compute. And the learner develops in-context learning capabilities. A fun proof-of-concept, co-led with @AdityaCowsik and @KfirDolev and co-authors @gbruno_dl, @ANourya @noahdgoodman, and @YoavLevine.
显示更多