注册并分享邀请链接,可获得视频播放与邀请奖励。

Yusuke Miyashita
@miyashita_03
Decentralized pretraining @Pluralis | Prev: @SakanaAILabs I make manim videos for @bycloudai
加入 April 2016
567 正在关注    698 粉丝
Just reading about Canon layer(very similar to SConv) and it’s saying even random convolution weights help lol SConv mixes recent four tokens into a single token representation, adding local feature that attention can leverage. Random convolution weights work because it’s like when residual connection is introduced by Resnet. SConv seems to help gradient flow and Zeyuan says “promote horizontal information flow across neighboring tokens”. No wonder Zeyuan and Marin both inject so many SConv layers in their models. More SConv layers, more horizontal information flow in the model
显示更多