註冊並分享邀請連結,可獲得影片播放與邀請獎勵。

Yusuke Miyashita
@miyashita_03
Decentralized pretraining @Pluralis | Prev: @SakanaAILabs I make manim videos for @bycloudai
加入 April 2016
567 正在關注    698 粉絲
Just reading about Canon layer(very similar to SConv) and it’s saying even random convolution weights help lol SConv mixes recent four tokens into a single token representation, adding local feature that attention can leverage. Random convolution weights work because it’s like when residual connection is introduced by Resnet. SConv seems to help gradient flow and Zeyuan says “promote horizontal information flow across neighboring tokens”. No wonder Zeyuan and Marin both inject so many SConv layers in their models. More SConv layers, more horizontal information flow in the model
顯示更多