登録して招待リンクを共有すると、動画再生報酬と紹介報酬を獲得できます。

Yusuke Miyashita
@miyashita_03
Decentralized pretraining @Pluralis | Prev: @SakanaAILabs I make manim videos for @bycloudai
参加 April 2016
567 フォロー中    698 ファン
Just reading about Canon layer(very similar to SConv) and it’s saying even random convolution weights help lol SConv mixes recent four tokens into a single token representation, adding local feature that attention can leverage. Random convolution weights work because it’s like when residual connection is introduced by Resnet. SConv seems to help gradient flow and Zeyuan says “promote horizontal information flow across neighboring tokens”. No wonder Zeyuan and Marin both inject so many SConv layers in their models. More SConv layers, more horizontal information flow in the model
もっと見る