Yusuke Miyashita
@miyashita_03
567 Following 698 Followers
Just reading about Canon layer(very similar to SConv) and it’s saying even random convolution weights help lol
SConv mixes recent four tokens into a single token representation, adding local feature that attention can leverage.
Random convolution weights work because it’s like when residual connection is introduced by Resnet. SConv seems to help gradient flow and Zeyuan says “promote horizontal information flow across neighboring tokens”. No wonder Zeyuan and Marin both inject so many SConv layers in their models. More SConv layers, more horizontal information flow in the model
Show more
