wh(@nrehiew_):Next big section is on residuals which has devolved to something extremely extremely cursed. They first compare HyperConnections (learnable write, read, mix operators) against a simplied variant where its just a linear combination of various branches and the output is written to a single branch in a roundrobin fashion. They compare both of these with their own variant GatedResidual: 1) RMSNorm each branch 2) Apply projection via a low rank bottleneck 3) Layer is the input coefficients of the different branches 4) Write uses a single branch level scalar

2026.08.27 02:50

Next big section is on residuals which has devolved to something extremely extremely cursed. They first compare HyperConnections (learnable write, read, mix operators) against a simplied variant where its just a linear combination of various branches and the output is written to a single branch in a roundrobin fashion. They compare both of these with their own variant GatedResidual: 1) RMSNorm each branch 2) Apply projection via a low rank bottleneck 3) Layer is the input coefficients of the different branches 4) Write uses a single branch level scalar

显示更多