Good read on Engram layers and why Chinese labs are gravitating towards them in their models out of necessity (they are HBM starved).
Sharing one more data point on Engram scaling: whereas Deepseek finds that trading off 20% of MoE params for Engram leads to all around improvements, Qwen team finds that downstream perf is approximately a wash at all choices of Engram vocab size.
もっと見る
