Jasper Lu(@lu__jasper):Good read on Engram layers and why Chinese labs are gravitating towards them in their models out of necessity (they are HBM starved). Sharing one more data point on Engram scaling: whereas Deepseek finds that trading off 20% of MoE params for Engram leads to all around improvements, Qwen team finds that downstream perf is approximately a wash at all choices of Engram vocab size.

Jasper Lu

@lu__jasper

continually learning. formerly taught models to design @figma, to see @nuro

Joined July 2009

331 Following 2.5K Followers

Jasper Lu@lu__jasper

2026.09.27 20:46

Good read on Engram layers and why Chinese labs are gravitating towards them in their models out of necessity (they are HBM starved). Sharing one more data point on Engram scaling: whereas Deepseek finds that trading off 20% of MoE params for Engram leads to all around improvements, Qwen team finds that downstream perf is approximately a wash at all choices of Engram vocab size.