注册并分享邀请链接,可获得视频播放与邀请奖励。

Steve Jurvetson
@FutureJurvetson
Co-founder of Future Ventures and DFJ, supporting passionate founders to forge a better future. Early VC investor in Tesla, SpaceX, Planet, Commonwealth Fusion.
加入 March 2010
70 正在关注    112.4K 粉丝
As in the brain, "ultra-long contexts may require subquadratic attention. Attend to everything, and the signal gets lost in noise. Too sparse, and we lose the signal. Hierarchical attention may be essential not just for efficiency, but for attention to work at all." — Zyphra
显示更多
Today we’re introducing Zyphra Hierarchical Attention (ZHA), a path to true subquadratic attention. ZHA searches a learned tree of compressed summaries, with projected speedups of 35× over dense attention and 5× over MSA at 10M tokens, increasing to ~400× and ~55× at 1B. 🧵
显示更多