註冊並分享邀請連結,可獲得影片播放與邀請獎勵。

Steve Jurvetson
@FutureJurvetson
Co-founder of Future Ventures and DFJ, supporting passionate founders to forge a better future. Early VC investor in Tesla, SpaceX, Planet, Commonwealth Fusion.
加入 March 2010
70 正在關注    112.4K 粉絲
As in the brain, "ultra-long contexts may require subquadratic attention. Attend to everything, and the signal gets lost in noise. Too sparse, and we lose the signal. Hierarchical attention may be essential not just for efficiency, but for attention to work at all." — Zyphra
顯示更多
Today we’re introducing Zyphra Hierarchical Attention (ZHA), a path to true subquadratic attention. ZHA searches a learned tree of compressed summaries, with projected speedups of 35× over dense attention and 5× over MSA at 10M tokens, increasing to ~400× and ~55× at 1B. 🧵
顯示更多