Steve Jurvetson(@FutureJurvetson):As in the brain, "ultra-long contexts may require subquadratic attention. Attend to everything, and the signal gets lost in noise. Too sparse, and we lose the signal. Hierarchical attention may be essential not just for efficiency, but for attention to work at all."

2026.11.02 23:53

As in the brain, "ultra-long contexts may require subquadratic attention. Attend to everything, and the signal gets lost in noise. Too sparse, and we lose the signal. Hierarchical attention may be essential not just for efficiency, but for attention to work at all." — Zyphra

Zyphra@ZyphraAI

2026.11.02 22:04

Today we’re introducing Zyphra Hierarchical Attention (ZHA), a path to true subquadratic attention. ZHA searches a learned tree of compressed summaries, with projected speedups of 35× over dense attention and 5× over MSA at 10M tokens, increasing to ~400× and ~55× at 1B. 🧵