Zyphra(@ZyphraAI):Today we’re introducing Zyphra Hierarchical Attention (ZHA), a path to true subquadratic attention. ZHA searches a learned tree of compressed summaries, with projected speedups of 35× over dense attention and 5× over MSA at 10M tokens, increasing to ~400× and ~55× at 1B. 🧵

2026.11.02 22:04

Today we’re introducing Zyphra Hierarchical Attention (ZHA), a path to true subquadratic attention. ZHA searches a learned tree of compressed summaries, with projected speedups of 35× over dense attention and 5× over MSA at 10M tokens, increasing to ~400× and ~55× at 1B. 🧵