We introduce Depth Hybrid Attention (DHA), which incorporates depth-wise value mixing into hybrid architectures with very little additional compute and cache overhead. By allowing layers to mix information across depth, DHA broadens each layer’s view and increases the model’s expressive capacity. Our DHA method adapts to multiple hybrid architectures, including Depth Window Attention (DWA) for GQA + SWA and Depth Delta Attention (DDA) for MLA + KDA. Across loss curves, short-context benchmark results, and long-context retrieval evaluations, both variants outperform their corresponding full-attention and hybrid-attention baselines, while retaining hybrid attention’s low KV-cache footprint. Project overview & code: