注册并分享邀请链接,可获得视频播放与邀请奖励。

Nanbeige
@nanbeige
Nanbeige LLM Lab, BOSS Zhipin (Kanzhun Limited)
加入 December 2023
436 正在关注    5.7K 粉丝
We introduce Depth Hybrid Attention (DHA), which incorporates depth-wise value mixing into hybrid architectures with very little additional compute and cache overhead. By allowing layers to mix information across depth, DHA broadens each layer’s view and increases the model’s expressive capacity. Our DHA method adapts to multiple hybrid architectures, including Depth Window Attention (DWA) for GQA + SWA and Depth Delta Attention (DDA) for MLA + KDA. Across loss curves, short-context benchmark results, and long-context retrieval evaluations, both variants outperform their corresponding full-attention and hybrid-attention baselines, while retaining hybrid attention’s low KV-cache footprint. Project overview & code:
显示更多
0
11
218
31
转发到社区