wh(@nrehiew_):Sparse Attention with Indexer also seems rather standard. Essentially, we use an indexer at block level in a compressed latent space. For training, the indexer is asked to predict the full attention score 1) Distill the dense attention scores into the indexer 2) KL loss on the indexer vs the full attention teacher

@nrehiew_

eng primarily, ml mostly, research previously

Joined October 2023

104 Following 18.5K Followers

wh@nrehiew_

2026.08.27 02:50

Sparse Attention with Indexer also seems rather standard. Essentially, we use an indexer at block level in a compressed latent space. For training, the indexer is asked to predict the full attention score 1) Distill the dense attention scores into the indexer 2) KL loss on the indexer vs the full attention teacher