๊ฐ€์ž… ํ›„ ์ดˆ๋Œ€ ๋งํฌ๋ฅผ ๊ณต์œ ํ•˜๋ฉด ๋™์˜์ƒ ์žฌ์ƒ ๋ฐ ์ดˆ๋Œ€ ๋ณด์ƒ์„ ๋ฐ›์„ ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

khazzz1c
@Chonghan_Liu
Turning electricity into tokens | Building
๊ฐ€์ž… May 2021
402 ํŒ”๋กœ์ž‰ ์ค‘    1.1K ํŒฌ
Read xiaomi's hysparse2 paper and compared it with dpskv41-flash. the parts where they split are pretty interesting. Dpsk still compresses the sequence a bit, then uses an extra lightweight indexer to pick top-k kv. xiaomi skips both. every token stays, no extra indexer, the full-attn layers just hand token-level indices straight to the sparse layers. Kv sharing is different too. dpsk passes main kv + top-k indices across its full / reindex / reuse layers and stores that main kv in fp4. this part especially feels like a big setup being planted ๐Ÿ‘€ xiaomi keeps it simpler, just two-level cross-layer sharing on uncompressed kv. Tbh the reindex part always felt kinda odd to me. when i was adapting v41 i kept wondering why they went straight for reindex. re-scoring the reused kv at those layers feels a bit brute-force, and my gut is closer to xiaomi here: let the full-attn layers pick once and just reuse it. the sequence compression side of v41 is genuinely novel tho. Feels like dpsk is still trimming kv from a few directions at once, while xiaomi bets that keeping every token and letting the backbone do the selection is cleaner.
๋” ๋ณด๊ธฐ