註冊並分享邀請連結,可獲得影片播放與邀請獎勵。

Max For AI
@MaxForAI
Head of hype @lobehub Prev @listenhub @Sapient_Int 没啥好看的,也就发点AI相关的内容🫡 同名小红书4万粉丝,公众号01Founder(长文首发) 观点仅代表个人 更多请访问网站⬇️(欢迎商务、合作
加入 September 2023
3K 正在關注    40.1K 粉絲
果然是从DS去的MiMo啊
Read xiaomi's hysparse2 paper and compared it with dpskv41-flash. the parts where they split are pretty interesting. Dpsk still compresses the sequence a bit, then uses an extra lightweight indexer to pick top-k kv. xiaomi skips both. every token stays, no extra indexer, the full-attn layers just hand token-level indices straight to the sparse layers. Kv sharing is different too. dpsk passes main kv + top-k indices across its full / reindex / reuse layers and stores that main kv in fp4. this part especially feels like a big setup being planted 👀 xiaomi keeps it simpler, just two-level cross-layer sharing on uncompressed kv. Tbh the reindex part always felt kinda odd to me. when i was adapting v41 i kept wondering why they went straight for reindex. re-scoring the reused kv at those layers feels a bit brute-force, and my gut is closer to xiaomi here: let the full-attn layers pick once and just reuse it. the sequence compression side of v41 is genuinely novel tho. Feels like dpsk is still trimming kv from a few directions at once, while xiaomi bets that keeping every token and letting the backbone do the selection is cleaner.
顯示更多