Binkg(@dummerspast39):Only 6 of ERNIE 5.5 Flash Base's 43 layers see every token individually >the other 37 squash every 128 tokens into 1 cached entry, plus a short window of recent tokens >those cheap layers get wide 512-dim heads, the 6 full-context layers get 256-dim >the 6 full-context layers use sparse attention, only over the top-k scored positions >still beats DeepSeek V4 Flash Base on 4 of 6 long-context benchmarks Baidu built the KV cache on a diet

2026.11.06 12:25

Only 6 of ERNIE 5.5 Flash Base's 43 layers see every token individually >the other 37 squash every 128 tokens into 1 cached entry, plus a short window of recent tokens >those cheap layers get wide 512-dim heads, the 6 full-context layers get 256-dim >the 6 full-context layers use sparse attention, only over the top-k scored positions >still beats DeepSeek V4 Flash Base on 4 of 6 long-context benchmarks Baidu built the KV cache on a diet