๊ฐ€์ž… ํ›„ ์ดˆ๋Œ€ ๋งํฌ๋ฅผ ๊ณต์œ ํ•˜๋ฉด ๋™์˜์ƒ ์žฌ์ƒ ๋ฐ ์ดˆ๋Œ€ ๋ณด์ƒ์„ ๋ฐ›์„ ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

David Hendrickson
@TeksEdge
CEO & Founder | PhD | Startup Advisor | @Columbia | Author Generative Software Engineering | ๐Ÿ”” Follow for AI & Vibe Coding Tips ๐Ÿ‘‡
๊ฐ€์ž… July 2023
552 ํŒ”๋กœ์ž‰ ์ค‘    12.5K ํŒฌ
A new dense 32B model ran a 239K-token conversation inside a 16GB GPU-memory budget. Agens Volundr 32B v1 compares well to Qwen3.8-27B. And the trick isn't just the ~2.8-bit quant. 54 of its 72 layers don't build a growing KV cache at all. ๐Ÿ‘€ Blockway released Agens Volundr 32B v1 today. Stats ๐Ÿง  32B parameters ๐Ÿ“š 262K configured context ๐Ÿ’พ ~2.82 bits/weight on the 16GB build ๐Ÿ”“ Apache 2.0 weights โš™๏ธ MIT-licensed Agens Engine ๐Ÿ“ฆ Official GGUFs already available Special Source ๐Ÿ‘‰ instead of every layer accumulating KV as context grows 72 layers โ†“ 54 keep fixed-size recurrent state โ†“ only 18 layers maintain growing KV cache That helps with long-context memory Using Agens Engine's hard 16 GiB GPU-memory cap, Blockway fed it a 219K-token document and reports at T=0.6 โšก Chat โ†’ 57 tps โšก Code โ†’ 79 tps โšก Thinking โ†’ 71 tps โšก Tool calls โ†’ 67 tps They pushed the limits of the 239K-token envelope while GPU memory peaked around 15.5 GiB. At short context, the same profile reaches roughly 86โ€“123 tps depending on workload. And there's already a separate 24GB profile using the 4-bit weights for cards in the RTX 4090 class. ๐ŸŽฏ Volundr attacks the other monster in the room, : the memory cost of long context. โš ๏ธ Those published 16GB and 24GB performance numbers were measured on a 48GB GPU running under hard memory caps, not yet on physical 16GB/24GB consumer cards. The official GGUFs work today with Blockway's llama.cpp fork. Stock llama.cpp support is still being upstreamed. Link in ALT
๋” ๋ณด๊ธฐ