注册并分享邀请链接,可获得视频播放与邀请奖励。

Alexey Fateev
@superalesha
how far can 4x RTX 3090s go?
加入 January 2026
469 正在关注    8.3K 粉丝
A 27B model in 12 GB of VRAM. Full 262K context. 70 tok/s. And now it is UNCENSORED. I ran it on my own RTX 3090, held to 12 GB. This is Mirai S Qwen3.8-27B, 2.4 bits per weight, an 11 GB file. With my llama.cpp port the same card did 82K context at 38 tok/s inside 12 GB. Then @CaryPalmerr took the port and did the hard part. Same card, same file, his serve: - context: 82K -> 262,144 with q8 KV - decode: 38 -> 69.9 tok/s - VRAM peak: 11.7 GB - prefill: 1,124 tok/s He keeps part of the KV cache in VRAM and the rest in pinned RAM, and he drafts with MTP at every depth. He checked every kernel against my fork token for token and wrote down every experiment, the failed ones too. On his real 12 GB RTX 4070 it does 76.6. My part is small. I added abliteration. You cant edit these weights, they are compressed codes, so I measured the refusal direction on the quantized model and the engine removes it while the model runs. A 1.3 MB file and 1 flag: - harmful prompts refused: 64 of 64 -> 0 of 64 - held-out set: 78 of 82 -> 0 of 82 - decode: 69.9 -> 69.7 tok/s Counted by a regex, and with thinking on 1 of 24 still refuses. Deep in the context it gets slower on my card: 27.9 tok/s at 60K. So thats 12 GB of VRAM, 262K context, 70 tok/s, abliterated. Open weights, open code, runs on your desk. Cary did a genius amount of work and almost nobody has seen it. Go follow @CaryPalmerr right now, I mean it. serve: weights + vector:
显示更多
0
35
1.2K
92
转发到社区