Ash Hart(@ashxhart):16GiB just for the chat cache. For Llama 3.1 8B, that's the calculated size of a full 16-bit KV cache at 128K tokens. Weights and runtime go on top. When someone says a model fits on a GPU, ask what context they're using. What GPU and model are you trying to squeeze more out of? 👀

Ash Hart

@ashxhart

Doing my bit to bring local AI to everyone | building | | Thoughts are my own.

参加 October 2014

474 フォロー中 11.7K ファン

Ash Hart@ashxhart

2026.10.27 23:17

16GiB just for the chat cache. For Llama 3.1 8B, that's the calculated size of a full 16-bit KV cache at 128K tokens. Weights and runtime go on top. When someone says a model fits on a GPU, ask what context they're using. What GPU and model are you trying to squeeze more out of? 👀