Ash Hart
@ashxhart
Doing my bit to bring local AI to everyone | building | | Thoughts are my own.
๊ฐ์
October 2014
474 ํ๋ก์ ์ค 11.7K ํฌ
16GiB just for the chat cache.
For Llama 3.1 8B, that's the calculated size of a full 16-bit KV cache at 128K tokens. Weights and runtime go on top.
When someone says a model fits on a GPU, ask what context they're using.
What GPU and model are you trying to squeeze more out of? ๐
๋ ๋ณด๊ธฐ
