註冊並分享邀請連結,可獲得影片播放與邀請獎勵。

Ash Hart
@ashxhart
Doing my bit to bring local AI to everyone  | building | | Thoughts are my own.
加入 October 2014
487 正在關注    13.8K 粉絲
16GiB just for the chat cache. For Llama 3.1 8B, that's the calculated size of a full 16-bit KV cache at 128K tokens. Weights and runtime go on top. When someone says a model fits on a GPU, ask what context they're using. What GPU and model are you trying to squeeze more out of? 👀
顯示更多