註冊並分享邀請連結,可獲得影片播放與邀請獎勵。

Ivan Fioravanti
@ivanfioravanti
LLM addicted, Apple MLX, NVIDIA DGX, Cloud, Kubernetes, Technology Advisor, Investor and Co-Founder / Board Member of CoreView. Local AI Ambassador.
加入 June 2009
1.5K 正在關注    45.4K 粉絲
The more I dig in the various inference engines out there, the more I see the shortcuts they take for speed. Don't assume that model layers are mapped 1-1 into your engine, some are quantizing before running, 8bit and even 4bit. I think I should post more on this, not to finger point anyone, but just to clarify that: more often than not, you bargain precision for speed. Kernel optimizations are great, but there is a physical limit (compute/memory bandwidth). There is no free lunch 🤷🏻‍♂️
顯示更多
0
21
109
6
轉發到社區