๊ฐ€์ž… ํ›„ ์ดˆ๋Œ€ ๋งํฌ๋ฅผ ๊ณต์œ ํ•˜๋ฉด ๋™์˜์ƒ ์žฌ์ƒ ๋ฐ ์ดˆ๋Œ€ ๋ณด์ƒ์„ ๋ฐ›์„ ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

Ivan Fioravanti
@ivanfioravanti
LLM addicted, Apple MLX, NVIDIA DGX, Cloud, Kubernetes, Technology Advisor, Investor and Co-Founder / Board Member of CoreView. Local AI Ambassador.
๊ฐ€์ž… June 2009
1.5K ํŒ”๋กœ์ž‰ ์ค‘    45.3K ํŒฌ
The more I dig in the various inference engines out there, the more I see the shortcuts they take for speed. Don't assume that model layers are mapped 1-1 into your engine, some are quantizing before running, 8bit and even 4bit. I think I should post more on this, not to finger point anyone, but just to clarify that: more often than not, you bargain precision for speed. Kernel optimizations are great, but there is a physical limit (compute/memory bandwidth). There is no free lunch ๐Ÿคท๐Ÿปโ€โ™‚๏ธ
๋” ๋ณด๊ธฐ