Register and share your invite link to earn from video plays and referrals.

Ivan Fioravanti
@ivanfioravanti
LLM addicted, Apple MLX, NVIDIA DGX, Cloud, Kubernetes, Technology Advisor, Investor and Co-Founder / Board Member of CoreView. Local AI Ambassador.
Joined June 2009
1.5K Following    45.3K Followers
The more I dig in the various inference engines out there, the more I see the shortcuts they take for speed. Don't assume that model layers are mapped 1-1 into your engine, some are quantizing before running, 8bit and even 4bit. I think I should post more on this, not to finger point anyone, but just to clarify that: more often than not, you bargain precision for speed. Kernel optimizations are great, but there is a physical limit (compute/memory bandwidth). There is no free lunch 🤷đŸģâ€â™‚ī¸
Show more