The more I dig in the various inference engines out there, the more I see the shortcuts they take for speed.
Don't assume that model layers are mapped 1-1 into your engine, some are quantizing before running, 8bit and even 4bit.
I think I should post more on this, not to finger point anyone, but just to clarify that: more often than not, you bargain precision for speed. Kernel optimizations are great, but there is a physical limit (compute/memory bandwidth).
There is no free lunch 🤷🏻♂️