It's great to see custom inference engines becoming increasingly popular, but be careful about output quality.
I ran some experiments last week and found how easy it is to get Astra or Opus to optimize an inference engine for a specific model or GPU, with impressive speedups.
But LLMs can also take questionable shortcuts. Many of the speedups I observed came from much more aggressive arithmetic approximations.
These approximations aren't necessarily bad, and output quality can remain excellent. Just make sure you validate it yourself.