Today we are releasing our speculative decoding implementation in our inference engine uzu.
Initially for Qwen3.6 27B, with support for Qwen3.8 27B and Muse Glimmer coming soon.
On Apple M5-series chips, we outperform MTPLX (MLX + speculative decoding) by almost 2x, and llama.cpp by over 3x at comparable quantization levels, with the strongest gains achieved on mathematical reasoning and coding tasks.
Run the model:
Mirai-M:
Mirai-L:
Explore the benchmarks:
Learn more about our speculative decoding implementation:
Our draft model, quantized checkpoint format, verification algorithm, and GPU kernels are co-designed from the ground up around the latest Apple M5 chips to take maximum advantage of GPU Neural Accelerators.
Unlike popular speculative decoding architectures such as model-native MTP, which produce small draft chains of 3-4 tokens at a time, we use extremely aggressive speculative budgets of 16-32 tokens. This enables us to use Neural Accelerator-backed GEMM kernels, achieving maximum utilization of hardware arithmetic throughput.
顯示更多