註冊並分享邀請連結,可獲得影片播放與邀請獎勵。

Mirai Labs
@trymirai
Frontier on-device AI lab. Models, runtime & infrastructure to make on-device AI interactive, ambient & continuous.
加入 January 2025
31 正在關注    906 粉絲
Today we are releasing our speculative decoding implementation in our inference engine uzu. Initially for Qwen3.6 27B, with support for Qwen3.8 27B and Muse Glimmer coming soon. On Apple M5-series chips, we outperform MTPLX (MLX + speculative decoding) by almost 2x, and llama.cpp by over 3x at comparable quantization levels, with the strongest gains achieved on mathematical reasoning and coding tasks. Run the model: Mirai-M: Mirai-L: Explore the benchmarks: Learn more about our speculative decoding implementation: Our draft model, quantized checkpoint format, verification algorithm, and GPU kernels are co-designed from the ground up around the latest Apple M5 chips to take maximum advantage of GPU Neural Accelerators. Unlike popular speculative decoding architectures such as model-native MTP, which produce small draft chains of 3-4 tokens at a time, we use extremely aggressive speculative budgets of 16-32 tokens. This enables us to use Neural Accelerator-backed GEMM kernels, achieving maximum utilization of hardware arithmetic throughput.
顯示更多
0
15
183
41
轉發到社區