登録して招待リンクを共有すると、動画再生報酬と紹介報酬を獲得できます。

Mirai Labs
@trymirai
Frontier on-device AI lab. Models, runtime & infrastructure to make on-device AI interactive, ambient & continuous.
参加 January 2025
31 フォロー中    906 ファン
Today we are releasing our speculative decoding implementation in our inference engine uzu. Initially for Qwen3.6 27B, with support for Qwen3.8 27B and Muse Glimmer coming soon. On Apple M5-series chips, we outperform MTPLX (MLX + speculative decoding) by almost 2x, and llama.cpp by over 3x at comparable quantization levels, with the strongest gains achieved on mathematical reasoning and coding tasks. Run the model: Mirai-M: Mirai-L: Explore the benchmarks: Learn more about our speculative decoding implementation: Our draft model, quantized checkpoint format, verification algorithm, and GPU kernels are co-designed from the ground up around the latest Apple M5 chips to take maximum advantage of GPU Neural Accelerators. Unlike popular speculative decoding architectures such as model-native MTP, which produce small draft chains of 3-4 tokens at a time, we use extremely aggressive speculative budgets of 16-32 tokens. This enables us to use Neural Accelerator-backed GEMM kernels, achieving maximum utilization of hardware arithmetic throughput.
もっと見る