注册并分享邀请链接,可获得视频播放与邀请奖励。

Mathias Lechner
@mlech26l
Cofounder/CTO at Liquid AI and Research Affiliate MIT
加入 December 2017
226 正在关注    2.3K 粉丝
Making on-device AI even fast + all open-weight
Today, we release DSpark draft models for LFM2.5-1.2B-Instruct, LFM2.5-2.6B, and LFM2.5-8B-A1B. These add a speculative decoding path that trades a minimal memory increase for a large decoding speedup without changing output quality. A lightweight draft model proposes a block of candidate tokens and the target model verifies them in a single forward pass. Across MATH500, GSM8K, HumanEval, MBPP, and MT-Bench at batch size 1: > Up to 3.18x throughput on an H100: LFM2.5-8B-A1B on MATH500, 428 → 1362 tok/s > Up to 2.87x on an M4 Max MacBook Pro: LFM2.5-1.2B-Instruct on HumanEval, 136 → 389 tok/s > LFM2.5-2.6B means: 2.67x on the H100 (323 → 864 tok/s), 2.27x on device (61 → 139 tok/s) > Under greedy decoding, the emitted sequence is identical to baseline by construction, so benchmark accuracy is unchanged. Each draft model is around 300M parameters, with embedding and LM head tied to its target model. The gain shows most in agentic workloads, where the model reasons before every tool call and the user waits through it all: on BFCL multi-tool scenarios, DSpark cuts LFM2.5-2.6B latency by nearly 50% on average. 🧵
显示更多
0
2
40
10
转发到社区