注册并分享邀请链接,可获得视频播放与邀请奖励。

Ash Hart
@ashxhart
Doing my bit to bring local AI to everyone  | building | | Thoughts are my own.
加入 October 2014
487 正在关注    14K 粉丝
A lot of speed-up tricks for AI inference engines sacrifice the quality or accuracy of the answers TensorFold uses exact speculative decoding to get you the main model's answer faster. A drafter proposes several tokens ahead (batched). The main model checks them together, keeps the matching prefix and corrects the first mismatch. With the same TensorFold setup, prompt and seed on a supported decoding path, the output matches drafting off, token for token. The main model still decides the answer, you just spend less time waiting for it. @tensorfold_dev
显示更多
0
10
56
3
转发到社区