註冊並分享邀請連結,可獲得影片播放與邀請獎勵。

Ash Hart
@ashxhart
Doing my bit to bring local AI to everyone  | building | | Thoughts are my own.
加入 October 2014
487 正在關注    13.8K 粉絲
A lot of speed-up tricks for AI inference engines sacrifice the quality or accuracy of the answers TensorFold uses exact speculative decoding to get you the main model's answer faster. A drafter proposes several tokens ahead (batched). The main model checks them together, keeps the matching prefix and corrects the first mismatch. With the same TensorFold setup, prompt and seed on a supported decoding path, the output matches drafting off, token for token. The main model still decides the answer, you just spend less time waiting for it. @tensorfold_dev
顯示更多
0
10
56
3
轉發到社區