注册并分享邀请链接,可获得视频播放与邀请奖励。

TheDavidTai
@TheDavidTai
Founder, hacker, stochastic tool. @microsoft escapee. Currently @eigenlabs. Checkout my github actions emulator:
加入 September 2018
722 正在关注    1.6K 粉丝
130-140tps that I and other contributors were able to achieve with MLX-Server, Tensorfold and the rest of the ecosystem is where the CPU and Memory both start to choke. We've pretty much squeezed all the juice for Qwen Flash Q4 experts quants. Next model please, @Alibaba_Qwen!
显示更多
Drafter speedups are not a 256 token demo trick. M5 Max, TensorFold, Qwen3.8 27B: 129 tok/s across 162K tokens of long output, against 24 tok/s with drafting off. 30 runs, 2.7K to 8.2K tokens each, three sampling settings. 5x holds when the answers get long.
显示更多