TheDavidTai
@TheDavidTai
Founder, hacker, stochastic tool. @microsoft escapee. Currently @eigenlabs.
Checkout my github actions emulator:
加入 September 2018
722 正在關注 1.6K 粉絲
130-140tps that I and other contributors were able to achieve with MLX-Server, Tensorfold and the rest of the ecosystem is where the CPU and Memory both start to choke.
We've pretty much squeezed all the juice for Qwen Flash Q4 experts quants. Next model please, @Alibaba_Qwen!
顯示更多
Drafter speedups are not a 256 token demo trick.
M5 Max, TensorFold, Qwen3.8 27B: 129 tok/s across 162K tokens of long output, against 24 tok/s with drafting off. 30 runs, 2.7K to 8.2K tokens each, three sampling settings.
5x holds when the answers get long.
顯示更多
