TheDavidTai
@TheDavidTai
Founder, hacker, stochastic tool. @microsoft escapee. Currently @eigenlabs.
Checkout my github actions emulator:
Joined September 2018
718 Following 1.6K Followers
130-140tps that I and other contributors were able to achieve with MLX-Server, Tensorfold and the rest of the ecosystem is where the CPU and Memory both start to choke.
We've pretty much squeezed all the juice for Qwen Flash Q4 experts quants. Next model please, @Alibaba_Qwen!
Show more
Drafter speedups are not a 256 token demo trick.
M5 Max, TensorFold, Qwen3.8 27B: 129 tok/s across 162K tokens of long output, against 24 tok/s with drafting off. 30 runs, 2.7K to 8.2K tokens each, three sampling settings.
5x holds when the answers get long.
Show more
