Hello. Quick intro post about myself: - 8 months ago I got tired of paying for AI credits. I thought I could build an inference server myself for cheaper. - I bought 2 RTX 3060s for $500 and it was a great disappointment. - The models back then were really bad and my hardware wasn't helping. I was able to run Qwen 3.5 27b at some ridiculously small quant (pre GSQ-RCO), at a great speed of 9 t/s and probably less than 100 t/s for prompt prefill. - I didn't give up. 3 months ago, I got exposed to Qwen 3.8 27b, and for the first time, it seemed like there was potential, and then I fell down the rabbit hole. - I decided to double down and buy 3 P100 Datacenter GPUs that were released in 2016 - when they came out, they were going for ~$6K; bought for $70 each. - Since then I've been working on optimizing these pre-tensor core GPUs for the best performance on modern models. - Today I'm at 50 t/s decode and 1500 t/s prompt prefill on Qwen3.8 Flash Next, serving 3 concurrent sessions at 200k context each. - Throughout the process, I’ve learned a ton about LLM inference, Linux, and AI hardware. That being said, this is still the beginning and I'm starting to hit some limitations. My overall mission is to replace or assist my cloud subscriptions with the cheapest hardware possible. If anyone has experience doing something similar, I'd love to learn more. Thanks @matt_feroz at getting me on Twitter.