🚨I just beat upstream llama.cpp AND Strata with my custom llama.cpp fork!!! When I started, qwen 3.8 27b decode began around 2~ token a second plain without mtp and dflash2. The drafters barely boosted numbers initially, it was disheartening to work on such low hardware and no money to spend on GPUs or server hardware. Feels incredible to be one week into October finally with results to share and finally able to run with some remarkable numbers from where I began! Since July I've been focused on Vulkan and AMD optimizations! I've been building since July! Back in June and July I predicted two things: 1. Open source models will get better and smaller catching up to frontier models. 2. Local inference engines will improve and new ones will come out. Both came true and are coming truer right now! I kept it private but posted on Arifi Labs on LinkedIn, x and instagram in September about it, dropping a hint until I had something solid to share like now! With so much recent noise and news from other developers I plan to finally make it public! Freetoken, Strata, beellama, bunllama have been making news but mostly for nvidia and cuda and or apple metal/mlx. None for vulkan. CUDA and Metal gaining 50%+ sometimes 100% speeds on existing hardware! What a time to be alive! My focus has been Vulkan. I worked on NO budget, no money, just my old and weak @Beelinkofficial SER 7 Pro. I initially started with only 32GB RAM then 48GB (two of the 32GB sticks I bought, one failed immediately making me mix my old and new, 64GB was a no). AND NO GPU just an iGPU A 780M! Then I scored a low budget upgrade, still no gpu: a barebone @Hi_MINISFORUM X1 470 Pro ( 1000cad) and 96GB of @CrucialMemory DDR5 5600mhz CL46 RAM (1000cad on FB marketplace). Cheap compared to most @AMD strix and nvidia DGX boxes! I couldn't keep working on my old machine. I needed an upgrade. The 780m to 890m iGPU jump wasn't significant, the newer Zen with more cores and npu nice, but no gpu! Nice additional nvme slots, better cooling and more supported ram was worth it long term with oculink for potential gpu later. I can load larger models now and I will be able to run a gpu. Still the real gains came from my changes to llama.cpp. With no GPU I hit: -20 t/s on @Alibaba_Qwen 3.8 Flash Next 125B (GSQ-RCO IQ3_S, MTP) -35 t/s on @ornith_ 1.5 35B (Q4_K_M, MTP) -12.1 t/s on @QwenDevs 3.8 27B (Q4_K_XL, DFlash2) -2.96 t/s on @Zai_org GLM 5.3 Flash 320B (GSQ-RCO 3.0-bit, DFlash2, streamed from NVMe) Qwen 3.8 27B started at 2 t/s plain and 8 t/s with dflash2, ornith 1.5 35b around 20 t/s, glm 5.3flash around 0.2 t/s and qwen 3.8 flash next around 10 t/s. Now I'm here. I'm confident more gains will come! 🎉🙌🥂Together we can bring AI to all and I'm proud of my first upcoming contribution to Open Source local AI. Releasing it soon