I ran the same Qwen3.6-35B-A3B at 256K context on an ASUS GX10 and a 64GB M2 Max. The main gap was prefill. The Mac needed 23–28 minutes before the first output token. Full results: 1. Prefill At 128K: GX10 TTFT 45.7s vs Mac 552.8s. Decode: 40.6 vs 28.7 tok/s. At 256K: GX10 127.5s vs Mac 1,541s median TTFT. Decode differs 1.4x. Prefill differs 12x. 2. MLX 4-bit beat 3-bit, 6-bit AND 8-bit 4-bit: 76 tok/s, 3/3 coding, 12/12 retrieval. 3-bit: failed executable coding in every single seed. 8-bit: half the speed and worse retrieval (mangled citations, 9/12). 3. MTP is not a free speed switch Same model, same Q4_K_M, same coding task, 3 seeds: GX10 CUDA: +26.9% decode. M2 Max Metal: −19.4%. Correctness held for both (12/12 executable) but speed didn't. Code: Raw dataset:

The AI memory wall is costing you throughput. Diamond Partner @WEKA fixes it at the infrastructure layer: 1000x KV-cache expansion, 20x faster #TTFT#. See it at SuperAI Singapore, June 10–11.

New free learning path on the Red Hat Developer Sandbox: compress, serve, and benchmark a model with @vllm_project, hands-on in Jupyter, no GPU needed. Here's what you'll actually do: ⚡ Quantize Qwen3 to W4A16 with LLM Compressor using GPTQ. Measure the result: 42% smaller, 8.2% perplexity increase. Learn to decide if that tradeoff fits your use case. 🚀 Connect to a running vLLM server and send requests via the OpenAI-compatible API. Watch 5 concurrent requests handled in real time. See prefix cache queries increment live via the Prometheus metrics endpoint. 📊 Run a GuideLLM benchmark: TTFT, inter-token latency, and E2E latency at p50, p95, and p99. Run Hellaswag with lm_eval. Cross-reference with the published model card to make a deployment decision backed by numbers. Less than an hour to complete. Free account. Built by @cedricclyburn and Michael Santos. 🙏

Most agentic stacks run into the same problems pretty quickly: reasoning and tool parsing drift across turns, KV cache reuse falls apart, or tools fire too late. We’ve been hardening Dynamo’s harness-facing path so @Claudeai Code, @OpenClaw, and @openai Codex-style agent patterns behave reliably on custom stacks and inference endpoints: • Stable prompts for KV reuse and lower TTFT • Interleaved reasoning + tool calls preserved across turns • Streaming tool dispatch instead of end-of-turn buffering • Harness behavior aligned with real multi-turn agent runtimes If you’re building your own agent stack or serving endpoint, this blog goes through the infrastructure issues that tend to show up in practice and the patterns we’ve been using to fix them. Tech blog ➡️

We push Prefill/Decode disaggregation beyond a single cluster: cross-datacenter + heterogeneous hardware, unlocking the potential for significantly lower cost per token. This was previously blocked by KV cache transfer overhead. The key enabler is our hybrid model (Kimi Linear), which reduces KV cache size and makes cross-DC PD practical. Validated on a 20x scaled-up Kimi Linear model: ✅ 1.54× throughput ✅ 64% ↓ P90 TTFT → Directly translating into lower token cost. More in Prefill-as-a-Service: