TL;DR Routing between models at the token level looks great on paper, yet running it on today's serving stacks makes things slower. This paper fixes that with a purpose-built runtime. Accepted at NeurIPS 2026. Title: TokenRouter: Efficient Serving System for Token-Level LLM Routing URL: Points 🎯 R2R matches 32B quality while sending just 5% of tokens to the 32B model, versus roughly 40% of queries under query-level routing 🧱 But vLLM and SGLang assume every request is step-synchronous, so fast models idle waiting on slow ones and batches fragment on every switch ✍️ Developers write only route, send, and receive, tracing how a single request flows, while the runtime handles async execution 🔀 Each model gets its own subserver, with client, decoding, and inter-model loops decoupled and KV-cache preserved across handoffs ⏳ Instead of firing on arrival, the scheduler buffers to a threshold; wait too long and the small model starves, so the optimum is derived from a discrete-time Markov chain 📈 Wins on all 15 algorithm-by-workload combinations, with 2.01-64.15x higher throughput 🧪 Throughput holds steady when output grows from 2,048 to 8,192 tokens, where the standard baseline loses 58-85% of its own The range is wide though, and on Co-LLM and R-Stitch the 32B model alone still runs faster. Worth remembering this isn't a universal win. #LLM# #Inference#