Across various tasks and models, Val shows that an oracle router would beat the best single model by 14 points while costing about 40% less.
But understanding how to evaluate router performance is much harder in production. We shared our work at
@notdiamond_ai on using simulated users to evaluate coding agent routing, accounting for long sessions, user interactions, and cache dynamics that impact real-world costs and performance. Check it out: