2/n Model routing is context-dependent
Model routing doesn't work at the LLM API level. The same request can have different difficulty level (hence different optimal model) depending on the context around it.
Take the task "What is our biggest competitive threat nowadays" as an example. For some organizations, doing a simple search "competitive threat" will bring up an up-to-date canonical report from a competitive analysis team. For others, the agent actually needs to comb through Gong calls, external Slack channels, emails, Salesforce to get to the answer.
Show more
Learnings from looking at 10,000+ agent traces in the enterprise
1/n The "jagged frontier" is here to stay
It's becoming clear that LLMs can be post-trained to solve whatever tasks we throw at them, as long as the task can be packaged into a verifiable RL environment. However, each model will be trained on a different data mix, and there is limited "cross-task transfer", leading to each "frontier" model being the best at different task distributions. We see this today with Claude/Kimi being better on visual/front-end, GPT being better on backend, and we'll likely keep seeing it because each lab will train their model on a different data mix
Show more
