A project I’ve been trying to do the last few days has been really hard: Reduce AI Costs for complex workflows
It’s obviously very do-able and the job for your eng + prod teams these days
But what’s been hard:
- Evals IRL aren’t that simple. In the real world, they don’t predict enough actual quality of output to fully rely on.
- Some models are just too slow for certain workflows in practice, no matter how fast first tokens are rendered. You have to be as fast or faster than Sonnet to work for this use case at least.
- A cheaper model that fails more often can -> massive cost overruns if keeps trying to solve a workflow. Solvable of course but a real issue.
This is the job today. But simply turning on OpenRouter (an elegant a product) … is just the very, very start.