Why are AI agents so expensive? Do more tokens actually lead to better performance? Which models are more token-efficient? Can agents predict their own token costs before execution?
These were the questions bugging us, so we wrote a paper to find out.
Excited to share "How Do AI Agents Spend Your Money? Analyzing and Predicting Token Consumption in Agentic Coding Tasks", led by
@Longju_Bai, with co-authors at Stanford, MIT, Google DeepMind, Microsoft AI, and All Hands AI.
A few findings that surprised us:
🔹 Agentic coding tasks consume ~1000× more tokens than chat or reasoning workloads. And input tokens, not output, become the dominant cost driver, because each round re-feeds the entire trajectory back into the model.
🔹 More tokens ≠ better outcomes. Runs on the same task can vary by up to 30× in token use, and accuracy often peaks at intermediate cost. Beyond that, extra spending tends to reflect redundant exploration and does not bring further performance gain.
🔹 Models differ substantially in token efficiency. On the same successfully solved tasks, Kimi-K2 and Claude Sonnet-4.5 use roughly twice as many tokens as GPT-5.2. The gap becomes even larger when all the models fail.
🔹 Human-rated task difficulty weakly predicts actual cost. "Easy" tasks for humans can be surprisingly expensive for agents, and vice versa. The classic "Moravec's Paradox" is also true for coding agents!
🔹 Agents struggle to predict their own costs. Self-prediction correlations top out around 0.39, and every model we tested systematically underestimates what a task will cost. Result-based pricing still has a long way to go when we cannot even figure out the token cost beforehand.
Together, these results suggest that cost prediction is a genuinely challenging task for current agents. We think this opens up real research questions around self-modeling, calibrated cost estimation, and pricing mechanisms that work under residual uncertainty.
Huge thanks to my collaborators:
@Longju_Bai, Zhemin Huang,
@sunjiao123sun_ ,
@xingyaow_ ,
@radamihalcea ,
@erikbryn ,
@alex_pentland
paper:
website: