🧵 With unlimited compute, how fast can agents surpass humans? We introduce Elo-per-token analysis to profile agent performance curves across multiple open-ended tasks.
• Agents initially scale faster than repeated sampling, but over long horizons converge toward their theoretical log-linear scaling curve. Humans, in contrast, improve superlinearly.
• These curves also tell us how to spend test-time compute: the scaling inflection point gives a simple rule for splitting a fixed budget across agent sessions. Split a long run in a principled way, and you can get significant gains over a single run.
• Fitting human-time and agent-token curves also gives us a fun way to translate AI compute into human time. Taking OpenAI’s ~130B-token Navier–Stokes run as input and extrapolating across the two curves gives an equivalent of ~41 years of work by a mathematician at 8 hours/day 😮.