Register and share your invite link to earn from video plays and referrals.

Sayash Kapoor
@sayashk
Incoming prof @UCBerkeley AI agents, policy, evals, AI for science Essay/newsletter (AI as Normal Technology): Book (AI Snake Oil):
Joined March 2015
2.5K Following    15.6K Followers
I have been using the new ultrafast mode for Astra 6 for the last couple of days. My main takeaways: 1. Token speed qualitatively changed how I interact with agents. The output speed is 300 tokens/second, 8x the regular speed. Rather than breaking flow and switching tasks, the model moves (just) fast enough that I can keep working on the same task, especially for work that requires frequent inputs from me. I am excited about ultrafast on Sol-class models, which are twice as fast as Astra. 2. Despite being 8x faster at generating tokens, the overall speedup is only 2-4x, since the time consumed for tool calling or running code remains the same. External factors (and not token generation speed) might soon become the main bottleneck for agentic work. (Notably @random_walker predicted both of these changes in: 3. One area where speed really helps is computer use. The previous generation of models wasn't reliable enough at computer use to be useful. Astra was reliable enough to be useful, but still slow. Astra with ultrafast resolves both concerns. Unlike tool calls for coding tasks, UIs react in milliseconds, so token speed is the main bottleneck. 4. At 300 tokens/sec, I am already struggling to keep up with the agent. But there is a lot more room for improvement. Cerebras serves GPT-OSS-120B at 3,000 tokens/sec and Sol at 750 tokens/sec. So in a year's time, it is plausible we'll see another 10x increase in speed in models with Astra-level capabilities. At that point, human oversight will become the bottleneck for what people can use computer-use agents for. 5. I can imagine blazing fast computer-use agents becoming extremely common for knowledge work. But I expect their use will be jagged (in the same way as LLM and agent adoption has been): given the oversight challenge, adoption will begin in verifiable tasks where the cost of errors is low (e.g., filling out a form using an agent before reviewing it yourself), and slowly diffuse to hard-to-verify tasks. 6. Given the challenges with human oversight, the key to this diffusion will be developing automated oversight tools. OpenAI's approval/verifier agents have already been useful to me on this front: The verifier understood my intent better than the computer-use agent, and stopped it from filling out a form incorrectly. Still, I expect there is a lot of headroom in improving verifiers for computer use, to make it as reliable in practice as auto mode in Claude Code/Codex is for coding tasks. 7. Ultrafast mode is undeniably resource intensive: I ran through my weekly limit in ~2 hours while filling out a bunch of forms and doing simple coding tasks. But given the pace of efficiency improvements (and the fact that Sol is 5x cheaper than Astra and already decent at computer use), I expect it to rapidly fall. Once fast, cheap models are common, users may choose "sub-frontier" models like Sol and make real tradeoffs between speed and quality.
Show more
Prediction: we will soon enter an era where the latency of agentic work is bottlenecked not by LLM inference speed but by tool use. Shell commands, build pipelines, computer use, web data extraction, and everything else that happens in the background is way slower than it needs to be, because it hasn't been subjected to the same kind of optimization pressure as LLMs, and isn't really getting faster.
Show more