If the same general agent can understand a car, an arm, a humanoid, etc. through their sensor/action interfaces, you start to decouple intelligence from the hardware platform itself. “LLM as puppeteer” is probably just the beginning. The bigger shift might be robots converging on an abstraction layer where a general agent can express intent without needing to understand the low-level control details of every embodiment. Almost like a hardware abstraction layer for physical intelligence.

I think “can an LLM control a robot?” is slightly the wrong question. With enough compute and enough time between actions, probably yes. What’s more interesting is what has to change in the architecture, training, and inference for these models to operate at robot speed. Robotics operates at multiple timescales. Deciding “Pick up the pan and place it on the stove.” can happen relatively slowly. Maintaining balance, correcting a slipping grasp, reacting to contact, or adjusting a joint trajectory cannot. A model that has to autoregressively generate a sequence of tokens before every physical correction is paying an enormous latency tax for capabilities you often don’t need at that layer. Making the LLM 2x or 5x faster helps, but it doesn’t really change the shape of the problem.