🦾 Can a single frozen general-purpose VLM drive a robot with no learned action model and no segmentation tools at all?
Title: MotorMind: Scaffolding General Vision Language Models for Zero-Shot Robot Manipulation
URL:
A harness that gets a general VLM behaving like a human teleoperator, without learned action experts, coding agents, or grounding tools like SAM3.
Three things stand out.
🎮 A mid-level action interface
Instead of emitting joint commands, the VLM proposes parameterized moves, rotations, and gripper operations. It picks the actions and their magnitudes while a deterministic controller converts them into embodiment-specific motion and records what the robot actually did, so an intended movement never gets mistaken for an achieved one.
🧠 Five roles on one frozen VLM
Planner, Executor, Monitor, Verifier, and Memory all share the same model but get different context and different output authority. The Monitor runs in the background and can fire a STOP alert that discards queued commands, say when the wrong object got grasped. Notably it never picks the corrective motion itself; that call goes to outcome assessment.
📈 It improves as the backbone does
On LIBERO-PRO, where zero-shot VLA policies all score 0.0%, it reaches 66.7%, and its 53.8% under perturbations edges out the fine-tuned OpenVLA-OFT at 51.2%. On a real xArm6 it averages 95%. Swapping the backbone for GPT-6 Sol lifts it to 83.3% with no redesign.
The reframing is the interesting part: zero-shot control as aligning general model capability with the right control abstraction, rather than learning a specialized policy.
#
Robotics# #
VLM#