Hide an object behind a wall in a video generation model, and it might come back as something else entirely. This paper tackles that object permanence failure head-on.
Title: Training Object Permanence in World Models
URL:
📝 Overview
The paper trains video models on object permanence and solidity — cognitive abilities human infants develop by six months old. It builds WROP, a benchmark of 150 synthetic task generators producing 1.5M training samples, and PWM-WROP, a 16B-parameter model fine-tuned on it.
❓ Problem Solved
Models like Sora let objects vanish behind occluders and reappear as different objects, or pass straight through solid barriers. These failures undermine higher-level reasoning about collisions and cause and effect.
💡 Method & Approach
150 Blender-generated tasks are organized into six occlusion and solidity families. Structural parameters like object count and trajectory vary systematically while surface parameters like color and lighting are randomized, preventing models from succeeding through memorization. PWM-WROP fine-tunes NVIDIA's Cosmos3-Nano on this data.
📊 Results
In 361 blind human pairwise comparisons, PWM-WROP ranked first among true-continuation models (Elo 1679.5), 224 points ahead of the runner-up. It topped every model on static occlusion tasks, but still struggled on solidity tasks like collisions.
🌍 Use Cases
The team released the training data, model weights, and PWM, a native-PyTorch training stack for AWS Trainium2 — laying groundwork for physically grounded world models.
#
WorldModels# #
VideoGeneration#