Game world models tend to generate pixels directly, but pose and geometry drift the longer you roll them out. Marionette simply gives up the parts that must be exact โ to a renderer.
Title: Marionette: Predicting World States, Rendering Geometry, Painting Appearance
URL:
A world model that delegates geometry, occlusion, and motion to a deterministic renderer and lets the neural net paint only appearance. Three highlights.
๐ฎ Highlight 1: Predict an explicit 276-D world state
Instead of pixels, it predicts a 3D state of joints, root trajectory, and rotations via two-stage autoregression. Control is just overwriting an action token (like a player's button press). Forcing a wrong action shifts the pose by ~31% โ proof the control truly bites.
๐ Highlight 2: A zero-parameter deterministic renderer
Six closed-form operations turn state into pose-control video, so world consistency, occlusion order, and metric scale hold by construction. Long-horizon failures are fixed by rules (e.g. a terrain collider) without retraining appearance โ cutting penetration by 66%.
๐จ Highlight 3: Video diffusion for appearance only
A Wan2.2-Fun-5B diffusion paints photorealistic RGB on top of exact geometry. Even via predicted state, FVD is 831 (799 from recorded state, 975 for pixel-autoregressive) โ no detectable fidelity loss.
Exactness to deterministic compute, appearance to the neural net โ a clean division of labor.
#
WorldModels# #
GameAI#