TL;DR: By generating perspective and bird's-eye-view (BEV) video together, a driving world model can make surrounding vehicles react to the ego vehicle's maneuvers.
BirdDreams: Joint Perspective and BEV Video Generation for a Plausible, Reactive Driving World Model
Key points
ใป๐ Prior models took surrounding-vehicle trajectories as external input, fixing them to logged motion
ใป๐บ๏ธ It generates perspective and BEV video jointly, so scene layout becomes part of the prediction rather than an input
ใป๐ง Two video diffusion transformers use joint attention at every block (about 4B parameters)
ใป๐ฒ Independent per-stream noise lets each branch learn to use the other's information
ใป๐ With ego-action conditioning, ADE drops about 35-36% vs OmniDreams (Plรผcker)
ใป๐ฅ In a human study (25 participants), BirdDreams was preferred 57% vs 24% for OmniDreams
Caveats: ego trajectories are reconstructed from generated video, and the human study compares only variants given the same road geometry.
#
WorldModel# #
AutonomousDriving#