Reconstruct a moving, deforming object in 4D (3D + time) from a single-camera video — even under heavy occlusion and large non-rigid motion 🎥
Title: Lift4D: Harmonizing Single-View 3D Estimation for 4D Reconstruction In-the-Wild
URL:
🎥 Overview
A test-time optimization framework combining single-view 3D prediction with deformable 3D Gaussian Splatting. It fuses learned geometry/appearance priors with the video to handle severe occlusion and non-rigid motion.
❓ Problem it solves
・Directly predicting 4D is limited by scarce training data and generalizes poorly
・Initializing 3D and refining with video alone breaks down on in-the-wild scenes with large deformation and occlusion
💡 Methodology
Three components:
・Causal Latent Conditioning: aligns an image-to-3D model temporally without retraining, coupling adjacent latents at the ODE level, with t₀ trading off consistency vs per-frame fidelity
・Deformable Gaussians: sparse control nodes with time-varying SE(3) transforms + linear blend skinning deform a canonical representation across frames
・Occlusion-aware refinement: compares only visible pixels, while a view-conditioned diffusion prior completes unobserved surfaces
📊 Results
・Compared against STAG4D, PAD3R, L4GM, DreamMesh4D, V2M4, BANMo
・On Pexels data, point-tracking error EPE 0.072 (vs 0.119–0.211 for competitors)
・~30 min per 32-frame video on a single H200 GPU
#
3DReconstruction# #
GaussianSplatting#