Finding a specific landmark in a city-scale 3D scene with tens of millions of points usually means ranking every candidate. Urbex instead lets an agent explore like a person would.
Title: Urbex: Agentic Spatial Grounding for City-Scale 3D Scenes
URL:
🏙️ The problem
City-scale 3D scenes can span hundreds of thousands to tens of millions of points per scene, and existing ranking-based methods rely on precomputed instance proposals that don't scale. This work reframes urban localization as something inherently sequential and exploratory.
🔧 The approach
A VLM agent uses three tools, landmark query, zoom-in, and oblique rendering, to explore a bird's-eye-view multi-view environment before committing a final bounding box. Trained with GRPO, the reward design combines IoU, center-distance shaping, and step penalties to optimize both accuracy and search efficiency.
📊 Results
On the CityRefer benchmark, Hit
@1 reaches up to 23.9% versus 14.2% for the baseline, while inference takes just 3.2 seconds versus 32.1 seconds, roughly a 10x speedup. Zero-shot transfer to an unseen urban scene still hits 27.2% Hit
@1.
🔍 Remaining challenges
Missing a named landmark drops Hit
@1 by 10.8 points, while coordinate noise barely matters; and when many same-class objects cluster nearby, local success falls from 64.7% to 25.1%.
#
SpatialAI# #
VLMAgents#