Step into the map with the Street View grounding feature in Project Genie from @GoogleDeepmind and @GoogleLabs. Announced at I/O, this research prototype uses locations from @GoogleMaps Street View as a foundation, letting you generate and explore interactive, 360-degree virtual environments from just a text prompt or real-world starting place. Cool, right? But… How does it actually work? 🤔 As an experimental tool, Project Genie tackles the "blank space" problem (showing both what’s in front of the camera and behind it) by utilizing Street View data to realistically generate a 360-degree view of the location you selected as the starting point for your world to generate from. Worlds generated by Genie are far more dynamic and rich because they’re created frame-by-frame based on the world description and user actions. By predicting each subsequent frame, Genie is able to simulate what it looks like to swim across an ocean, or hike to the top of a peak, marking a massive shift in interactive media and simulation pipelines. What real-world place would you want to step into and explore?

For over a decade, Google Street View unknowingly captured the everyday lives of an elderly couple living in front of their small home in Indonesia. Year after year, the couple was seen sitting together outside their front door. Then, they disappeared one by one.

.@bilawalsidhu reveals how the Seoul World Model generates realistic training data for robotics: "Seoul World Model is a really interesting paper, very similar to what Google's doing with Street View grounding. If you ask Genie to give you a representation of the Palace of Fine Arts, it'll give you a plausible reconstruction, but it ain't exactly it." "One way to constrain these models is, we could just do what we do in LLMs, which is RAG, retrieval-augmented generation." "Can we do a form of spatial RAG where you just pull in the nearest reference image of reality to condition the generation? That lets you do a lot of these use cases that are interesting for robotics." "If you wanna create a bunch of training data for downtown Seoul in different lighting conditions, really cool way to go about doing that. That's exactly what you see folks like NVIDIA trying to do with Omniverse as this 3D simulator and Cosmos."

.@bilawalsidhu explains the crucial difference between plausible and factual world models: "Y'all had the @maticrobots CEO on. I frigging love that thing. It's basically a little Street View car using simultaneous localization and mapping that's building and refining this model of your space, and taking action based on what you ask it to do." "It needs to be super grounded in reality because you don't want this thing chewing your wires or walking into a wall that doesn't actually exist. In situations like that, or self-driving vehicles, oh yeah, the street is actually here. This is exactly where the lane markers are, and based on this I predict this is where this person's going. The stakes are very, very high." "When you're talking about making something like a rough simulation for just pixels, let's say for a YouTube video or a piece of entertainment content, who cares if it's absolutely perfect? So that's how I think about that divide."

Have you explored Project Genie yet? 🌎 We just launched a huge set of updates! You can now simulate worlds grounded in @GoogleMaps Street View, manage your creations in a new library, and share them externally. Let's dive in: 1️⃣ Street View Grounding: You can now generate worlds starting from real places! Just tap the Maps pin to choose a location in the US and you can pick a style like "Desert Sands" or "Ocean World." Describe your character, and Genie ties your starting location to real-world imagery. 2️⃣ Library: You can now store and organize your generated worlds. Easily explore past creations or remix your favorites to iterate on your ideas. 3️⃣ External sharing: You can now share your generated worlds. Anyone can watch a video preview of your world, and Ultra subscribers can explore or remix them. Check out a shared world here: ✨ And now, Project Genie is gradually rolling out to all eligible Google AI Ultra subscribers globally (18+). Jump in and start creating today!