Introducing EgoVerse: an ecosystem for robot learning from egocentric human data. Built and tested by 4 research labs + 3 industry partners, EgoVerse enables both science and scaling 1300+ hrs, 240 scenes, 2000+ tasks, and growing Dataset design, findings, and ecosystem 🧵

We started this moonshot project a year ago. Now we are excited to share our progress on robot learning from egocentric human data 🕶️🤲 Key idea: Egocentric human data is robot data in disguise. By bridging the kinematic, visual, and distributional gap, we can directly leverage human data to scale up imitation learning. My thoughts: 1. Human data is robot data: We perceive with our eyes and act with our body and hands. If we can learn across robot embodiment, why not humans? 2. Data collection needs to be passive: Just as the Internet evolved into an unintentional data repository for AI, we envision systems that effortlessly capture embodied experiences from human activities, without humans’ conscious participation. 3. The data capturing technology is ready: Project Aria glasses by @RealityLabs @meta_aria capture all the information we need to turn human data into robot data: fisheye camera, 3D hand tracks, SLAM, … We will see more ubiquitous devices entering the consumer market. 4. The next generation foundation model will learn from embodied human data: The human sensorimotor experience is largely missing from today’s AI training data. Next-gen foundation models will understand humans from a human’s perspective, driven by large-scale embodied human data. See @simar_kareer's thread for more details about this project.