NEW SKILL: This team take humanoid parkour to the next level!
This project involves
@zhenkirito123,
@Yuanhang__Zhang,
@pabbeel,
@carlo_sferrazza,
@GuanyaShi, and others.
Called Perceptive Humanoid Parkour (PHP), it lets a Unitree G1 humanoid (29 DoF, 1.3 m) run long-horizon, vision-based parkour and decide on its own whether to step over, climb onto, vault, or roll off obstacles, using only onboard depth sensing (a 30 Hz camera) plus a discrete 2D velocity command from the operator.
It builds the motions by motion matching, a nearest-neighbor search in a 27-dimensional feature space (imported from the video-game industry) that stitches retargeted atomic human clips into long-horizon kinematic trajectories through a shared Locomotion to Skill to Locomotion manifold.
It then trains per-motion RL tracking experts (with a privileged height scan) and distills them into one depth-based multi-skill student via DAgger combined with RL.
It replaces hand-scripted skill sequencing and operator-triggered skill selection.
The onboard depth-only student holds 0.95 to 1.00 success across obstacle heights in simulation, while a naive end-to-end depth policy collapses from 0.95 to 0.07/0.08 at 58 and 76 cm.
The upside comes from distilling privileged-height-scan experts and from motion-matching composition, not from training onboard from scratch.
The operator sends only a coarse 2D velocity (W/A/D keys), and the robot itself picks which skill to run from the depth image.
The training data contains only single-obstacle traversals, so the ability to chain skills across a multi-obstacle course is generalization the policy was never explicitly taught.
Transitions are constructed in kinematics, not rediscovered by learning.
Every skill enters and exits through a shared locomotion manifold, so no hand-captured skill-to-skill transition clip is needed.
Motion matching densifies a sparse library into smooth long-horizon trajectories before any RL.
The entire parkour library is only about 66 seconds of mocap, and each skill is just a few seconds.
Ablating approach-distance variety ("Extreme Distances") drops the 76 and 94 cm climbs to 0.62 and 0.64, and "Half Density" drops the 76 cm climb to 0.32.
Motion matching's job is to manufacture stride-phase and approach variety out of seconds of data.
It clears a 1.25 m wall (96 percent of its 1.3 m height) in 3.63 s, and a cat-vault peaks at 3.41 m/s clearing a 0.4 m by 0.5 m obstacle in 0.8 s.
A single 30 Hz depth stream drives both traversal and runtime re-planning, shown by displacing obstacles about 0.5 m mid-run.