Register and share your invite link to earn from video plays and referrals.

Léo
@LeoKharon
Robotics research & updates. Co-host @roboticsstack, the weekly pod on what's actually shipping in 🤖
Joined November 2022
469 Following    1.9K Followers
NEW RESEARCH: Robot hands can now play the piano! It involves @amberxie_, @HaozhiQ, and @DorsaSadigh. Called HandelBot, it is a bimanual system that plays real songs on a real piano using two Tesollo DG-5F dexterous hands, one on a Franka Panda arm and one on an FR3 arm, using only three fingers per hand (index, middle, ring; 9 residual action dimensions per hand). A policy trained entirely in simulation with RL is adapted to the real piano in two stages: 1. a structured refinement step that repeatedly executes the trajectory, compares the intended key against the key actually pressed, and nudges each finger's lateral joint to correct it. 2. then residual RL that learns fine corrective actions on top. The arm wrist poses are scripted from sheet music. The only real-world feedback, and the reward, is the keyboard's own MIDI output. There are no cameras and no tactile sensing. It replaces collecting large real demonstration sets for millimeter-precise contact tasks. The DG-5F fingertip (about 2.2 cm) is wider than a piano key (about 2.16 cm), so a single fingertip physically straddles two keys. Which is why sub-millimeter lateral alignment is the whole game and why the thumb and pinky are dropped, only three fingers per hand fit, and songs must be re-fingered and the hands separated by octaves to avoid arm collision. The real-world adaptation uses only the keyboard's MIDI output: as the per-finger error signal for the lateral refinement and as the sole RL reward (key-press reward, no fingering or energy terms). No camera, no touch. The task's own output is a dense, objective reward for free, the cleanest instance of the instrument being the reward function. For this task, 30 to 60 minutes of real data beats 40 million sim steps deployed directly. RL trained from scratch on hardware (mean F1 55.8) beats the open-loop sim policy (41.8) on 4 of 5 songs, therefore sim pretraining's value is as an exploration-reducing initialization. The scope is narrow and honest: five short pieces (16 to 33 s), three fingers per hand, no chords beyond three notes, no thumb, and the hardest piece (Fur Elise) tops out at 66 to 71 F1 with large left-hand jumps as named failures. Refinement can only fix mis-located presses, not missed ones, and its heuristics are piano-specific (the authors suggest a VLM to generalize the coarse step).
Show more