At 10:06pm on Aug 3, I watched a robot do something I thought was years away.
We were working on few-gradient learning with GEN-1.5, wondering how far we were from landing physical prompting: show the robot a task once, and it just does it. We YOLOed it, and the robot imitated exactly what we demonstrated, with 0 fine-tuning. We gave more prompts for different tasks. More successful rollouts landed. I could not believe my eyes.
Looking back, this represents the culmination of a huge amount of work across the entire team, many failed experiments, and years of groundwork from the community. I just happened to be the one standing in front of the robot when it all clicked.
@andyzengineer once asked me: "If Generalist failed tomorrow, what would we want to have left behind that could benefit humanity?"
I believe we found one answer: an existence proof of physical prompting.