Quick note on data annotation for robotics ICL. For manipulation model we construct 3D pose to give clear action demonstration. Similarly, I guess we need clear intent demonstration as well, esp for long-horizon task and VLM to call action expert. In @SkildAI S1 blog, it says "since the demonstration may come from a different scene, viewpoint, or embodiment, the policy must implicitly learn the demonstrator's intent, functional correspondences, and task progress to predict the appropriate actions". Basically it introduces some variation and let the model to learn the core intent shared by those variation. Here are 3 types of annotation that helps to specify the intent: 1. outcome annotation: no matter the procedure, the outcome is stable. e.g. ON(mug,table)→INSIDE(mug,dishwasher) 2. role annotation: like the demo in S1 blog, you tell the robot to water the plant with a can but there is only a cup, then the robot knows to use the cup bc they share the same functional role (as the picture shows) 3. trajectory variation under same intent: teach them that intent is not the trajectory. Carry out a list of episodes with same goal but different trajectory