The more we think about robotics, the more it seems the field is moving toward increasingly specialized datasets.
Better foundation models increase the value of task/environment-specific datasets. As base models become more capable, annotation becomes more about data structuring. The challenge shifts toward adapting data to the environments, objectives, and edge cases a model will encounter.
Over the past several months, our robotics team has been running a large number of experiments around the question of how do you maximize the training signal from the same raw data?
One conclusion we've become increasingly convinced of is that there isn't a universal annotation pipeline for robotics. Different tasks require different combinations of models, verification, and human expertise to produce the highest-quality datasets.
The great
@AndrewLeeMaas and Mitali Potnis from our team have put together a paper walking through the ideas and experiments behind this approach.
The full paper showcasing examples of how we have applied these annotations is in the comments.