The more robot data we collect, the more important it becomes to know what is actually inside it.
Which action happened at which time? Where did the robot fail? What happened immediately before the failure? Did it recover? When did the task move into another stage?
Raw video contains all of this information, but structured annotations are what make it easier to actually work with.