Day 11 of making an autonomous G1 bartender. We started to rent some 5090 gpus because for some reason, renting 60 5090s is cheaper than an hour of using codex astra medium. However, I only rented 4 becuase making the 5090s communicate with each other is tough. Today was the first time we had the entire policy from the very start to the very end go through on the G1. Unfortunately, it took 23 MINUTES TO EXECUTE. It was supposed to take around 3 minutes and that was already too long. Let me give you guys some reasons as to why it took so long.
The configured 40 Hz was only a target. Deadline enforcement had been disabled, so slow iterations continued instead of aborting; the runtime did not maintain 40 Hz.
Camera segmentation commonly took 38-44 ms, already longer than the 25 ms budget for an entire 40 Hz iteration.
DDS body/hand publication sometimes took 25–53 ms. That could consume or exceed the complete 25 ms period before inference and safety checks were included.
Camera-frame or embedding unavailability produced hold iterations: the robot received another hold command, but the learned policy did not advance one step.
Body and Dex3 state synchronization occasionally exceeded its skew threshold, causing state acquisition retries and additional waiting.
DDS timing-gap rejection could retry the same command. Those retries consumed time without advancing the policy counter.
Entry and ownership transfer added approximately 20–25 seconds, but this explains only a small portion of the 23 minutes.
Vision encoding, GRU inference, state validation, DDS publication, and scheduling ran in the same control pipeline. Their combined latency accumulated on every one of the 7,722 steps.
The observed 25–53 ms DDS and 38–44 ms segmentation times alone do not fully explain an average of 179 ms per policy step. There must also have been repeated waits, retries, holds, or another unmeasured blocking section.
The old run did not have per-stage timing telemetry, so an exact millisecond breakdown cannot be reconstructed retroactively.
The newly deployed runtime now logs effective Hz, hold behavior, camera fetch/decode, segmentation, vision encoding, policy computation, state guards, DDS publication, scheduler delay, and complete tick latency. The next run will show precisely where the missing approximately 154 ms per step is going.
It is actual insanity how many things could go wrong. fortunately, this task doesnt really change its state too much so if things are slower than anticipated such as not recieving a frame until 250 ms later, it should still be fine.
Show more