As much as I dislike the rising prices of gpus, they are still really necessary. If you are doing ML training, a 5090 is still the most cost effective gpu since clustering multiple small gpus is usually not too efficient. For reference a 5090 is about 4-5 times faster at training than a DGX spark and a Thor is even slower. They do have different purposes but I made the mistake of thinking that the sparks would be close to the best for all general purpose ML

Is there anyone that wanted to test some policies on a G1 or Go2? I’m very fortunate to have access to both and I remember when I was playing around with G1s in sim without any real hardware. I will implement any policies and look at the code to ensure that it is transferable. The goal at WendyOS is to make the entire process of physical AI and hardware deployments easier for everyone so please send me a repo. I am also down to chat about how to improve your policy and to share information. If you know someone who may be interested, please let me know!

Is anyone else having issues with the G1 being screwed in too tightly? I swear that every G1 has at least 2 screws that get completely ruined. Anyone know of a good way to take out the screws that are broken?

What do people do to relax? I feel like I’m too out of it to play a league game but my brain needs something to do. I found chess to be a good middle ground but after hitting 2100, I feel like I can’t do it to relax too much. Tennis requires too much preemptive planning.

You never know how much you don’t know. Robotics RL is actually difficult. I know the high level reasonings behind the pipeline and the ML aspect but pattern matching is not enough. Why did my run improve by 2% and then crash to 100% failure? The KL term is too low for the learning rate. The critic’s gradient is too high. The physics engine isn’t accurate and causes grasping to fail. The tactile sensors are creating nans. All of the sensors are making training very slow. Does the sim actually transfer to real. The hz in sim is high compared to the inputs we get in real. The policy isn’t moving the robot. Too little generalization. The entropy was turned to 0 because it was most efficient to do nothing because of a simple reward error and we kept it for 3 days. Physics sim and had a fp32 vs fp64 replay error. Don’t even get me started with the numerous lies Astra told me that cost me days and also the thousand I spent in credits. I’m not even talking about reward shaping, the real aspect, or how painful it was to do data collection. That’s exactly why Wendy decided that we need to standardize development and make it really easy for everyone to deploy and test out policies. Most companies and people I talked to are not the goats at all subjects (except Wendy people ;)). Most people have their niche areas and get really good at it. The people that are RL experts and know hardware very well and specifically robotics are incredibly rare. But thankfully, it’s the most fun field in the world. Happy building everyone ;)

Day 12 of making an autonomous G1 bartender. First thing is first, the policy on the real G1 is now 30s in total, just like in sim. However, our sim policy is still not optimal and will not pick up all cans. Because we dont have much time and need to get this working in the next 5 days and RL will just not work if hyperparams are off, We decided to rent out a bunch of ~25 5090s (they are under a dollar each) and run a massive number of Protein (from pufferlib) sweep variations. We manually tested out simple things like changing the reward values from completing an entire task and we got completely different success rates within 24 hours. The default rewards that astra gave us were not ideal. So, with this Protein sweep, we have a single 5090 as an orchestrator and 4 other 5090s computing different starting values and this gives us 5 batches of sweeps. I have decided that treating the reward numbers could also improve the training, so by tmrw, I really hope that we get good hyperparameters so that my sanity could remain :)

Day 11 of making an autonomous G1 bartender. We started to rent some 5090 gpus because for some reason, renting 60 5090s is cheaper than an hour of using codex astra medium. However, I only rented 4 becuase making the 5090s communicate with each other is tough. Today was the first time we had the entire policy from the very start to the very end go through on the G1. Unfortunately, it took 23 MINUTES TO EXECUTE. It was supposed to take around 3 minutes and that was already too long. Let me give you guys some reasons as to why it took so long. The configured 40 Hz was only a target. Deadline enforcement had been disabled, so slow iterations continued instead of aborting; the runtime did not maintain 40 Hz. Camera segmentation commonly took 38-44 ms, already longer than the 25 ms budget for an entire 40 Hz iteration. DDS body/hand publication sometimes took 25–53 ms. That could consume or exceed the complete 25 ms period before inference and safety checks were included. Camera-frame or embedding unavailability produced hold iterations: the robot received another hold command, but the learned policy did not advance one step. Body and Dex3 state synchronization occasionally exceeded its skew threshold, causing state acquisition retries and additional waiting. DDS timing-gap rejection could retry the same command. Those retries consumed time without advancing the policy counter. Entry and ownership transfer added approximately 20–25 seconds, but this explains only a small portion of the 23 minutes. Vision encoding, GRU inference, state validation, DDS publication, and scheduling ran in the same control pipeline. Their combined latency accumulated on every one of the 7,722 steps. The observed 25–53 ms DDS and 38–44 ms segmentation times alone do not fully explain an average of 179 ms per policy step. There must also have been repeated waits, retries, holds, or another unmeasured blocking section. The old run did not have per-stage timing telemetry, so an exact millisecond breakdown cannot be reconstructed retroactively. The newly deployed runtime now logs effective Hz, hold behavior, camera fetch/decode, segmentation, vision encoding, policy computation, state guards, DDS publication, scheduler delay, and complete tick latency. The next run will show precisely where the missing approximately 154 ms per step is going. It is actual insanity how many things could go wrong. fortunately, this task doesnt really change its state too much so if things are slower than anticipated such as not recieving a frame until 250 ms later, it should still be fine.

Many people dont know but Mujoco cpu and MJWarp do have different implementations. The main difference we identified is how the engines represent contact between the cylindrical can and the fingers. At the exact same hand and can pose, CPU Mujoco generated 15 hand can contact points, versus 4 in the original MJWarp implementation. Those points determine the forces and friction supporting the grip. So TLDR, we asked astra to make an expert video which looked successful but we didnt know that it used normal Mujoco so our MJWarp in sim could never perform the action correctly.

Day 9 of making an autonomous G1 Bartender. It appears that when trying to transfer a policy from sim to real, the joints moved too quickly. I thought that this was a simple fix and to just create an adapter to move the joints below a certain threshold but there needed to be a lot more converting than anticipated. I went back to the sim -> behavior cloning -> humanoid movement copies -> astra controlling the G1 and realized that the original ~67 samples that were provided had really bad flicks and that we may need to redo the entire process. I made astra generate a new episode of picking up a can in sim and tested it out on the G1. After changing the limitations that astra set on the joint limits, I finally got the G1 to move acording to the controlled sim (not policy). Now I'll be generating ~30 episodes with astra and restart the entire training process. The majority of the morning was spent redesigning sim to real. I kept asking what the inputs to the policy were and astra constantly told me that it was RGB-D (i thought normal images + depth), joint velocities, accelerations, and their position (the robot knows this in real life too). But there was actually an entire processing step trying to get the hand to can locations and the RGB image was not being fed into the network at all. All gpts do is lie, smh.

I swear astra is making me go crazy. It’s lying about input features, it started training on the cpu, and then I had to tell it 3 times and that we are still using some portions of the cpu. It just straight up lies and says that things are not possible to do in mjlab. It ran 4 parallel envs for some reason. It told me 20 times that it could turn the waist manually in running mode on the g1 when I knew that was false. And the worst part is that in a day, 20x usage disappears for the week. Maybe it’s time to go back to sol. Astra is pretty insane at being a controller though.

astra lied to me and said that the only inputs are the RGB-D values and joint velocities and positions (that it knows internally). That wasted about a days worth of implementation. But its really funny to see the first behavior cloning policy completely crash out even when the MAE is literally 0.032 percent per joint. Chaos theory is really funny.

Day 6 of making an autonomous bartender G1. It appears as though people were not lying about astra's ability to control robots in sim (using a controller, not a policy). We encountered some issues with the previous hand placements being too advantageous from our previous tele-op version and decided to implement everything from the default regular mode position (we can only turn waist in regular mode). We are also including the entire pipeline in this policy so as soon as someone asks for a drink, it will hand it to them without us implementing anything else. So now, since Astra is insane at using the G1 in sim, I spun up 100 subagents to spawn the coke can in many different positions and using only the G1 POV, it should pick up the can and place it in the blue box. If this works, I will never need to do tele-op again and my eyes will be saved. Again, I will show everything I do live and will show how this will look like at a conference so everyone will be able to try it out!