I recently had a conversation with a NASA JPL engineer that was bemoaning the fact that the internal “Planetary Protection” department at NASA threw up substantial roadblocks for Mars missions in the name of protecting the pristine Martian environment. They will surely be apoplectic about this. I kind of think we should just get it over with and empty a septic tank on mars so nobody can imagine it needs dedicated protection. Project Vandalize Mars.

Very few will care about the actual reinforcement learning details here, but watching me grope around for understanding in the chat log might be of broader interest: I have often noticed that our estimated Q values are higher than the observed returns, sometimes substantially so, and that can’t be good for performance. For policy decisions, only the relative values matter, but an offset negatively impacts bootstrapping, so I was excited to see this paper on Relative Value Learning: After struggling a bit with “the Banach space of bounded antisymmetric pairwise functions“, I realized all that is essentially going on is subtracting two value functions. Given this framing, Chat was able to simplify the machinery into a particularly elegant form: Subtracting the mean TD error from the individual sample TD errors gives all the benefits of relative value learning. Summarized as "relative Bellman regression is TD-error centering". Unfortunately, while you can create examples where this should be very valuable (someone should write a proper paper on it!), it didn’t actually improve performance on my tasks. However, I think this has usefully narrowed down what is actually happening. Bellman iteration naturally corrects towards a correct absolute value, but only when the bootstrap values are also training targets. With Q-learning, you are often / mostly bootstrapping from a max-action that was not actually taken, so it never sees any downward Bellman pressure. This is consistent with another result I have noted: learning state value functions offline from a frozen policy without actions doesn’t seem to suffer from any value overestimation.

I recently went through a translation of Musashi’s Book of Five Rings. The introduction chronicled the evolution of swordsmanship and martial arts in general from pragmatic battlefield necessities to sports, historical curiosities, and hobbies. It was interesting to hear how in the post-WW2 occupation when martial arts were banned, Judo was represented as equivalent to western wrestling, and Kendo later to western fencing. The poignant undercurrent for me is that I can see many programming skills following that path. There are probably dozens of people reading this that remember hand assembling opcodes to hex and still have some magic numbers burned into their memory, but even the small group of people still programming in assembly today (hey, @FFmpeg !) don’t work at that primitive level now. AI is making many other programming skills much less critical. We aren’t there yet, but carefully writing code completely by hand is moving from a -jitsu to a -do. Code-do? Codo? That’s ok! The retro computing scene is delightful, full of people building and exercising old skills for the love of it. But don’t be the out of touch Kung Fu master, heir to lifetimes of tradition, that gets mauled by an amateur MMA fighter. Musashi would probably have been pretty enthusiastic about assault rifles.

If the target market is real time robotics, the Jetson Thor system, with 128 GB of memory, but only 273 GB/s of bandwidth, seems over-provisioned with expensive memory. You want the model evaluating at tens of fps, so it can't use more than 10 GB of weights at most. More memory could be used if the model is a wide mixture of experts, or a large strategic planning model is operating in small time slices over a longer period of time, and more memory always makes development life easier, but for cost optimized systems, you should be able to get by with much less.

Now that serious industrialization of space in the name of data centers is within the Overton window, it feels like time to bring up Rotovators again. Even if Starship completely meets all design goals with low overhead orbital reusability, it isn’t the end-all of cost effective access to space. Momentum exchange tethers could substantially reduce propellant consumption and operational challenges. The idea is that you have a tapered cable a few hundred km long attached to a substantial dead weight, and you set it spinning like one spoke of a wheel as it orbits earth. A rocket could fly up to space at much less than orbital velocity and grab onto the end of the rotating tether, then get pulled up to a faster speed and fling the payload off into a stable orbit. Falcon precision landings (with suicide burns!), and now SuperHeavy catches demonstrate that big rockets can precisely dock with infrastructure, and doing it in space is generally easier than doing it on the ground. The big win is in electrodynamic tethers, where you can theoretically use (lots and lots of) solar panels to raise the tether orbit after it has pulled itself lower while picking up a payload. That would be a game changer, but it hasn’t been unequivocally demonstrated to be practical. Even without that, just using chemical propulsion, it could still give significant efficiencies. Only the mass of the payload that is actually flung off the top of the tether needs to have the velocity made up by engine burns. The rest of the mass of the rocket “gives it back” when it drops back off at the bottom several rotations later over the launch site. Because the pickup and dropoff happen at much less than orbital velocity, there is no terrible reentry heating challenge. The vehicle gets simpler and more robust. So, you could rebalance a two stage rocket to be more propellant efficient, or you could make a reusable single stage to tether rocket. A SSTT might still use more propellant per kg of payload than Starship, but the operational advantages would be large. You lose lots of flexibility with tether systems compared to conventional rockets, but it would be like putting deep water harbors in key places.

I still see the Scratch “visual programming” environment positioned as an introductory path to programming for kids, but unlike game modding, I have never heard a “success story”, where someone credits their early experience with Scratch as key to a fulfilling career. Anyone?

Trista gave me personalized AI generated comic books and bookmarks on my birthday this year, and they made me smile. I noted a couple places where the rendering glitched or the model lost a little bit of coherence in the story, but these have come a long way from the early on-demand kids books created with traditional coding practices.

I finished @MorlockP ‘s Red State Mars last week, and I thought the afterword offered a really interesting insight into the creative process behind the book. Travis is a tech guy (and state representative) that writes “Good Old Fashioned Science Fiction”, and I have enjoyed his work. If you are inclined that way, go ahead and read the book. If you are put off by the implied political bias, maybe read the afterword and see if the themes sound interesting. I wound up being briefly referenced in the context of zero-G combat, since Russ Blink and I did try a little bit of judo on a parabolic airplane flight long ago. Sixty seconds of experience puts us ahead of at least 99.999999% of the population, but I bet there are some Russian cosmonauts that have done some more serious and extended tussling, which would be fun to hear about in an interview sometime.

Whenever I look at a building wall with a random-seeming mix of two brick colors, I think about decoding it as a bit stream. Surely a university building somewhere has put something meaningful in the pattern.

A chunk of the RL community really doesn’t like replay buffers. Concerns about the cost of storing a million observations are valid, but the recent evangelization of “streaming” RL, where observations are just used a single time and discarded, feels like a poor design point to me. It is interesting that it can now work at all, but I’m pretty sure that the optimal number of saved observation buffers is not “one” at any memory constraint. There would certainly be useful things to do with a managed buffer of just hundreds of sparse observations, even if you didn’t do bootstrapping from them.

One of the Keen researchers asked for guidance on compute expenditures. I replied: “Visualize the stack of $100 bills being set on fire with each run and weigh it against the knowledge sought.” In many ways, research today is in a better position to turn money into insight than ever before, but it can still be squandered.

I sat down to try the Evercade Doom port, and before I knew it, I was on level four, and realizing that I had forgotten where all the keycards were. It wound up taking me 15 minutes each on level four and five. Still a great game after all these years!

I have been trying to find something meaningful to say about the Id Software layoffs. My “Microsoft will probably be a good steward of the brand” statement isn’t aging well, and this is certainly going to dampen the mood of the founder reunion at QuakeCon next month. I’m saddened, but I can’t muster anger or outrage over it. I don’t have access to the books, but I suspect that Id Software was a marginal business from Microsoft’s perspective. I believe the reports that Minecraft revenues have been carrying several other studios. To continue being produced long term, games need to succeed, not just be beloved. Games are competing with every other option for spending your leisure time and money, and the competition is brutal. You can’t rule out the possibility that executives are idiots, but that shouldn’t be your default belief. I don’t think there is any obvious path that would have doubled the revenue from Id games. Could they have gotten more with a different pricing strategy? Could they have created more things for fans to buy? Could they have cost effectively marketed in a way that reached more players that would have loved and bought the games? Could they have changed the game designs and broadened the appeal to more players without alienating existing ones? Could they have produced the games at a lower cost, faster or cheaper? I really don’t know. The game isn’t over yet, and I hope the studio rallies through.