John Carmack(@ID_AA_Carmack):A chunk of the RL community really doesn’t like replay buffers. Concerns about the cost of storing a million observations are valid, but the recent evangelization of “streaming” RL, where observations are just used a single time and discarded, feels like a poor design point to me. It is interesting that it can now work at all, but I’m pretty sure that the optimal number of saved observation buffers is not “one” at any memory constraint. There would certainly be useful things to do with a managed buffer of just hundreds of sparse observations, even if you didn’t do bootstrapping from them.

2026.08.17 03:30

A chunk of the RL community really doesn’t like replay buffers. Concerns about the cost of storing a million observations are valid, but the recent evangelization of “streaming” RL, where observations are just used a single time and discarded, feels like a poor design point to me. It is interesting that it can now work at all, but I’m pretty sure that the optimal number of saved observation buffers is not “one” at any memory constraint. There would certainly be useful things to do with a managed buffer of just hundreds of sparse observations, even if you didn’t do bootstrapping from them.

显示更多

1.1K

转发到社区

热门用户