You Jiacheng(@YouJiacheng):"backprop lacks search and barely explores the loss landscape" if so, we should take a backprop trained checkpoint (loss &lt; 3 under GPT-2 tokenizer) and "CPT" it to see whether these zero-th order methods are effective.

2026.10.05 23:56

"backprop lacks search and barely explores the loss landscape" if so, we should take a backprop trained checkpoint (loss < 3 under GPT-2 tokenizer) and "CPT" it to see whether these zero-th order methods are effective.

Samip@industriaalist

2026.10.05 19:41

2/ As you scale the population, Dust beats backprop at 100k and 1M tokens and comes close at 10M and 20M. At 20M, the current trend even puts Dust's asymptote below backprop! EGGROLL is way worse at every budget. We describe in the post how Dust also integrates well with Adam.