Samip(@industriaalist):Replacing backprop with zeroth-order optimization is inevitable. Introducing Noise: a speedrun for pretraining without backprop. Last week we presented Dust, the first method to approach backprop with SGD for pretraining transformers. We think another 1,000x improvement on top of it is possible, and it will require *co-designing* credit assignment, optimizers, architectures, and eventually, hardware. We built Noise to iterate on this faster.

Samip

@industriaalist

cofounder Q Labs ( solving sample efficiency and generalization

参加 June 2016

116 フォロー中 6.8K ファン

Samip@industriaalist

2026.10.12 20:17

Replacing backprop with zeroth-order optimization is inevitable. Introducing Noise: a speedrun for pretraining without backprop. Last week we presented Dust, the first method to approach backprop with SGD for pretraining transformers. We think another 1,000x improvement on top of it is possible, and it will require *co-designing* credit assignment, optimizers, architectures, and eventually, hardware. We built Noise to iterate on this faster.