Samip
@industriaalist
cofounder Q Labs (
solving sample efficiency and generalization
116 Following 6.8K Followers
Replacing backprop with zeroth-order optimization is inevitable.
Introducing Noise: a speedrun for pretraining without backprop.
Last week we presented Dust, the first method to approach backprop with SGD for pretraining transformers. We think another 1,000x improvement on top of it is possible, and it will require *co-designing* credit assignment, optimizers, architectures, and eventually, hardware.
We built Noise to iterate on this faster.
Show more
2/ As you scale the population, Dust beats backprop at 100k and 1M tokens and comes close at 10M and 20M.
At 20M, the current trend even puts Dust's asymptote below backprop! EGGROLL is way worse at every budget.
We describe in the post how Dust also integrates well with Adam.
Show more
We've figured out how to *pretrain transformers* with zeroth-order optimization and no backprop.
Many of the core assumptions in optimization research are completely wrong.
(paper out soon)
Show more
We think computational depth is the missing scaling axis, i.e. we should be doing a lot more deep learning!
Every other axis has been scaled by OOMs over the past few years (params, data, sparsity, test-time reasoning), but depth has been stuck at ~100 layers since GPT-3.
We've found that LLMs are both *severely* depth-bottlenecked and bad at using the depth they have, and that architectural interventions that lift this bottleneck efficiently lead to gains that increase with compute.
w/ @akshayvegesna
Show more
We've been obsessed with bending the scaling laws recently and our new paper shows that looping with model growth improves the scaling exponent, leading to gain that compounds with compute!
Everyone assumes architectural changes only give constant factor gains and pretraining progress mostly comes from data (e.g. @dwarkesh_sp's recent post).
We found that model growth, looping, and boundary operators result in compute multipliers over standard transformers that grow exponentially with each OOM of compute.
- 1.55x at 1e20 FLOPs and 2.7x projected at 1e25.
- Matches GPT-3 13B on CORE with 20x less compute
w/ @charllechen, @akshayvegesna, @andrewgwils
🧵
Show more
