Charlie O'Neill
@oneill_c
The sea is the sea
The old man is an old man
The boy is a boy and the fish is a fish
The sharks are all sharks no better and no worse
Joined September 2016
1.1K Following 22.2K Followers
this was a nice starting point to analyse a now age old question, great work Dwarkesh and Jerry! Am excited to see people extend this work, many suggestions in the post itself and the comments which people should work on
Show more
Pretraining progress seems to be coming mostly from data improvements.
@who_is_jerbear and I pretrained combinations of year-representative open model recipes and data corpuses across 2019 to 2025 at various small scales.
Data improvements contributed 3.24x as many compute multipliers as model improvements did (12.0x vs 3.7x).
And the gains stack independently - a better dataset helps every architecture about equally, and vice versa.
Here are full results, plus what we think this means for the future of AI progress:
Show more
