Will Held
@WilliamBarrHeld
Open LLM Training @
Formerly ML PhD w/ @Diyi_Yang, 🦙 @AIatMeta, Assistant @GoogleAI, اللغة العربية @NYUAbuDhabi
Burqueño
1.1K Following 3.4K Followers
Great research to read that informed this pipeline
Organize the Web
FineWeb-Edu
Data Curation cannot be Compute Agnostic
RegMix
OLMix
Show more
There’s an enormous amount of open training data on Hugging Face. What does it take to make it work together 🤗?
For Marin’s 535B run, we built on 25T tokens from 152 datasets with licenses permitting training.
Here’s the work between downloading those and training a model 🧵
Show more
To train better open models, we need predictable scaling.
Delphi is Marin’s first step: we pretrained many small models with one recipe, then extrapolated 300× to predict a 25B-param / 600B-token run with just 0.2% error.
Getting there took some work 🧵
Show more
